Category: Development

Re-recuts and Last Minute bugs

It looks like the next cut of 10.30 will be a release candidate.  I am not sure when we’ll release it, but I am hoping for this weekend.

If you installed 10.30b8 on a full beta, and thus downloaded 500 MB of DSF recuts, I’m afraid to say you’ll end up re-downloading them again for the RC.  A user reported bad edge matching between the recuts and the original tiles for one airport.  The underlying bug is one that could affect any seam between a recut and an original tile, so to be safe I’m recutting the entire recut set with the fixed code.  So…I apologize for the bandwidth hit, but I think better safe and sorry here.

If you do find a last minute bug that you have not reported, please be sure to:

1. Report it on the bug reporter.  If you haven’t used that page, we don’t know about it.  Blog comments do not count, forum posts do not count, and random emails to people with x-plane.com email addresses do not count!  Don’t send a bug to a place where it will be lost this late in the game!*
2. Please make sure the bug is complete, with detailed instructions to reproduce it step by step and a way to get the reproduction materials necessary.  This close to RC, we no longer have time to spend a few days going up and back to understand the bug and then realize we don’t even have the payware ACF we need!

There are two features that will ship whenever they ship, e.g. in 1030 if they’re ready, or we’ll cut a 10.31 in a week or two if we have to:

1. Oculus Rift support.  There is a new SDK for the Rift, and it’s not backward compatible with the old one, so Philipp has some coding to do there.  I think we’re going to enable the Rift as soon as it’s done, but given how few sets are already sold (since they’re just low-res developer previews right now) we’re not going to halt 1030; we’ll cut a quick bug fix release if we need to.
2. Cylindrical Projection.  Austin reworked cylindrical projection for pro users for 1030; if we find we need to get a few more refinements into the warping code, that may go into a bug fix release too.

My goal is an RC by the weekend, but that’s more of an aspiration than a promise.

* If you like a bug and don’t want it fixed, perhaps emailing me directly or posting it on the forum is the best thing you can do!

Fixing X-Plane 10.30 Performance

This week I received my new PC, set it up, and was able to actually look at 10.30 performance problems with AMD hardware on the Catalyst 13-9 drivers.  With the actual “problematic” configuration (and a hard drive that wasn’t barfing up sectors like an infant with reflux) it only took about 15 minutes to find the actual problem.

Which was…face palm…a GL drawing code path that I had disabled during private testing (due to low performance) was actually not disabled at all due to a lack of parenthesis.  So I’m an idiot.

What’s interesting is the relative effects of this code path on multiple platforms.

• On OS X, this code path runs at pretty much full speed, so having it in the beta didn’t reveal any actual problems.  Hence while my PC was dead, my few attempts to reproduce the reported behavior on my Mac totally failed.
• The code path is slower on both Nvidia 331.88 drivers and AMD 14-4 drivers.  But…1030’s basic performance is faster than 10.25, so the two canceled out, yielding somewhat reasonable looking fps tests.  That’s why when I asked Philipp to try to reproduce things on an NVidia windows box, he told me things didn’t look broken.
• On the AMD 13-9 drivers, the code path is absolute death.  I believe that this is where the majority of bug reports were coming from.
• The actual total performance win from 10.25 to 10.30 appears to be quite a bit bigger on NVidia hardware than AMD hardware. For some reason, one of the modernization changes in 1030 is more important for NV than AMD.  I don’t know what the specific changes that are helping are – the goal of 1030 is to simply use the best code paths we can find.

Here’s an excerpt from my performance testing numbers – this is with fps test 3 with the 747 replay.  The hardware for the test is a GeForce 680 GTX and a Radeon HD 7970.  Expect better fps with the NV card – it’s a generation newer than the AMD card.  Numbers are framerate, probably good +/- 1 fps.

Driver/hw   10.25       10.30b5      10.30b6 (fixed)
NV 331.88   34          34           42
AMD 13-9    28          11           30
AMD 14-4    27          28           29

So you can see here that on better functioning drivers (331.88, 14-4) even with the performance problems of 10.30b5 the net change in framerate from 10.25 was positive. But once I remove the stupid code path, things get a lot better.

That 11 fps in 10.30b5 with 13-9 is the bad code path behaving badly.  That will be fixed in the next beta.

I am hoping that this code path is also responsible for the remaining “low fps in local map” reports, but I haven’t confirmed that yet.

Your Mileage Will Vary

One of the problems with performance testing is that X-Plane’s content varies by add-ons, so how much the problems with 10.30b5 affect you will have a lot to do with which aircraft you fly. This is why some users already thought 10.30 was a win over 10.25 performance wise, and others thought the build was totally unusable.  How much speedup you get will depend on the kind of content you use.

AMD: What Drivers Should I Use

The 14-4 drivers are slightly slower than the 13-9 drivers for either shipping X-Plane 10.25 or 10.30b6 (not yet released).  I believe this represents a fundamental (small) loss of performance with the newer drivers. The loss is actually smaller than it looks if you view the raw fps numbers with fractions, and the precision for the tests isn’t better than 1 fps.

If you can run the 14-4 drivers you should – they fix a bug in sRGB blending with HDR that makes the 13-9 drivers look rather weird with HDR mode.

If you are stuck on the 13-9 drivers because you have old hardware (the oldest GPU the new drivers support is the HD 6000 series – so if you have the HD 5870 or older you need the older drivers) at least 10.30b6 will improve performance.

Clouds Are Their Own Performance Problem

The other big performance problem that I am aware of besides this one is that clouds use a lot more fill rate than they used to.  This is my next bug to try to fix – it is totally separate from this one, and affects all platforms.  My goal is to get the clouds to use the same fill rate as they did in 10.25.  That work isn’t done yet, but at least now I have hardware to work on.

Some Notes From Beta 1

The next X-Plane 10.30 beta should be out tonight (or maybe tomorrow morning); the release notes are already updated with the latest fixes, and we’re just waiting on upload.

There were some “boneheaded” bugs in beta 1 – things that just didn’t work, should have worked, and were easily fixed; we have addressed all of those that were reported for beta 2.  Despite about half a dozen boneheaded bugs, I actually think the quality of public beta 1 – relative to the amount of source change – was pretty strong.  Remember that 10.30 has perhaps 2x the amount of code change of a normal major patch, so if there isn’t too much nuclear then I think we’re doing okay.

Aircraft Authors: please test beta 2!  I think we may be able to keep 10.30’s beta period down to 4 weeks, which is a lot shorter than normal.  And the remaining bugs are all in clouds and performance, so please do not wait to test your aircraft.  Eugeny contacted me on problems with the A320 Neo on beta 1, and we have X-Plane fixed for beta 2. Don’t wait!

Performance: there have been all sorts of performance comments from users.  I think there are three separate things going on:

1. For general use, X-Plane 10.30 is faster than 10.25.
2. X-Plane 10.30’s clouds are definitely slower/more fill rate intensive than 10.25.
3. There’s something wrong with some combination of AMD drivers and 10.30 that can cause total framerate death.

I am investigating both points 2 and 3, but neither are fixed for beta 2; we wanted to get some of the stupid things fixed ASAP (e.g. the ATIS not working).

To make matters worse, my PC’s SATA controller seems to have finally lost it’s mind. Philipp had to cut the Windows build, and this has halted me from investigating AMD performance problems.  (In my initial tests I couldn’t repro anything but now I’m stuck.)

Clouds: The other area of bug fixing on my plate that simply isn’t addressed in beta 2 is the cloud locations; several users have correctly reported that the bases of the clouds are simply not where they expect.  Since there is an unfortunate link between rendering settings and fixing this bug (if the puffs we use to visualize clouds change, it effectively moves the cloud bases) I have to fix both cloud performance and cloud bases at the same time.

The expected behavior for the clouds, I think, is that if you set a stratus layer with no storms and fly an ILS, you should “break out” (that is, make visual contact with the runway environment) right around the elevation set in the weather screen.  (This is not what is happening now, which is why there is an open bug.)  If you set more ‘creative’ weather, X-Plane will start to vary things and you won’t get “reliable” break-outs.

METARs: One last note on weather: if you find that X-Plane has misinterpreted a METAR (and my sympathy is with X-Plane, because METARs just contain the most random stuff in them sometimes), then please be sure to include in your bug report:

1. The actual METAR.rwx file!  No METAR file, no way we can possibly debug.  To everyone who has sent METAR files, thank you!
2. The location of the aircraft.  A fairly precise latitude/longitude, or a FIX or navaid that you are over is good.  The easiest is when you are at an airport and can send us an ICAO code.
3. A description of what weather you actually got.

X-Plane 10.30 does have a new METAR parser so we’re trying to catch the bugs.

Airports and DSFs: we are looking at including both some fixed DSFs and additional 3-d airports in 10.30.  Both depend on the tech being ready; the airport gateway is in the deployment phase and if it goes smoothly, we could be ready to post airports.  I’m still trying to get to the bottom of DSF bugs, but I am close.  If either of these content updates miss 10.30, we’ll release them in a small patch as soon as possible.

Physics and Magic

Perhaps I am too optimistic, but: rendering solid, opaque objects that look realistic in real-time (read: for games and rendering engines) is rapidly becoming a solved problem. I look at the bleeding edge of the top-tier shooters as a preview of the direction for the field of real-time rendering*, and for the last year, physically based rendering (PBR) has been front and center.

I’ll write more about PBR in another post, but the short version is that, in PBR, the equations for light interacting with materials in a rendering engine are modeled more closely on real-world physical equations; the goal is something that is very accurate to real world optics, and produces realistic results over its entire range of inputs.

PBR is made possible by increased computing power on the GPU.  (And here when I say computing power I really do mean computing, or ALU power; PBR requires more math per pixel to calculate for a wider range of optical effects, often with more expensive but better approximations of the underlying optics.)  We now have big enough GPUs to do “real physics” (well, real-ish) when it comes to light and solid objects.  This is a huge step forward when you consider that a decade ago, calculating lighting equations per-pixel was marginal and had to be done in the wrong color space for speed.**

So for solid objects like airplane fuselages, runways, buildings, baggage trucks, etc., we have physics.

Then there are clouds, and the air, and rain and fog.  For those we have magic.

By magic, I do not mean something truly magical, something outside the realm of what is possible with computer graphics.  Rather, I mean the definition a professional magician like Penn Jilette might use: trickery.  To quote Penn***:

The only secret in magic, there’s only one, and that is, the secret must be ugly.  You cannot have a beautiful secret…So, in magic, what you want is an idea that is not beautiful…If I have to say, he’s lying about that, and there’s gaffer’s tape over behind there, and, ah, they’re not actually telling you the exact truth here, and it gets so…you don’t get an a-ha.

But there’s another side to magic: to make the trick look magical, the magician has to practice – the trick is a craft, and it can be finely honed.  That’s why, before Teller does the Red Ball trick, Penn has to tell you how it’s done; you can’t appreciate the workmanship if you don’t know how much workmanship went in.  The secret of the trick may be ugly, but the craftsmanship can be beautiful.****

Clouds, fog, atmospheric scattering – in X-Plane (and pretty much all rendering engines), these are not physics, they are magic, meaning they are tricks.  The sky is a hemisphere around your head, like in The Truman Show.  The color of the clouds are just painted onto textures with photoshop.

It’s trickery, but I feel no shame in these things, because the real physics are approximately a gajllion times more expensive than we can possibly afford on today’s hardware.  A single square meter of ground (on an overcast day) might be lit by 10^18 photons per second – a lot more when the sun comes out, and each of those photons can take a different path through the atmosphere, bouncing off molecules multiple times before arriving at a surface.  The actual optics of clouds and the atmosphere is incredibly complex.

The hard work and craft of the trickery is in making all of the tricks work together to make a unified effect, and this is where some of my time has been spent over the last few days.  It is frustrating work because there isn’t a right answer (the way there might be in the programming of physics equations); instead it is a series of tricks and compromises that try to make an illusion as convincing as possible given limited resources.

Whenever I describe the code behind atmospheric scattering, clouds and fog to Chris, he is always horrified by the complexity, by the layers of tricks.  But such complexity matches Teller’s the definition of a functional magic trick:

Make the secret a lot more trouble than the trick seems worth. You will be fooled by a trick if it involves more time, money and practice than you (or any other sane onlooker) would be willing to invest.

If clouds and scattering weren’t so much trouble, they wouldn’t be magic.

https://www.youtube.com/watch?v=FFq-fn-nSZI

* The field of real-time rendering includes computer programs that produce graphics at high framerates, like games and CAD programs.  This contrasts it with the movie industry, where they can spend two years with a giant render farm to produce 90 minutes of final output.

** In 2004, we had the GeForce FX series and the Radeon 9700.  The 9700 was insanely powerful for its time, but it only had a 24-bit internal floating point path, a 96 instruction shader limit, etc.; you could do per-pixel calculations but burning shader instructions to work in linear space wasn’t on anyone’s mind.  The Geforce FX series was significantly less powerful.  This chapter wasn’t published until 2007 – 3 years later.

*** The link is to a Radio Lab story about a man who tries to understand how his grandparents performed a mentalism act on the radio – it’s a good listen!

**** In Fool Us, many times Penn and Teller knows exactly how the trick was done, but still praise the magician for their presentation – because they know how the trick was done, they know how hard it is to do it well.  Within the show, they seem to have a preference for tricks that are not easy and require skill to perform.

Overwriting X-Plane Files Is Like Smoking In A Fireworks Factory

My previous post described the dangers of relying on art controls for an add-on that needs to be forward compatible with future versions of X-Plane; the short version is that art controls are an internal tool and not a public SDK, so we don’t try to maintain compatibility with updates.

So I figured I’d blog about the other dangerous way to build an add-on: creating an add-on by modifying X-Plane’s files.

All of the well-supported forward compatible ways to make add-ons (scenery packs, aircraft, and plugins) work by adding new files to X-Plane, usually in a folder specific to your add-on (a scenery pack, an aircraft folder, or a fat plugin).  This is very intentional – it ensures that add-ons don’t clash with our installer/updater over ownership of specific files.

But if you make an add-on that requires the user to replace one of our files with your own, all bets are off, and future patches of X-Plane will probably both screw up your add-on and be an updater nightmare for your users.  The best bet is to avoid modding our files.

Use the Library

The library system lets you virtually replace our files without actually touching them by mapping your own art assets to a given library path.  This is the right way to customize the scenery; editing the files on disk is not a good choice.  We may move files around within our own art assets in an update, but the library paths remain the same.  Thus building a library replacement is relatively safe; hacking the library scenery packs inside “Resources/default scenery” is not.

You should also use the library when you want to use our art assets; do not copy our art assets out of X-Plane into your scenery pack.  If you use the library, improvements to the art assets are automatically transferred to your scenery pack.

Some Dangerous Files

Here are some files that really aren’t meant to be modified in an add-on:

• Our shaders
• Random bitmaps in Resources/bitmaps, including sky and cloud textures
• Lights.txt (the configuration file for named lights)

These files are all really part of X-Plane’s implementation and change regularly as we try to improve the sim.

You Can’t Share Naked Textures

You might notice that there is no way to directly use X-Plane’s textures in your add-on. At best you can use the .ter or .pol file that references them.

This is intentional!  Our artists may need to remap the location of elements within a texture as part of a future sim change, and when this happens, add-ons reference the texture would show the wrong part of the image.  That’s why add-ons can only reference some asset that itself references the texture; if we change the texture, we change our .pol files or .ter files or what-have-you to compensate for the UV map change.

X-Plane Is Not a Library

Austin and I get a steady stream of “can we just get access to X for our add-on” where X is some part of the sim that the add-on maker would like to re-use.  Unfortunately most cases where internals aren’t public are because we can’t safely share them; X-Plane is an application first and a library or platform second, and in many of these cases, to share these internal parts of X-Plane would be to commit to not ever making them better, which we don’t want to do.

Art Controls Are An Active Volcano

I received an email yesterday that I found a little bit concerning; an author wrote this:

I hope 10.30 is not going to change any of the datarefs we as a community have identified and are using to make the sim better and if it does I will work to modify the scripts to make them work again.

Here’s the problem: his add-on is one of the new Lua scripts that changes the internal art controls for atmospheric scattering – it is almost certainly not built on public x-Plane SDK datarefs, it is built upon finding, locating, and hacking the internal art controls in the sim.

So I hope the author isn’t going to mind working to modify his scripts again, because those art controls will change.

Simply put, building an add-on based on art controls is like building a house next to an active volcano – it might be a nice place to live, but you’re going to have to rebuild your house from time to time.  Be prepared to do your work over, because art controls will change – it’s a matter of when, not if.  The rest of this post is some warnings, scare tactics, and an explanation of why the art controls change.

Scary Warnings, Etc.

In my past comments, both public and private, I have tried to be clear that the art controls are not a stable interface, they are subject to change, and that you can’t build an add-on that will keep working while we patch if you use them.  (I’d also hope that the fact that every art control starts with sim/private would get people’s attention.)  But I apparently never wrote one big public global “no warranty” message*, so…um…here we go.

The art controls are not a public interface to make X-Plane add-ons.  They are an internal development tool.  They are unsupported, undocumented, unsafe, and most importantly subject to change with every patch of X-Plane.  If you create an add-on that requires reading or writing the art controls, you can expect that your add-on will stop working when X-Plane is updated.  When your add-on breaks, please do not complain or file a bug.

So hopefully that makes things clear.  Art controls will change,you’ll have to do your work over, and we’re telling (and have been telling) you that since people first figured out how to access them.  If you learned about art controls from someone other than me and weren’t aware of this, the person who taught you forgot to tell you the most important thing.

What Is An Art Control

An art control is an internal hook into our engine that we use to tune and calibrate algorithms, measure performance, and debug problems.  I started adding art controls so that:

• Our art team could “tune” X-Plane without having to have the source code or recompile the sim and
• So I could see the results of changes to the engine without having to quit and restart X-Plane.

They are a critical part of X-Plane development.

They are not, however, some kind of public interface for third party development.  We have public datarefs, listed on the X-Plane SDK website, that let you replace the flight model, customize the control surfaces, etc.  Those datarefs are a real “interface” to the sim – even when we modify X-Plane, we try to keep that interface working.  Many of the public datarefs are over ten years old now.

By comparison, the art controls aren’t an interface – they are deeply tied to the how of the engine, e.g. the specific algorithm we are using today.  If we come up with a better algorithm, those tuning controls lose relevance and the art control goes away.

Why Don’t You Make the Art Controls Public

The immediate question that most developers have when they hear my rantexplanation of art controls is: why don’t you make the art controls public so we can use them?

It’s a good question, particularly when the art controls are useful.  The answer is that the art controls don’t represent a specification for what the sim does, they represent side-effects of the algorithm we are currently using.  Let me use atmospheric scattering as an example.

Atmospheric scattering is essentially a mathematical formula (run on the GPU) that makes the color of pixels “more blue” when they are farther away.  The art controls for scattering become constants in that formula – thus you can change the constants (but not the overall shape) of the formula via art controls.

But here’s the problem: the formula I use isn’t very good.**  The result is that when the constants are high (for “heavy” scattering) the far view turns a really rich, electric blue that just doesn’t look anything like the planet we actually live on.

As long as we keep this formula, we’re stuck with two options, neither of which are much fun:

1. Keep scattering turned way down.  This is what we ship with; clearly users want more scattering – you can see that by all of the positive reactions to add-ons that crank the art controls up.
2. Turn up the scattering and try to ignore the blue.  This is what most of the scripts that hack the scattering art controls do, and I would say it’s clearly a trade-off.

So I am working now on a better formula, one that won’t produce incorrect colors at high scattering levels.  But when that formula is done, it will be a different formula, with different constants in different places in the equations.

And that is why art controls change. The very code that gave birth to them changes, and they get deleted and replaced with new art controls.  The only way to keep the scattering controls forever is to never fix the flaws in the current formulas.

What Art Controls Will Change, and When

First, I hope that if I’ve made one thing clear, it’s this: if you want your add-on to keep working with future X-Plane patches, you need to avoid the art controls.

Realistically, the art controls tend to change when there is major development in the sub-system that they are part of.  So the shadow art controls will change any time we try to make the shadows better; the scattering art controls will change any time we try to make scattering better.

So if you are just tinkering with X-Plane, the good news is: if your art controls have changed, there’s a chance that you may be able to achieve a better effect using the new code.

One final note: if you want to build an add-on that needs to work past version updates (e.g. a payware add-on where having to follow our patch schedule is not a good idea) and the only way you can accomplish your goals is with art controls, send me an email.  We may be able to add a real interface to replace art control hacking.  This won’t always be possible, but sometimes it is, and the results will be a lot less work for your add-on in the long term and happier customers.

* See Resources/settings.txt for the warranty on hacking settings.txt.

** In my defense, the real look of the sky comes from a huge number of photons crashing into a huge number of molecules a huge number of times – we’re not even remotely close to having enough computing power to solve that honestly, so all scattering code is based on gross simplifications of gross simplifications.