Blog

Some Notes From Beta 1

The next X-Plane 10.30 beta should be out tonight (or maybe tomorrow morning); the release notes are already updated with the latest fixes, and we’re just waiting on upload.

There were some “boneheaded” bugs in beta 1 – things that just didn’t work, should have worked, and were easily fixed; we have addressed all of those that were reported for beta 2.  Despite about half a dozen boneheaded bugs, I actually think the quality of public beta 1 – relative to the amount of source change – was pretty strong.  Remember that 10.30 has perhaps 2x the amount of code change of a normal major patch, so if there isn’t too much nuclear then I think we’re doing okay.

Aircraft Authors: please test beta 2!  I think we may be able to keep 10.30’s beta period down to 4 weeks, which is a lot shorter than normal.  And the remaining bugs are all in clouds and performance, so please do not wait to test your aircraft.  Eugeny contacted me on problems with the A320 Neo on beta 1, and we have X-Plane fixed for beta 2. Don’t wait!

Performance: there have been all sorts of performance comments from users.  I think there are three separate things going on:

1. For general use, X-Plane 10.30 is faster than 10.25.
2. X-Plane 10.30’s clouds are definitely slower/more fill rate intensive than 10.25.
3. There’s something wrong with some combination of AMD drivers and 10.30 that can cause total framerate death.

I am investigating both points 2 and 3, but neither are fixed for beta 2; we wanted to get some of the stupid things fixed ASAP (e.g. the ATIS not working).

To make matters worse, my PC’s SATA controller seems to have finally lost it’s mind. Philipp had to cut the Windows build, and this has halted me from investigating AMD performance problems.  (In my initial tests I couldn’t repro anything but now I’m stuck.)

Clouds: The other area of bug fixing on my plate that simply isn’t addressed in beta 2 is the cloud locations; several users have correctly reported that the bases of the clouds are simply not where they expect.  Since there is an unfortunate link between rendering settings and fixing this bug (if the puffs we use to visualize clouds change, it effectively moves the cloud bases) I have to fix both cloud performance and cloud bases at the same time.

The expected behavior for the clouds, I think, is that if you set a stratus layer with no storms and fly an ILS, you should “break out” (that is, make visual contact with the runway environment) right around the elevation set in the weather screen.  (This is not what is happening now, which is why there is an open bug.)  If you set more ‘creative’ weather, X-Plane will start to vary things and you won’t get “reliable” break-outs.

METARs: One last note on weather: if you find that X-Plane has misinterpreted a METAR (and my sympathy is with X-Plane, because METARs just contain the most random stuff in them sometimes), then please be sure to include in your bug report:

1. The actual METAR.rwx file!  No METAR file, no way we can possibly debug.  To everyone who has sent METAR files, thank you!
2. The location of the aircraft.  A fairly precise latitude/longitude, or a FIX or navaid that you are over is good.  The easiest is when you are at an airport and can send us an ICAO code.
3. A description of what weather you actually got.

X-Plane 10.30 does have a new METAR parser so we’re trying to catch the bugs.

Airports and DSFs: we are looking at including both some fixed DSFs and additional 3-d airports in 10.30.  Both depend on the tech being ready; the airport gateway is in the deployment phase and if it goes smoothly, we could be ready to post airports.  I’m still trying to get to the bottom of DSF bugs, but I am close.  If either of these content updates miss 10.30, we’ll release them in a small patch as soon as possible.

Physics and Magic

Perhaps I am too optimistic, but: rendering solid, opaque objects that look realistic in real-time (read: for games and rendering engines) is rapidly becoming a solved problem. I look at the bleeding edge of the top-tier shooters as a preview of the direction for the field of real-time rendering*, and for the last year, physically based rendering (PBR) has been front and center.

I’ll write more about PBR in another post, but the short version is that, in PBR, the equations for light interacting with materials in a rendering engine are modeled more closely on real-world physical equations; the goal is something that is very accurate to real world optics, and produces realistic results over its entire range of inputs.

PBR is made possible by increased computing power on the GPU.  (And here when I say computing power I really do mean computing, or ALU power; PBR requires more math per pixel to calculate for a wider range of optical effects, often with more expensive but better approximations of the underlying optics.)  We now have big enough GPUs to do “real physics” (well, real-ish) when it comes to light and solid objects.  This is a huge step forward when you consider that a decade ago, calculating lighting equations per-pixel was marginal and had to be done in the wrong color space for speed.**

So for solid objects like airplane fuselages, runways, buildings, baggage trucks, etc., we have physics.

Then there are clouds, and the air, and rain and fog.  For those we have magic.

By magic, I do not mean something truly magical, something outside the realm of what is possible with computer graphics.  Rather, I mean the definition a professional magician like Penn Jilette might use: trickery.  To quote Penn***:

The only secret in magic, there’s only one, and that is, the secret must be ugly.  You cannot have a beautiful secret…So, in magic, what you want is an idea that is not beautiful…If I have to say, he’s lying about that, and there’s gaffer’s tape over behind there, and, ah, they’re not actually telling you the exact truth here, and it gets so…you don’t get an a-ha.

But there’s another side to magic: to make the trick look magical, the magician has to practice – the trick is a craft, and it can be finely honed.  That’s why, before Teller does the Red Ball trick, Penn has to tell you how it’s done; you can’t appreciate the workmanship if you don’t know how much workmanship went in.  The secret of the trick may be ugly, but the craftsmanship can be beautiful.****

Clouds, fog, atmospheric scattering – in X-Plane (and pretty much all rendering engines), these are not physics, they are magic, meaning they are tricks.  The sky is a hemisphere around your head, like in The Truman Show.  The color of the clouds are just painted onto textures with photoshop.

It’s trickery, but I feel no shame in these things, because the real physics are approximately a gajllion times more expensive than we can possibly afford on today’s hardware.  A single square meter of ground (on an overcast day) might be lit by 10^18 photons per second – a lot more when the sun comes out, and each of those photons can take a different path through the atmosphere, bouncing off molecules multiple times before arriving at a surface.  The actual optics of clouds and the atmosphere is incredibly complex.

The hard work and craft of the trickery is in making all of the tricks work together to make a unified effect, and this is where some of my time has been spent over the last few days.  It is frustrating work because there isn’t a right answer (the way there might be in the programming of physics equations); instead it is a series of tricks and compromises that try to make an illusion as convincing as possible given limited resources.

Whenever I describe the code behind atmospheric scattering, clouds and fog to Chris, he is always horrified by the complexity, by the layers of tricks.  But such complexity matches Teller’s the definition of a functional magic trick:

Make the secret a lot more trouble than the trick seems worth. You will be fooled by a trick if it involves more time, money and practice than you (or any other sane onlooker) would be willing to invest.

If clouds and scattering weren’t so much trouble, they wouldn’t be magic.

https://www.youtube.com/watch?v=FFq-fn-nSZI

* The field of real-time rendering includes computer programs that produce graphics at high framerates, like games and CAD programs.  This contrasts it with the movie industry, where they can spend two years with a giant render farm to produce 90 minutes of final output.

** In 2004, we had the GeForce FX series and the Radeon 9700.  The 9700 was insanely powerful for its time, but it only had a 24-bit internal floating point path, a 96 instruction shader limit, etc.; you could do per-pixel calculations but burning shader instructions to work in linear space wasn’t on anyone’s mind.  The Geforce FX series was significantly less powerful.  This chapter wasn’t published until 2007 – 3 years later.

*** The link is to a Radio Lab story about a man who tries to understand how his grandparents performed a mentalism act on the radio – it’s a good listen!

**** In Fool Us, many times Penn and Teller knows exactly how the trick was done, but still praise the magician for their presentation – because they know how the trick was done, they know how hard it is to do it well.  Within the show, they seem to have a preference for tricks that are not easy and require skill to perform.

Overwriting X-Plane Files Is Like Smoking In A Fireworks Factory

My previous post described the dangers of relying on art controls for an add-on that needs to be forward compatible with future versions of X-Plane; the short version is that art controls are an internal tool and not a public SDK, so we don’t try to maintain compatibility with updates.

So I figured I’d blog about the other dangerous way to build an add-on: creating an add-on by modifying X-Plane’s files.

All of the well-supported forward compatible ways to make add-ons (scenery packs, aircraft, and plugins) work by adding new files to X-Plane, usually in a folder specific to your add-on (a scenery pack, an aircraft folder, or a fat plugin).  This is very intentional – it ensures that add-ons don’t clash with our installer/updater over ownership of specific files.

But if you make an add-on that requires the user to replace one of our files with your own, all bets are off, and future patches of X-Plane will probably both screw up your add-on and be an updater nightmare for your users.  The best bet is to avoid modding our files.

Use the Library

The library system lets you virtually replace our files without actually touching them by mapping your own art assets to a given library path.  This is the right way to customize the scenery; editing the files on disk is not a good choice.  We may move files around within our own art assets in an update, but the library paths remain the same.  Thus building a library replacement is relatively safe; hacking the library scenery packs inside “Resources/default scenery” is not.

You should also use the library when you want to use our art assets; do not copy our art assets out of X-Plane into your scenery pack.  If you use the library, improvements to the art assets are automatically transferred to your scenery pack.

Some Dangerous Files

Here are some files that really aren’t meant to be modified in an add-on:

• Our shaders
• Random bitmaps in Resources/bitmaps, including sky and cloud textures
• Lights.txt (the configuration file for named lights)

These files are all really part of X-Plane’s implementation and change regularly as we try to improve the sim.

You Can’t Share Naked Textures

You might notice that there is no way to directly use X-Plane’s textures in your add-on. At best you can use the .ter or .pol file that references them.

This is intentional!  Our artists may need to remap the location of elements within a texture as part of a future sim change, and when this happens, add-ons reference the texture would show the wrong part of the image.  That’s why add-ons can only reference some asset that itself references the texture; if we change the texture, we change our .pol files or .ter files or what-have-you to compensate for the UV map change.

X-Plane Is Not a Library

Austin and I get a steady stream of “can we just get access to X for our add-on” where X is some part of the sim that the add-on maker would like to re-use.  Unfortunately most cases where internals aren’t public are because we can’t safely share them; X-Plane is an application first and a library or platform second, and in many of these cases, to share these internal parts of X-Plane would be to commit to not ever making them better, which we don’t want to do.

Art Controls Are An Active Volcano

I received an email yesterday that I found a little bit concerning; an author wrote this:

I hope 10.30 is not going to change any of the datarefs we as a community have identified and are using to make the sim better and if it does I will work to modify the scripts to make them work again.

Here’s the problem: his add-on is one of the new Lua scripts that changes the internal art controls for atmospheric scattering – it is almost certainly not built on public x-Plane SDK datarefs, it is built upon finding, locating, and hacking the internal art controls in the sim.

So I hope the author isn’t going to mind working to modify his scripts again, because those art controls will change.

Simply put, building an add-on based on art controls is like building a house next to an active volcano – it might be a nice place to live, but you’re going to have to rebuild your house from time to time.  Be prepared to do your work over, because art controls will change – it’s a matter of when, not if.  The rest of this post is some warnings, scare tactics, and an explanation of why the art controls change.

Scary Warnings, Etc.

In my past comments, both public and private, I have tried to be clear that the art controls are not a stable interface, they are subject to change, and that you can’t build an add-on that will keep working while we patch if you use them.  (I’d also hope that the fact that every art control starts with sim/private would get people’s attention.)  But I apparently never wrote one big public global “no warranty” message*, so…um…here we go.

The art controls are not a public interface to make X-Plane add-ons.  They are an internal development tool.  They are unsupported, undocumented, unsafe, and most importantly subject to change with every patch of X-Plane.  If you create an add-on that requires reading or writing the art controls, you can expect that your add-on will stop working when X-Plane is updated.  When your add-on breaks, please do not complain or file a bug.

So hopefully that makes things clear.  Art controls will change,you’ll have to do your work over, and we’re telling (and have been telling) you that since people first figured out how to access them.  If you learned about art controls from someone other than me and weren’t aware of this, the person who taught you forgot to tell you the most important thing.

What Is An Art Control

An art control is an internal hook into our engine that we use to tune and calibrate algorithms, measure performance, and debug problems.  I started adding art controls so that:

• Our art team could “tune” X-Plane without having to have the source code or recompile the sim and
• So I could see the results of changes to the engine without having to quit and restart X-Plane.

They are a critical part of X-Plane development.

They are not, however, some kind of public interface for third party development.  We have public datarefs, listed on the X-Plane SDK website, that let you replace the flight model, customize the control surfaces, etc.  Those datarefs are a real “interface” to the sim – even when we modify X-Plane, we try to keep that interface working.  Many of the public datarefs are over ten years old now.

By comparison, the art controls aren’t an interface – they are deeply tied to the how of the engine, e.g. the specific algorithm we are using today.  If we come up with a better algorithm, those tuning controls lose relevance and the art control goes away.

Why Don’t You Make the Art Controls Public

The immediate question that most developers have when they hear my rantexplanation of art controls is: why don’t you make the art controls public so we can use them?

It’s a good question, particularly when the art controls are useful.  The answer is that the art controls don’t represent a specification for what the sim does, they represent side-effects of the algorithm we are currently using.  Let me use atmospheric scattering as an example.

Atmospheric scattering is essentially a mathematical formula (run on the GPU) that makes the color of pixels “more blue” when they are farther away.  The art controls for scattering become constants in that formula – thus you can change the constants (but not the overall shape) of the formula via art controls.

But here’s the problem: the formula I use isn’t very good.**  The result is that when the constants are high (for “heavy” scattering) the far view turns a really rich, electric blue that just doesn’t look anything like the planet we actually live on.

As long as we keep this formula, we’re stuck with two options, neither of which are much fun:

1. Keep scattering turned way down.  This is what we ship with; clearly users want more scattering – you can see that by all of the positive reactions to add-ons that crank the art controls up.
2. Turn up the scattering and try to ignore the blue.  This is what most of the scripts that hack the scattering art controls do, and I would say it’s clearly a trade-off.

So I am working now on a better formula, one that won’t produce incorrect colors at high scattering levels.  But when that formula is done, it will be a different formula, with different constants in different places in the equations.

And that is why art controls change. The very code that gave birth to them changes, and they get deleted and replaced with new art controls.  The only way to keep the scattering controls forever is to never fix the flaws in the current formulas.

What Art Controls Will Change, and When

First, I hope that if I’ve made one thing clear, it’s this: if you want your add-on to keep working with future X-Plane patches, you need to avoid the art controls.

Realistically, the art controls tend to change when there is major development in the sub-system that they are part of.  So the shadow art controls will change any time we try to make the shadows better; the scattering art controls will change any time we try to make scattering better.

So if you are just tinkering with X-Plane, the good news is: if your art controls have changed, there’s a chance that you may be able to achieve a better effect using the new code.

One final note: if you want to build an add-on that needs to work past version updates (e.g. a payware add-on where having to follow our patch schedule is not a good idea) and the only way you can accomplish your goals is with art controls, send me an email.  We may be able to add a real interface to replace art control hacking.  This won’t always be possible, but sometimes it is, and the results will be a lot less work for your add-on in the long term and happier customers.

* See Resources/settings.txt for the warranty on hacking settings.txt.

** In my defense, the real look of the sky comes from a huge number of photons crashing into a huge number of molecules a huge number of times – we’re not even remotely close to having enough computing power to solve that honestly, so all scattering code is based on gross simplifications of gross simplifications.