Category: Development

The Cloud Texture Layout Didn’t Change

I heard quite a bit from the cloud and weather tweakers when we started 1030.  At the beginning of the beta, I thought I’d have the visibility and fog improvements ready to go, and so I posted about possible changes.

Those features got pushed out to a future patch.  The cloud lighting did change.  So what was the total set of changes?

• Scattering and lighting did not change, and there are no changes to art controls.  So hacks to atmospheric scattering are unaffected.
• The format (size, color format, usage) of the cloud textures did not change – if you have a replacement cloud texture set, you can keep using it.
• The actual default cloud textures did change – I beat them up a bit in PhotoShop.   Hence if you have custom cloud textures installed, the updater will yelp.
• The lighting and placement of the cloud puffs has changed, so color choices for custom textures may have to be revisited.
• Art controls to kill clouds are still there – but if you want our clouds gone, use the new override_clouds dataref!
• There are new art controls controlling the (now redone) white-out effect.

So the total impact on cloud and atmosphere tweaks should be relatively low – but the lighting definitely changed on cloud puffs.

My expectation is that when the scattering art controls change (and they will change someday) it will be due to a change in the scattering code itself that will be “worth it”, e.g. a major iteration to improve the look of scattering itself.

Where Did London Go (Linux Testing Needed)

Update: several people have volunteered, and the installers are now tested – thanks!

One of the remaining bugs in 1030: when you install a 1030 update after having received DSF recuts (e.g. the update from 1030b8 to 1030 RC 1)  the Aerosoft custom EGLL pack disappears.

It turns out this is a bug in our installer/updater – the bug has actually been in the installer/updater for a long time but normally didn’t act up (unless you modified your installation in some very unorthodox ways).

I am working on a new rev of the installer (3.22) that will fix this.  But I need Linux volunteers to test, especially the 32-bit edition.  If you can run 32 and 64-bit apps on Linux, please email me (ben at x-plane dot com) and I will set you up with test apps.

There are a few straggling bugs that we’re hoping to fix in RC2 this week.

Update: I totally forgot this announcement for Linux users*: the new 3.22 installers come in 32 and (for the first time) 64-bit flavors.

This means that with X-Plane 10.30 and the new installer, all X-Plane apps come in a 64-bit app, and you don’t have to install multi-arch gunk on your 64-bit Linux install to run X-Plane.

We will continue to support 32-bits on all platforms for the lifetime of X-Plane 10, and the installer, Airfoil-Maker and Plane-Maker will remain 32-bit on Mac/Windows, where 32 bit support is guaranteed without any hassle.

* This is what happens when I sleep even less than normal.  You can only imagine how good these installers must be. 🙂

Re-recuts and Last Minute bugs

It looks like the next cut of 10.30 will be a release candidate.  I am not sure when we’ll release it, but I am hoping for this weekend.

If you installed 10.30b8 on a full beta, and thus downloaded 500 MB of DSF recuts, I’m afraid to say you’ll end up re-downloading them again for the RC.  A user reported bad edge matching between the recuts and the original tiles for one airport.  The underlying bug is one that could affect any seam between a recut and an original tile, so to be safe I’m recutting the entire recut set with the fixed code.  So…I apologize for the bandwidth hit, but I think better safe and sorry here.

If you do find a last minute bug that you have not reported, please be sure to:

1. Report it on the bug reporter.  If you haven’t used that page, we don’t know about it.  Blog comments do not count, forum posts do not count, and random emails to people with x-plane.com email addresses do not count!  Don’t send a bug to a place where it will be lost this late in the game!*
2. Please make sure the bug is complete, with detailed instructions to reproduce it step by step and a way to get the reproduction materials necessary.  This close to RC, we no longer have time to spend a few days going up and back to understand the bug and then realize we don’t even have the payware ACF we need!

There are two features that will ship whenever they ship, e.g. in 1030 if they’re ready, or we’ll cut a 10.31 in a week or two if we have to:

1. Oculus Rift support.  There is a new SDK for the Rift, and it’s not backward compatible with the old one, so Philipp has some coding to do there.  I think we’re going to enable the Rift as soon as it’s done, but given how few sets are already sold (since they’re just low-res developer previews right now) we’re not going to halt 1030; we’ll cut a quick bug fix release if we need to.
2. Cylindrical Projection.  Austin reworked cylindrical projection for pro users for 1030; if we find we need to get a few more refinements into the warping code, that may go into a bug fix release too.

My goal is an RC by the weekend, but that’s more of an aspiration than a promise.

* If you like a bug and don’t want it fixed, perhaps emailing me directly or posting it on the forum is the best thing you can do!

Fixing X-Plane 10.30 Performance

This week I received my new PC, set it up, and was able to actually look at 10.30 performance problems with AMD hardware on the Catalyst 13-9 drivers.  With the actual “problematic” configuration (and a hard drive that wasn’t barfing up sectors like an infant with reflux) it only took about 15 minutes to find the actual problem.

Which was…face palm…a GL drawing code path that I had disabled during private testing (due to low performance) was actually not disabled at all due to a lack of parenthesis.  So I’m an idiot.

What’s interesting is the relative effects of this code path on multiple platforms.

• On OS X, this code path runs at pretty much full speed, so having it in the beta didn’t reveal any actual problems.  Hence while my PC was dead, my few attempts to reproduce the reported behavior on my Mac totally failed.
• The code path is slower on both Nvidia 331.88 drivers and AMD 14-4 drivers.  But…1030’s basic performance is faster than 10.25, so the two canceled out, yielding somewhat reasonable looking fps tests.  That’s why when I asked Philipp to try to reproduce things on an NVidia windows box, he told me things didn’t look broken.
• On the AMD 13-9 drivers, the code path is absolute death.  I believe that this is where the majority of bug reports were coming from.
• The actual total performance win from 10.25 to 10.30 appears to be quite a bit bigger on NVidia hardware than AMD hardware. For some reason, one of the modernization changes in 1030 is more important for NV than AMD.  I don’t know what the specific changes that are helping are – the goal of 1030 is to simply use the best code paths we can find.

Here’s an excerpt from my performance testing numbers – this is with fps test 3 with the 747 replay.  The hardware for the test is a GeForce 680 GTX and a Radeon HD 7970.  Expect better fps with the NV card – it’s a generation newer than the AMD card.  Numbers are framerate, probably good +/- 1 fps.

Driver/hw   10.25       10.30b5      10.30b6 (fixed)
NV 331.88   34          34           42
AMD 13-9    28          11           30
AMD 14-4    27          28           29

So you can see here that on better functioning drivers (331.88, 14-4) even with the performance problems of 10.30b5 the net change in framerate from 10.25 was positive. But once I remove the stupid code path, things get a lot better.

That 11 fps in 10.30b5 with 13-9 is the bad code path behaving badly.  That will be fixed in the next beta.

I am hoping that this code path is also responsible for the remaining “low fps in local map” reports, but I haven’t confirmed that yet.

Your Mileage Will Vary

One of the problems with performance testing is that X-Plane’s content varies by add-ons, so how much the problems with 10.30b5 affect you will have a lot to do with which aircraft you fly. This is why some users already thought 10.30 was a win over 10.25 performance wise, and others thought the build was totally unusable.  How much speedup you get will depend on the kind of content you use.

AMD: What Drivers Should I Use

The 14-4 drivers are slightly slower than the 13-9 drivers for either shipping X-Plane 10.25 or 10.30b6 (not yet released).  I believe this represents a fundamental (small) loss of performance with the newer drivers. The loss is actually smaller than it looks if you view the raw fps numbers with fractions, and the precision for the tests isn’t better than 1 fps.

If you can run the 14-4 drivers you should – they fix a bug in sRGB blending with HDR that makes the 13-9 drivers look rather weird with HDR mode.

If you are stuck on the 13-9 drivers because you have old hardware (the oldest GPU the new drivers support is the HD 6000 series – so if you have the HD 5870 or older you need the older drivers) at least 10.30b6 will improve performance.

Clouds Are Their Own Performance Problem

The other big performance problem that I am aware of besides this one is that clouds use a lot more fill rate than they used to.  This is my next bug to try to fix – it is totally separate from this one, and affects all platforms.  My goal is to get the clouds to use the same fill rate as they did in 10.25.  That work isn’t done yet, but at least now I have hardware to work on.

Some Notes From Beta 1

The next X-Plane 10.30 beta should be out tonight (or maybe tomorrow morning); the release notes are already updated with the latest fixes, and we’re just waiting on upload.

There were some “boneheaded” bugs in beta 1 – things that just didn’t work, should have worked, and were easily fixed; we have addressed all of those that were reported for beta 2.  Despite about half a dozen boneheaded bugs, I actually think the quality of public beta 1 – relative to the amount of source change – was pretty strong.  Remember that 10.30 has perhaps 2x the amount of code change of a normal major patch, so if there isn’t too much nuclear then I think we’re doing okay.

Aircraft Authors: please test beta 2!  I think we may be able to keep 10.30’s beta period down to 4 weeks, which is a lot shorter than normal.  And the remaining bugs are all in clouds and performance, so please do not wait to test your aircraft.  Eugeny contacted me on problems with the A320 Neo on beta 1, and we have X-Plane fixed for beta 2. Don’t wait!

Performance: there have been all sorts of performance comments from users.  I think there are three separate things going on:

1. For general use, X-Plane 10.30 is faster than 10.25.
2. X-Plane 10.30’s clouds are definitely slower/more fill rate intensive than 10.25.
3. There’s something wrong with some combination of AMD drivers and 10.30 that can cause total framerate death.

I am investigating both points 2 and 3, but neither are fixed for beta 2; we wanted to get some of the stupid things fixed ASAP (e.g. the ATIS not working).

To make matters worse, my PC’s SATA controller seems to have finally lost it’s mind. Philipp had to cut the Windows build, and this has halted me from investigating AMD performance problems.  (In my initial tests I couldn’t repro anything but now I’m stuck.)

Clouds: The other area of bug fixing on my plate that simply isn’t addressed in beta 2 is the cloud locations; several users have correctly reported that the bases of the clouds are simply not where they expect.  Since there is an unfortunate link between rendering settings and fixing this bug (if the puffs we use to visualize clouds change, it effectively moves the cloud bases) I have to fix both cloud performance and cloud bases at the same time.

The expected behavior for the clouds, I think, is that if you set a stratus layer with no storms and fly an ILS, you should “break out” (that is, make visual contact with the runway environment) right around the elevation set in the weather screen.  (This is not what is happening now, which is why there is an open bug.)  If you set more ‘creative’ weather, X-Plane will start to vary things and you won’t get “reliable” break-outs.

METARs: One last note on weather: if you find that X-Plane has misinterpreted a METAR (and my sympathy is with X-Plane, because METARs just contain the most random stuff in them sometimes), then please be sure to include in your bug report:

1. The actual METAR.rwx file!  No METAR file, no way we can possibly debug.  To everyone who has sent METAR files, thank you!
2. The location of the aircraft.  A fairly precise latitude/longitude, or a FIX or navaid that you are over is good.  The easiest is when you are at an airport and can send us an ICAO code.
3. A description of what weather you actually got.

X-Plane 10.30 does have a new METAR parser so we’re trying to catch the bugs.

Airports and DSFs: we are looking at including both some fixed DSFs and additional 3-d airports in 10.30.  Both depend on the tech being ready; the airport gateway is in the deployment phase and if it goes smoothly, we could be ready to post airports.  I’m still trying to get to the bottom of DSF bugs, but I am close.  If either of these content updates miss 10.30, we’ll release them in a small patch as soon as possible.

Physics and Magic

Perhaps I am too optimistic, but: rendering solid, opaque objects that look realistic in real-time (read: for games and rendering engines) is rapidly becoming a solved problem. I look at the bleeding edge of the top-tier shooters as a preview of the direction for the field of real-time rendering*, and for the last year, physically based rendering (PBR) has been front and center.

I’ll write more about PBR in another post, but the short version is that, in PBR, the equations for light interacting with materials in a rendering engine are modeled more closely on real-world physical equations; the goal is something that is very accurate to real world optics, and produces realistic results over its entire range of inputs.

PBR is made possible by increased computing power on the GPU.  (And here when I say computing power I really do mean computing, or ALU power; PBR requires more math per pixel to calculate for a wider range of optical effects, often with more expensive but better approximations of the underlying optics.)  We now have big enough GPUs to do “real physics” (well, real-ish) when it comes to light and solid objects.  This is a huge step forward when you consider that a decade ago, calculating lighting equations per-pixel was marginal and had to be done in the wrong color space for speed.**

So for solid objects like airplane fuselages, runways, buildings, baggage trucks, etc., we have physics.

Then there are clouds, and the air, and rain and fog.  For those we have magic.

By magic, I do not mean something truly magical, something outside the realm of what is possible with computer graphics.  Rather, I mean the definition a professional magician like Penn Jilette might use: trickery.  To quote Penn***:

The only secret in magic, there’s only one, and that is, the secret must be ugly.  You cannot have a beautiful secret…So, in magic, what you want is an idea that is not beautiful…If I have to say, he’s lying about that, and there’s gaffer’s tape over behind there, and, ah, they’re not actually telling you the exact truth here, and it gets so…you don’t get an a-ha.

But there’s another side to magic: to make the trick look magical, the magician has to practice – the trick is a craft, and it can be finely honed.  That’s why, before Teller does the Red Ball trick, Penn has to tell you how it’s done; you can’t appreciate the workmanship if you don’t know how much workmanship went in.  The secret of the trick may be ugly, but the craftsmanship can be beautiful.****

Clouds, fog, atmospheric scattering – in X-Plane (and pretty much all rendering engines), these are not physics, they are magic, meaning they are tricks.  The sky is a hemisphere around your head, like in The Truman Show.  The color of the clouds are just painted onto textures with photoshop.

It’s trickery, but I feel no shame in these things, because the real physics are approximately a gajllion times more expensive than we can possibly afford on today’s hardware.  A single square meter of ground (on an overcast day) might be lit by 10^18 photons per second – a lot more when the sun comes out, and each of those photons can take a different path through the atmosphere, bouncing off molecules multiple times before arriving at a surface.  The actual optics of clouds and the atmosphere is incredibly complex.

The hard work and craft of the trickery is in making all of the tricks work together to make a unified effect, and this is where some of my time has been spent over the last few days.  It is frustrating work because there isn’t a right answer (the way there might be in the programming of physics equations); instead it is a series of tricks and compromises that try to make an illusion as convincing as possible given limited resources.

Whenever I describe the code behind atmospheric scattering, clouds and fog to Chris, he is always horrified by the complexity, by the layers of tricks.  But such complexity matches Teller’s the definition of a functional magic trick:

Make the secret a lot more trouble than the trick seems worth. You will be fooled by a trick if it involves more time, money and practice than you (or any other sane onlooker) would be willing to invest.

If clouds and scattering weren’t so much trouble, they wouldn’t be magic.

https://www.youtube.com/watch?v=FFq-fn-nSZI

* The field of real-time rendering includes computer programs that produce graphics at high framerates, like games and CAD programs.  This contrasts it with the movie industry, where they can spend two years with a giant render farm to produce 90 minutes of final output.

** In 2004, we had the GeForce FX series and the Radeon 9700.  The 9700 was insanely powerful for its time, but it only had a 24-bit internal floating point path, a 96 instruction shader limit, etc.; you could do per-pixel calculations but burning shader instructions to work in linear space wasn’t on anyone’s mind.  The Geforce FX series was significantly less powerful.  This chapter wasn’t published until 2007 – 3 years later.

*** The link is to a Radio Lab story about a man who tries to understand how his grandparents performed a mentalism act on the radio – it’s a good listen!

**** In Fool Us, many times Penn and Teller knows exactly how the trick was done, but still praise the magician for their presentation – because they know how the trick was done, they know how hard it is to do it well.  Within the show, they seem to have a preference for tricks that are not easy and require skill to perform.