Announcements on the plethora of betas that went up this weekend.
Edit: beta 5 is temporarily offline while we resolve a missing scenery texture. So you’ll get beta 4 if you update.
X-Plane 10.04 beta 5 is now available. Release notes here. I think we will be winding down the 10.04 patch in the next week and going on to 10.05. There are a number of cosmetic bugs that I hope to get fixed shortly; it looks like they’ll have to go into 10.05 and not 10.04.
What’s the logic behind this patching? Basically we’re going to keep doing small patches until we get a certain set of bugs fixed – bugs that we think are important and that we can fix without tearing the sim to bits. The patch runs sometimes have to be closed off to burn new DVD masters* (this is the case with 10.04). In all cases, if you don’t want to deal with beta chaos, you can ignore the betas and get a new patch every few weeks with a little more performance and a few more bugs fixed.
I spent some time yesterday retuning cloud performance and the effect of the cloud rendering setting. Besides trying to find a better performance-quality trade-off point, the old slider had way too much range on it. It defaulted to 50%, but that 50% point really represented “a good setting for a high end gaming desktop”; going past that would put you into the range of “this isn’t ever a good idea, but the engine can do this without crashing”. The side effect was that running at 25% clouds (a very reasonable setting) would feel demoralizing because the setting scale was so vast.
The new scale marks a “100%” point at about where I would put it for high quality if you’re just sitting on GPU power (e.g. small screen, nice GPU). You might still want to turn it down for performance reasons. The range from 100%-150% is highly experimental land…if you want to turn it up to eleven, we’re not going to stop you.
WorldEditor Developer Preview
There is now a binary build of WED 1.2 (download here). This is a developer preview: WED is still in environment and doesn’t meet the criteria for beta software. Basically: do feel free to poke around with it, but don’t use it on your real work. Don’t edit your scenery packs with it – make a copy first. WED 1.2 has not been tested enough to use for production yet!
The main feature of interest is ATC taxiway editing: WED 1.2 contains complete support for taxiway networks, as well as “flow” information to control the direction of operations. A number of weird ATC bugs actually stem from the automatically generated taxi layouts that X-Plane produces when the apt.dat layout contains no taxi information. By providing a custom layout, you can get the AI planes to behave quite a bit better.
* We periodically re-burn the disc 1 master to put the latest sim on it whenever we are preparing new DVD inventory. If you have an old DVD, please do not panic! Once the updater is run, the sim looks the same no matter which DVD you have.
Chris and I bashed at the developer site (that is, this site today); sorry about any dead URLs – things should be more stable now.
I have (finally) posted a new build of XPTools; there are two items of note:
- The new build of DSFTool, which can read and write X-Plane 10’s DSF raster data layers.
- The new DDSTool can write gamma 2.2 DDS files for X-Plane 10. This should provide better color fidelity for content that targets only X-Plane 10.
We’ve been making steady forward progress with the scenery tools over the last few weeks; finally there is some stuff to see.
Moving the Website
First: we’re moving the scenery tools into WordPress. All of the new tools and docs are hosted at developer.x-plane.com; over the next few weeks I’ll try to make the theme less bare-bones. As I update documentation to version 10, I will move it here.
What about the Wiki? We are retiring it as our content management system. MediaWiki makes everything possible, but it makes nothing easy. Well, that’s not true – it makes it easy to collect link spam.
We can possibly keep the Wiki open on a smaller scale for developers, but my preference is strongly in favor of a mailing list.
New Tools
The first binary tools release is a final version of WED 1.1. If you have been using a WED 1.1 beta, get the final one – it fixes some bugs and adds control over object density in custom scenery.
I will have binary releases of the base scenery tools pack including DSFTool and a WED 1.2 preview over the next few days.
New Documentation
The DSF specifications are updated – see here. The DSFTool binary will let you create version 10-style DSFs. I will continue to update the documentation over the next few weeks.
New Source Code Repository
The scenery tools source code is now natively hosted in GIT – previously it was hosted in CVS and was bridge to GIT for public preview; the bridge broke down during v10 development a few months ago.
Chris and I are now using GIT internally, putting us in sync with the Linux community; if you want to work on the scenery tools or modify them beyond patching, you can clone the repository and merge in changes.
I originally meant to write a post about deprecation and feature evolution a while ago when I accidentally (and temporarily) broke the way textures are located in scenery packs and put it back. In X-Plane 10.04 betas we are now fighting with apt.dat validation, so the topic is perhaps again relevant.
X-Plane is continuously patched. We put out about 4 major patches a year plus numerous bug fixes for the life of the product. One of the implications of this is that even if we are planning to drop a feature, we can’t just go from “it’s there” to “it’s gone” in a patch, because users get grumpy if their content doesn’t work after a free (and hard to avoid) update. No one wants to pick between getting the latest perf tuning and having working content.
The way we try to cope with this is by easing in sim changes that are necessary in the long term but disruptive in the short term. We try to make the stop light go yellow before it goes red. Here are some examples:
- If we are going to drop a file format, we can set the sim to give one of those nasty “scenery pack” warnings with log info before we drop the file format. Now the developer knows that the file format is going away, but the pack isn’t instantly broken for users.
- If we are going to drop an airplane feature, we can support it in the sim but drop it from the Plane-Maker UI (or new file format). Now authors can’t proceed forward on tech without addressing the change, but old content still works.
- If we find a case of invalid data, we can log it as invalid before we simply refuse to load it.
I originally meant to post this back in 10.02 betas when I removed the code to handle the old (X-Plane 7) texture file location system and discovered that a bunch of otherwise current scenery packs still use it. The new (art asset relative) system has been available since version 8 (that’d be 7 years) but X-Plane never warned that obsolete packages were doing something that was going away, so I put the code back with a warning.
We’re running into another version of this with custom airports that put ramp starts, windsocks and beacons in strange places. I suspect that some of this represents accidental bad data and some of it represents intentional authoring – attempts to use X-plane in a manner that we didn’t quite expect. The need for data validation comes from the ATC system – with ATC and AI planes using apt.dat, weird data causes significantly more problems than it used to.
I don’t know what the final outcome will be – we’re still looking at some of the apt.dat files that fail. But my expectation is that:
- When we’re done, apt.dat files that were flyable in 10.03 will be flyable in 10.04.
- Some apt.dat files from 10.03 will give off warnings in 10.04 if they do things that we consider to be illegal or need to be implemented in a different manner.
X-Plane 10.04 beta 2 is out now. Release notes here; like all betas, you’ll need to run the updater and click “get betas” to see this – it won’t auto-update until we go final. Report bugs here. Do not post bug reports on the blog.
A Temporary Outage
The beta was actually ready last night, but iWeb, one of our hosting providers, had a fairly major outage in the data center that the update servers are hosted in. The servers are now back on the net, so updates should work normally.
Our overall strategy for updates is to spread out the update pool over a few providers so that temporary outages only reduce capacity, not availability. In the short term, however, we have a lot of capacity at iWeb because their pricing is really aggressive and we had to push a lot of bits for the X-Plane 10.0 roll-out. This is the first outage they’ve ever had; if we see continued reliability problems we’ll get more aggressive about rebuilding our demo/update mirror.
Controlling Scalability
With 10.04 beta 2, the attached objects on the parking garage facade now shows more or less cars based on the object rendering setting. This means that if you have a computer that cannot instance and you run with lower object density, the KSEA is going to be a lot less expensive than it used to be. The facade engine also allows more control over object placement, which Tom was able to use to further reduce overall object count.
Our plan is to include this kind of object density control on all parts of the airport lego brick library so that the amount of “ramp clutter” is variable by user settings. Chris and I were in the terminal at KCLT waiting for a flight and looking out across the ramp, one thing was clear: there’s a ton of crap floating around the ramp at a major airport. With variable density objects, authors can put down major “mini-scenes” (e.g. a jetway plus docking position) and we will be able to scale from the minimum objects to make the sim functional all the way up to a ton of clutter for machines with good instancing performance.
What’s On the White Board
I have a white board on my wall with some of the highest priority bugs scribbled all over the place; I like it because it gives me an immediate view of what can’t get ignored, and when I am on the phone I can simply write down an issue that someone brings up. So far the white board issues tend to be in a few categories: crashes and serious bugs, performance, authoring platform features, and rendering artifacts.
With 10.04 beta 2 I finally had a chance to start killing off artifacts; the taxi lines should (previously they’d get all bent out of shape when they went around sharp curves) and the water shader is better. (The water shader still isn’t done, but there should be no more hard lines in it.)
At this point rendering errors and performance still dominate the board and my next few weeks of coding.
I spent a few hours last night working on X-Plane’s facade engine – that is, the part of X-Plane that builds buildings, fences and structures from their shape traced on the ground. Besides fixing some v9 compatibility bugs, I had a chance to add in a few last features that didn’t quite make the 10.00 cut.
I used Tom’s “classic”-style parking garage (you’ve seen it – it’s the one used at KSEA in the demo area!) to test some of my work; some of the work on the facade engine illustrates three different ways to optimize a program.
Be More Efficient
The first way to optimize is to find ways to accomplish the same work in less time – that is, to write more efficient code. This is certainly the least risky form of optimization: there’s no trade-offs, you just get more for your hardware dollar.
In the case of Tom’s parking garage, after putting geometry in place for the roof of each floor (to fix a rendering bug) I noticed that we were building the mesh for each floor twice – once for the roof of the level and again (but facing in the opposite direction) for the floor of the next level. I added some code to the engine to recycle the mesh, but create duplicate indices; now two-sided roofs in facades use about half the vertex count.
There are still a number of places in X-Plane 10 where we can be more efficient. But rewriting code to be better takes time; these performance changes will come out incrementally, and any one change may not be that noticeable. What is important though is the net effect of improving efficiency over and over again.
Cheat!
This sounds negative, but when it comes to computer graphics, cheating is preferable. Clouds behind the panel? Just cheat – don’t draw them! Trees too far away? Use a billboard! No airplanes in the game? Use a painting for the sky and don’t bother with 3-d clouds. What I am referring to here is finding ways to deliver the appearance of eye popping graphics for significantly less work. This kind of optimization is critical to making a competitive rendering engine.
In the case of Tom’s garage, the facade engine puts the same number of cars on every level of the garage. This is hugely wasteful; while you might want cars on the roof, you don’t need very many on the lower levels. The lower levels are only visible from the side, and from the side even a few cars are enough to create the appearance of a non-empty garage.
So I am adding to the facade engine the ability to specify mid-level objects and roof objects separately; Tom can use that to “thin out” the objects in the garage. The result will be a big reduction in OBJ count for almost no visual quality loss.
I think there’s a lot of fine tuning we can do like this to make sure that we spend our hardware budget only on things that really matter most.
Scale Down
This isn’t really a type of optimization, but it’s important to make sure that the rendering engine can throw visual detail out to meet the requirements of older computers. One of the problems with X-Plane 10 is that some parts of the rendering engine only have one mode: “maxed out”.
10.04 beta 2 will allow facades to specify attached objects (like the cars on the parking garage roof) that decrease in density with the rendering settings. The result will be a full garage when objects are on “insane” but a mostly empty one with just a few cars when objects are on “default”.
Tom and I are working to put this kind of object density control in all of Seattle (as well as the airport library it uses and the rendering engine components that the library uses); the result should be a Seattle demo area that becomes significantly cheaper when the rendering settings are dialed back just a bit.
There are three nasty bugs haunting X-Plane 10 that are on my plate right now; these are serious bugs that pretty much ruin the X-Plane experience for those who hit them. They are marked in the bug base as “hair on fire” – the highest priority we have for a bug. (Yes, the bug base literally has a priority category called “hair on fire”!)
Every morning I have to ask “can I close any of my hair on fire” bugs, but these three have been nasty enough that a status update is warranted, even before we have a fix.
Bug 1: Poor Performance with ATI Hardware on Windows
A code path used by the cars and clouds is stalling in the ATI driver on Windows. I don’t know why this is, but we are trying to follow up with ATI directly on this. The result is pretty unfortunate: if you have any cars (even “siberian”) or any clouds (even scattered) framerate absolutely tanks.
As it turns out, if you set weather to clear and turn off cars completely, ATI performance on Windows is pretty good! Obviously that’s not a useful work-around, but if you have ATI hardware and are wondering what kind of fps you’ll get once we fix this, those are the two features to turn off. (I suspect rain may hit this code path too.)
As with all driver-related problems, I must point out: I do not know if it’s our fault or ATI’s, and the party at fault may not be the party to fix it. Apps work around driver bugs and drivers work around illegal app behavior all the time…that’s just how it goes. Philosophically, the OpenGL spec doesn’t require any particular API calls to be fast, so you can’t really call a performance bug a bug at all. It’s ATI’s job to make the card really fast and my job to speculate which subset of OpenGL will cause the card to be its fastest.
I have a Radeon 7970 and will write up some of my performance findings later. Unfortunately with the clouds affected by the problem code path, it’s hard to see how the card will really do.
(OpenGL nerds: the problem is that glBufferData with an in-flight VBO and a NULL pointer is stalling. This particular use of glBufferData is meant to tell the GPU that we want a new fresh buffer, with the old buffer being asynchronously discarded once the GPU has completed drawing. This “orphan the buffer” pattern is the standard way to do high performance vertex streaming in OpenGL prior to glMapBufferRange.)
Bug 2: Hang on Start with NVidia Hardware (Windows)
A small percentage of Windows users get a hang on start with NVidia hardware. We’ve had this bug for years, but it’s more of a problem in version 10, where command-line work-aronuds are not available. We have identified four third party programs (Omnipage, Matrox Powerdesk, Window Blinds, and TeamViewer) that all seem to cause the problem at least some of the time, but some users have this problem with no obvious cause. There isn’t a hardware or driver pattern to it either.
Until recently, we could never reproduce this bug in-house; it was only through the patience of users that we were able to characterize exactly what was going wrong via copious logging. But a few days ago we got a break: a user reported Window Blinds to be hanging X-Plane, and unlike before, Window Blinds hangs our machines up too! (That’s a huge step forward in getting a bug fixed.)
At this point we’re working directly with NVidia to try to find out what’s going on, as the hang occurs inside their driver code. The same rant applies here as above: until we know exactly what’s going on, we can’t say whether this is a driver bug or an app bug.
Bug 3: Crashes Near Munich on Linux
Linux users see crashes near EDDM with memory exhaustion, even at very low rendering settings. This bug appears to be Linux specific and we do now know what’s going on.
For reasons that I cannot yet fathom, the math deep inside the ATC airport taxi layout auto-generator goes haywire on Linux, resulting in an infinite loop that allocates memory, eventually crashing X-Plane.
What’s infuriating about this bug is that it is a Heisenbug: all of my attempts to diagnose or “prove” what is going on causes the math to stabilize and the app to function. The bug does not appear when the optimizer is off; it certainly looks like an optimization bug but I hate to call foul on the compiler guys without a ton of evidence. It’s just too easy to write C++ floating point code with sharp edges.
I don’t have a good time frame for the two video card related bugs; I consider us lucky to be able to communicate with ATI and NVidia at all. I don’t have insight into their development process and if I did, it would be covered under our NDA. The only thing I can say is that these are top priority bugs.
The Linux bug is still in my court; I last put it down in a state of GCc-induced rage. I believe I will be able to get it fixed some time during the 10.04 beta process.
Final unrelated thought: all of the GPUs in my office, listed in performance order…
GeForce 5200 FX, GeForce 6200, Radeon 9600 XT, Radeon 9700, Radeon X1600M, GeForce 8800 GT, Radeon HD 4870, GeForce 285 GTX, Radeon HD 6950, GeForce 580 GTX, Radeon HD 7970
A question came up in the comments to my previous post:
Any suggestions on the best way to edit DDS files on a Mac?
The answer is simple: do not edit DDS files!!!
The problem is simple: DDS is a lossy compressed format. Every time you go from an uncompressed/losslessly compressed format (like PNG, TIFF, or BMP) to a lossy format (like DDS or JPEG) you are going to lose a little bit of image quality. So if you re-open your DDS, edit it, and resave, your graphics program is going to make a round trip and reduce quality.
Therefore the work flow should be:
- Edit the original PNG or photoshop master file.
- When done, re-export to DDS.
So each time you work, you work from a lossless or uncompressed master (like PNG or a gimp/photoshop file) and then re-export to DDS.
Thanks to Captain Peter Hager for sending me his A340 as a nice test case for texture compression. Peter (like many aircraft authors) has never been very happy with the artifacts introduced by DXT compression.
First the basics: DXT1-5 are a series of texture compression formats. They store the same number of pixels in a smaller number of bytes via clever encodings of the pixels.
When your image has no alpha channel, the savings from DXT1 compression are 8:1. (Modern graphics cards store RGB images in 32 bits for alignment, so the 4-bits-per-pixel DXT encoding is an 8:1 savings, not the 6:1 you’d expect on paper.) RGBA images take 8 bits per pixel, for a 4:1 savings. That’s as much as reducing the image size by a factor of 2 in each dimension.
DDS is an image format that contains a compressed image. DDS files load faster (because the image is already compressed), look better (because the image is compressed using a slower, higher quality process before sim load) and let you pick the DXT algorithm choice. (In some cases, the choice between DXT3 or DXT5 matters!) However, once an image is compressed into a DDS file, the pixels lost in compression never come back – DXT is a lossy compression format! (This is just like saving a JPG as a PNG – if the JPG quality was too low, the PNG will contain the same blocky artifacts that the JPG had.)
Compression is useful for more than just saving VRAM – because the amount of data needed to texture a given pixel is smaller (4 bits per pixel instead of 32) compressed textures can speed up framerate if the memory bus between the GPU and its own VRAM is congested.
Now let’s look at some pictures.
This is the Swiss Air livery at extreme res, uncompressed, sourced from a PNG. This is our reference image for how good the original image can look before we start trading off quality for VRAM.
This is the same image, extreme res, texture compression on, sourced from a PNG. It just looks awful. The graphics card has done the compression, and it has done a fast job, not a high quality job. (In the driver’s defense, X-Plane asks for a fast compression rather than a high quality one to avoid slowing the sim down.)
This is also compressed!!! But in this case the livery was pre-compressed using XGrinder/DDSTool (which in turn uses libSquish). DDSTool sets libsquish for maximum quality and then takes quite a while to grind each texture, but the results are better use of those compressed bits. (DXT is a lossy compression, so more compute time can find a better use of the finite quality available in the compressed texture.)
Finally, this is the uncompressed texture at “very high” res – that is, taken down one resolution level. Compare this image to the compressed one above. For an alpha-blended texture, the trade-off is one “size” increment (2x smaller on both sides for 4x VRAM savings) vs. compression. I submit that the compressed version is a better way to save VRAM than down-sizing.
Have Your Cake and Eat It in X-Plane 10
In X-Plane 9 you had to make a trade-off: ship a compressed image in a DDS file and have the very best look be extreme res compressed or ship a PNG file, and have the texture look terrible when texture compression is on. You can see from the above images what the options looked like.
In X-Plane 10 you can do both. If you ship a DDS and a PNG file, X-Plane will load the PNG file when texture compression is off and the DDS file when it is on. The result is a pre-compressed texture for users who need texture compression and an uncompressed texture for those who can afford the VRAM.
VRAM and Memory
One last note on texture resolution for users: most drivers use virtual address space to load textures, and thus texture memory eats into X-Plane’s 2/3/4 GB address space limit. Therefore with high VRAM cards (e.g. a 3 GB card) it probably isn’t possible to absolutely max out texture res until we go 64 bit. For example, on the Mac (where the driver uses a lot of address space and we only get 3.5 GB) running the sim on extreme + uncompressed texture res will almost always run you out of memory.
One thing I have noticed from looking at user’s settings and our own machines is that the textures resolution selector goes in pretty big jumps. Each resolution change cuts VRAM by 75%, so if you are over VRAM (and having performance problems because of that), the next setting down will probably leave VRAM unused.
So I have a todo item to look at putting incremental texture resolutions in that derez the scenery but not the airplane, for example, to hit some in-between points. In that context, we may be able to support mixed compression, where some airplane elements remain uncompressed, but the scenery is compressed.
The recommended practice for developers remains the same:
- If your textures look better without compression, ship DDS and PNG files, and pre-compress using XGrinder/DDSTool.
- If your textures don’t improve much by running uncompressed, ship only DDS files, pre-compressed with XGrinder/DDSTool.
One Exception: Orthophotos
There is one exception to the X-Plane 10 practice of shipping both DDS and PNGs: if you are using orthophotos via either a .pol or .ter with the LOAD_CENTER directive, use DDS and only DDS.
LOAD_CENTER causes your images to be reloaded at high or low res depending on the distance from the user’s aircraft to the image; this can save a lot of VRAM by reducing resolution where the user cannot see the detail. Because DDS contains pre-resized images, it is much more efficient than PNG for load center; I therefore recommend DDS-only for orthophotos.
I’ve been meaning to write a post about the difference between general purpose flight simulators and AAA first person shooters for a while; a particular feature in X-Plane 10.04 brings some of these differences to light.
A typical high-end first person shooter is a closed game; the art content and the rendering engine ship together, and the art content is authored specifically for (or repurposed for) that particular rendering engine. The content is also completely available ahead of time; it can be authored without worrying about rendering engine changes or the addition of third party add-ons. Similarly if a rendering engine change requires revising all art content, this is at least possible, since all art assets live in house.
Compare this to a general purpose flight simulator. The rendering engine and art content may be revised on different schedules. The artists building third party content have limited access to the developers writing the rendering engine, and the rendering engine acts like more of a platform than a closed product.
Because X-Plane’s rendering engine is a platform, rather than part of a closed product, we try to keep optimization and performance work generic. When possible, we put in optimizations that act automatically without author participation. When we do have optimization advice, it tends to be generic advice that applies over a wide range of rendering engine revisions. (For example, using less OBJ ATTributes and atlasing many small textures into one big texture have been good advice for at least the last seven years.)
Sometimes, however, it becomes necessary to put in authoring features that are somewhat specific to the rendering engine and optimization. This is what has happened with cockpit pre-filling.
Before we go any further, you may want to read about pre-filling here. (One thing I am trying to do is to put the specific tutorials and documentation into permanent articles and not blog posts that are lost in the archives after a few months.) Basically when we pre-fill, we draw part of the airplane early on to mask out clouds, saving fill rate on the GPU, improving framerate.*
For the 2-d panel, we put in pre-fill in X-Plane; no artist intervention is needed. But for the 3-d cockpit, we can’t easily pre-fill without you setting some check-boxes on your airplanes.
The problem with the 3-d cockpit is that it is, in its entirety, rather expensive to draw. So we can’t just draw the entire 3-d airplane (for the purpose of masking clouds) or we’d be burning a possibly large amount of CPU to save a possibly large amount of fillrate. The right trade-off depends on the actual plane, and X-Plane can’t figure out automatically what the right thing to do is.
Hence the pre-fill check-box in X-Plane 10.04. With the pre-fill check box, you (the author) mark your content for optimization to hit the right balance of CPU and fill rate savings. (See the articles linked above for guidelines on how to set the flags sanely.)
Fortunately if the check boxes are set wrong, the worst thing you get is poor frame-rate (not crashing). But this is a rare case where third party authors can (and have to) input tuning data directly into X-Plane.
(And for concerned users: if an airplane doesn’t have the pre-fill check-boxes set up, the performance is the same as 10.03 – there’s no loss, this is just an opt-in optimization.)
* If you are asking at this point “why didn’t you guys do this years ago, you idiots?!?” the answer is: because it never mattered before. X-Plane 9 simply doesn’t use much fill rate, so it never made sense to spend CPU time to save fill rate.
