I’m listening to Marco Arment and Dan Benjamin debate whether a user setting could have avoided the incident where a man’s iPhone rang during a concert. I also just finished reading the Steve Jobs biography. It got me thinking about the trade-offs between a closed non-configurable system and an open, configurable one. This comes up in an X-plane context every time we poke at the cloud rendering settings and someone asks whether we can put in more settings for additional configuration. What trade-off is right for X-Plane?
Chris suggested an analogy to me a few weeks ago that I think is pretty good: tuning the rendering engine in X-Plane should be like tuning the timing in your car. It should be possible, if you are a serious expert and enthusiast, to change the tuning (and if doing so wrecks your engine, well, you were asking for it by opening up the engine), but it shouldn’t be easy. Regular users of the car expect the manufacturer to have a lot more expertise, and to set the timing to an optimal setting up front.
I think the same thing goes for X-Plane. I should use as much expertise as I have to maximize efficiency no matter what trade-off is being made; at no point should dialing in a setting result in stupid behavior inside the engine. It’s up to me to know what can be inefficient and avoid that happening.
So: the rendering settings are going to remain as simple as possible, and any time we can make them more simple, less likely to be set up wrong, or any time we can realign them to be better aligned with user constructs (“more houses”) instead of internal constructs (“higher instancing density ratio!”) we’ll take that win. It should not be the user’s job to tune the engine, and any time that is necessary, that’s a failure by me to pre-tune the sim, not a feature.
I don’t think we are even close to the kind of rendering settings that are really useful to non-expert users; one of the frustrations with performance tuning over the last few months has been the number of times a user has sent me rendering settings and it was clear that the settings were inefficient for the given hardware.
The answer is not to try to make every user into an expert in the OpenGL pipeline; the answer is to change the settings so that users don’t have to be experts. This is not easy to do; the reason it’s not done yet is that we will have to create new code and do real hard work to make the interface simple. One theme that both Steve Jobs and Jonathan Ive describe when discussing their design philosophy is that, to make something simple in its final product, you have to dig down and master the complexity that you want to get rid of. To make the rendering settings simple but still powerful, we are going to have to solve these performance-efficiency-hardware problems ourselves and encapsulate that knowledge in code, and that’s not trivial.
Underneath the settings are a whole pile of art controls, and if you really want to, you’re more than welcome to enter any value you want in there. If you discover that you’re better at tuning the engine than I am, I will be very happy to hear what you did!
This attitude is not as extreme as the “Steve Jobs” approach; Apple devices often use custom screws to ensure that normal people can’t open their devices even if they want to. We are hiding implementation out of sight, but we are not locking that implementation at all. The settings file is plain text, the art controls are viewable with a publicy available plugin (DataRef Editor).
X-Plane has always been a hackable sim, and one of the things I used to enjoy back in X-Plane 6 was just looking at all the stuff in the resources folder; the raw textures were pretty amazing to view in themselves.
My goal here is not to take that away; only to make hacking and opening up the sim something you can opt into if you are curious, instead of something mandatory to get good performance.
Announcements on the plethora of betas that went up this weekend.
Edit: beta 5 is temporarily offline while we resolve a missing scenery texture. So you’ll get beta 4 if you update.
X-Plane 10.04 beta 5 is now available. Release notes here. I think we will be winding down the 10.04 patch in the next week and going on to 10.05. There are a number of cosmetic bugs that I hope to get fixed shortly; it looks like they’ll have to go into 10.05 and not 10.04.
What’s the logic behind this patching? Basically we’re going to keep doing small patches until we get a certain set of bugs fixed – bugs that we think are important and that we can fix without tearing the sim to bits. The patch runs sometimes have to be closed off to burn new DVD masters* (this is the case with 10.04). In all cases, if you don’t want to deal with beta chaos, you can ignore the betas and get a new patch every few weeks with a little more performance and a few more bugs fixed.
I spent some time yesterday retuning cloud performance and the effect of the cloud rendering setting. Besides trying to find a better performance-quality trade-off point, the old slider had way too much range on it. It defaulted to 50%, but that 50% point really represented “a good setting for a high end gaming desktop”; going past that would put you into the range of “this isn’t ever a good idea, but the engine can do this without crashing”. The side effect was that running at 25% clouds (a very reasonable setting) would feel demoralizing because the setting scale was so vast.
The new scale marks a “100%” point at about where I would put it for high quality if you’re just sitting on GPU power (e.g. small screen, nice GPU). You might still want to turn it down for performance reasons. The range from 100%-150% is highly experimental land…if you want to turn it up to eleven, we’re not going to stop you.
WorldEditor Developer Preview
There is now a binary build of WED 1.2 (download here). This is a developer preview: WED is still in environment and doesn’t meet the criteria for beta software. Basically: do feel free to poke around with it, but don’t use it on your real work. Don’t edit your scenery packs with it – make a copy first. WED 1.2 has not been tested enough to use for production yet!
The main feature of interest is ATC taxiway editing: WED 1.2 contains complete support for taxiway networks, as well as “flow” information to control the direction of operations. A number of weird ATC bugs actually stem from the automatically generated taxi layouts that X-Plane produces when the apt.dat layout contains no taxi information. By providing a custom layout, you can get the AI planes to behave quite a bit better.
* We periodically re-burn the disc 1 master to put the latest sim on it whenever we are preparing new DVD inventory. If you have an old DVD, please do not panic! Once the updater is run, the sim looks the same no matter which DVD you have.
Chris and I bashed at the developer site (that is, this site today); sorry about any dead URLs – things should be more stable now.
I have (finally) posted a new build of XPTools; there are two items of note:
- The new build of DSFTool, which can read and write X-Plane 10’s DSF raster data layers.
- The new DDSTool can write gamma 2.2 DDS files for X-Plane 10. This should provide better color fidelity for content that targets only X-Plane 10.
We’ve been making steady forward progress with the scenery tools over the last few weeks; finally there is some stuff to see.
Moving the Website
First: we’re moving the scenery tools into WordPress. All of the new tools and docs are hosted at developer.x-plane.com; over the next few weeks I’ll try to make the theme less bare-bones. As I update documentation to version 10, I will move it here.
What about the Wiki? We are retiring it as our content management system. MediaWiki makes everything possible, but it makes nothing easy. Well, that’s not true – it makes it easy to collect link spam.
We can possibly keep the Wiki open on a smaller scale for developers, but my preference is strongly in favor of a mailing list.
New Tools
The first binary tools release is a final version of WED 1.1. If you have been using a WED 1.1 beta, get the final one – it fixes some bugs and adds control over object density in custom scenery.
I will have binary releases of the base scenery tools pack including DSFTool and a WED 1.2 preview over the next few days.
New Documentation
The DSF specifications are updated – see here. The DSFTool binary will let you create version 10-style DSFs. I will continue to update the documentation over the next few weeks.
New Source Code Repository
The scenery tools source code is now natively hosted in GIT – previously it was hosted in CVS and was bridge to GIT for public preview; the bridge broke down during v10 development a few months ago.
Chris and I are now using GIT internally, putting us in sync with the Linux community; if you want to work on the scenery tools or modify them beyond patching, you can clone the repository and merge in changes.
10.04 Beta 4 was released last night. The changelog can be found here as usual. Remember, you have to run the updater with the box checked that tells it you WANT to participate in the beta testing in order to get this version. Bug reports go here!
Posted in News
by
Chris Serio |
I originally meant to write a post about deprecation and feature evolution a while ago when I accidentally (and temporarily) broke the way textures are located in scenery packs and put it back. In X-Plane 10.04 betas we are now fighting with apt.dat validation, so the topic is perhaps again relevant.
X-Plane is continuously patched. We put out about 4 major patches a year plus numerous bug fixes for the life of the product. One of the implications of this is that even if we are planning to drop a feature, we can’t just go from “it’s there” to “it’s gone” in a patch, because users get grumpy if their content doesn’t work after a free (and hard to avoid) update. No one wants to pick between getting the latest perf tuning and having working content.
The way we try to cope with this is by easing in sim changes that are necessary in the long term but disruptive in the short term. We try to make the stop light go yellow before it goes red. Here are some examples:
- If we are going to drop a file format, we can set the sim to give one of those nasty “scenery pack” warnings with log info before we drop the file format. Now the developer knows that the file format is going away, but the pack isn’t instantly broken for users.
- If we are going to drop an airplane feature, we can support it in the sim but drop it from the Plane-Maker UI (or new file format). Now authors can’t proceed forward on tech without addressing the change, but old content still works.
- If we find a case of invalid data, we can log it as invalid before we simply refuse to load it.
I originally meant to post this back in 10.02 betas when I removed the code to handle the old (X-Plane 7) texture file location system and discovered that a bunch of otherwise current scenery packs still use it. The new (art asset relative) system has been available since version 8 (that’d be 7 years) but X-Plane never warned that obsolete packages were doing something that was going away, so I put the code back with a warning.
We’re running into another version of this with custom airports that put ramp starts, windsocks and beacons in strange places. I suspect that some of this represents accidental bad data and some of it represents intentional authoring – attempts to use X-plane in a manner that we didn’t quite expect. The need for data validation comes from the ATC system – with ATC and AI planes using apt.dat, weird data causes significantly more problems than it used to.
I don’t know what the final outcome will be – we’re still looking at some of the apt.dat files that fail. But my expectation is that:
- When we’re done, apt.dat files that were flyable in 10.03 will be flyable in 10.04.
- Some apt.dat files from 10.03 will give off warnings in 10.04 if they do things that we consider to be illegal or need to be implemented in a different manner.
X-Plane 10.04 beta 2 is out now. Release notes here; like all betas, you’ll need to run the updater and click “get betas” to see this – it won’t auto-update until we go final. Report bugs here. Do not post bug reports on the blog.
A Temporary Outage
The beta was actually ready last night, but iWeb, one of our hosting providers, had a fairly major outage in the data center that the update servers are hosted in. The servers are now back on the net, so updates should work normally.
Our overall strategy for updates is to spread out the update pool over a few providers so that temporary outages only reduce capacity, not availability. In the short term, however, we have a lot of capacity at iWeb because their pricing is really aggressive and we had to push a lot of bits for the X-Plane 10.0 roll-out. This is the first outage they’ve ever had; if we see continued reliability problems we’ll get more aggressive about rebuilding our demo/update mirror.
Controlling Scalability
With 10.04 beta 2, the attached objects on the parking garage facade now shows more or less cars based on the object rendering setting. This means that if you have a computer that cannot instance and you run with lower object density, the KSEA is going to be a lot less expensive than it used to be. The facade engine also allows more control over object placement, which Tom was able to use to further reduce overall object count.
Our plan is to include this kind of object density control on all parts of the airport lego brick library so that the amount of “ramp clutter” is variable by user settings. Chris and I were in the terminal at KCLT waiting for a flight and looking out across the ramp, one thing was clear: there’s a ton of crap floating around the ramp at a major airport. With variable density objects, authors can put down major “mini-scenes” (e.g. a jetway plus docking position) and we will be able to scale from the minimum objects to make the sim functional all the way up to a ton of clutter for machines with good instancing performance.
What’s On the White Board
I have a white board on my wall with some of the highest priority bugs scribbled all over the place; I like it because it gives me an immediate view of what can’t get ignored, and when I am on the phone I can simply write down an issue that someone brings up. So far the white board issues tend to be in a few categories: crashes and serious bugs, performance, authoring platform features, and rendering artifacts.
With 10.04 beta 2 I finally had a chance to start killing off artifacts; the taxi lines should (previously they’d get all bent out of shape when they went around sharp curves) and the water shader is better. (The water shader still isn’t done, but there should be no more hard lines in it.)
At this point rendering errors and performance still dominate the board and my next few weeks of coding.
Last week I received a number of emails about the technical details of v10 scenery. My short answer has always been “please wait for the docs” – that is, I am trying to spend more time writing docs quickly for everyone and less time writing one-off emails. I was hoping to be further along in the doc process, but at least I can say I had a little bit of time to work on WED last week.
So I apologize to everyone who is stuck waiting, but we are getting closer to a useful package of “stuff” (that is, tools, source code and docs). I think you’ll need both tools and source to take advantage of X-Plane 10.
I spent a few hours last night working on X-Plane’s facade engine – that is, the part of X-Plane that builds buildings, fences and structures from their shape traced on the ground. Besides fixing some v9 compatibility bugs, I had a chance to add in a few last features that didn’t quite make the 10.00 cut.
I used Tom’s “classic”-style parking garage (you’ve seen it – it’s the one used at KSEA in the demo area!) to test some of my work; some of the work on the facade engine illustrates three different ways to optimize a program.
Be More Efficient
The first way to optimize is to find ways to accomplish the same work in less time – that is, to write more efficient code. This is certainly the least risky form of optimization: there’s no trade-offs, you just get more for your hardware dollar.
In the case of Tom’s parking garage, after putting geometry in place for the roof of each floor (to fix a rendering bug) I noticed that we were building the mesh for each floor twice – once for the roof of the level and again (but facing in the opposite direction) for the floor of the next level. I added some code to the engine to recycle the mesh, but create duplicate indices; now two-sided roofs in facades use about half the vertex count.
There are still a number of places in X-Plane 10 where we can be more efficient. But rewriting code to be better takes time; these performance changes will come out incrementally, and any one change may not be that noticeable. What is important though is the net effect of improving efficiency over and over again.
Cheat!
This sounds negative, but when it comes to computer graphics, cheating is preferable. Clouds behind the panel? Just cheat – don’t draw them! Trees too far away? Use a billboard! No airplanes in the game? Use a painting for the sky and don’t bother with 3-d clouds. What I am referring to here is finding ways to deliver the appearance of eye popping graphics for significantly less work. This kind of optimization is critical to making a competitive rendering engine.
In the case of Tom’s garage, the facade engine puts the same number of cars on every level of the garage. This is hugely wasteful; while you might want cars on the roof, you don’t need very many on the lower levels. The lower levels are only visible from the side, and from the side even a few cars are enough to create the appearance of a non-empty garage.
So I am adding to the facade engine the ability to specify mid-level objects and roof objects separately; Tom can use that to “thin out” the objects in the garage. The result will be a big reduction in OBJ count for almost no visual quality loss.
I think there’s a lot of fine tuning we can do like this to make sure that we spend our hardware budget only on things that really matter most.
Scale Down
This isn’t really a type of optimization, but it’s important to make sure that the rendering engine can throw visual detail out to meet the requirements of older computers. One of the problems with X-Plane 10 is that some parts of the rendering engine only have one mode: “maxed out”.
10.04 beta 2 will allow facades to specify attached objects (like the cars on the parking garage roof) that decrease in density with the rendering settings. The result will be a full garage when objects are on “insane” but a mostly empty one with just a few cars when objects are on “default”.
Tom and I are working to put this kind of object density control in all of Seattle (as well as the airport library it uses and the rendering engine components that the library uses); the result should be a Seattle demo area that becomes significantly cheaper when the rendering settings are dialed back just a bit.
There are three nasty bugs haunting X-Plane 10 that are on my plate right now; these are serious bugs that pretty much ruin the X-Plane experience for those who hit them. They are marked in the bug base as “hair on fire” – the highest priority we have for a bug. (Yes, the bug base literally has a priority category called “hair on fire”!)
Every morning I have to ask “can I close any of my hair on fire” bugs, but these three have been nasty enough that a status update is warranted, even before we have a fix.
Bug 1: Poor Performance with ATI Hardware on Windows
A code path used by the cars and clouds is stalling in the ATI driver on Windows. I don’t know why this is, but we are trying to follow up with ATI directly on this. The result is pretty unfortunate: if you have any cars (even “siberian”) or any clouds (even scattered) framerate absolutely tanks.
As it turns out, if you set weather to clear and turn off cars completely, ATI performance on Windows is pretty good! Obviously that’s not a useful work-around, but if you have ATI hardware and are wondering what kind of fps you’ll get once we fix this, those are the two features to turn off. (I suspect rain may hit this code path too.)
As with all driver-related problems, I must point out: I do not know if it’s our fault or ATI’s, and the party at fault may not be the party to fix it. Apps work around driver bugs and drivers work around illegal app behavior all the time…that’s just how it goes. Philosophically, the OpenGL spec doesn’t require any particular API calls to be fast, so you can’t really call a performance bug a bug at all. It’s ATI’s job to make the card really fast and my job to speculate which subset of OpenGL will cause the card to be its fastest.
I have a Radeon 7970 and will write up some of my performance findings later. Unfortunately with the clouds affected by the problem code path, it’s hard to see how the card will really do.
(OpenGL nerds: the problem is that glBufferData with an in-flight VBO and a NULL pointer is stalling. This particular use of glBufferData is meant to tell the GPU that we want a new fresh buffer, with the old buffer being asynchronously discarded once the GPU has completed drawing. This “orphan the buffer” pattern is the standard way to do high performance vertex streaming in OpenGL prior to glMapBufferRange.)
Bug 2: Hang on Start with NVidia Hardware (Windows)
A small percentage of Windows users get a hang on start with NVidia hardware. We’ve had this bug for years, but it’s more of a problem in version 10, where command-line work-aronuds are not available. We have identified four third party programs (Omnipage, Matrox Powerdesk, Window Blinds, and TeamViewer) that all seem to cause the problem at least some of the time, but some users have this problem with no obvious cause. There isn’t a hardware or driver pattern to it either.
Until recently, we could never reproduce this bug in-house; it was only through the patience of users that we were able to characterize exactly what was going wrong via copious logging. But a few days ago we got a break: a user reported Window Blinds to be hanging X-Plane, and unlike before, Window Blinds hangs our machines up too! (That’s a huge step forward in getting a bug fixed.)
At this point we’re working directly with NVidia to try to find out what’s going on, as the hang occurs inside their driver code. The same rant applies here as above: until we know exactly what’s going on, we can’t say whether this is a driver bug or an app bug.
Bug 3: Crashes Near Munich on Linux
Linux users see crashes near EDDM with memory exhaustion, even at very low rendering settings. This bug appears to be Linux specific and we do now know what’s going on.
For reasons that I cannot yet fathom, the math deep inside the ATC airport taxi layout auto-generator goes haywire on Linux, resulting in an infinite loop that allocates memory, eventually crashing X-Plane.
What’s infuriating about this bug is that it is a Heisenbug: all of my attempts to diagnose or “prove” what is going on causes the math to stabilize and the app to function. The bug does not appear when the optimizer is off; it certainly looks like an optimization bug but I hate to call foul on the compiler guys without a ton of evidence. It’s just too easy to write C++ floating point code with sharp edges.
I don’t have a good time frame for the two video card related bugs; I consider us lucky to be able to communicate with ATI and NVidia at all. I don’t have insight into their development process and if I did, it would be covered under our NDA. The only thing I can say is that these are top priority bugs.
The Linux bug is still in my court; I last put it down in a state of GCc-induced rage. I believe I will be able to get it fixed some time during the 10.04 beta process.
Final unrelated thought: all of the GPUs in my office, listed in performance order…
GeForce 5200 FX, GeForce 6200, Radeon 9600 XT, Radeon 9700, Radeon X1600M, GeForce 8800 GT, Radeon HD 4870, GeForce 285 GTX, Radeon HD 6950, GeForce 580 GTX, Radeon HD 7970
