First: X-Plane 10.04 rc3 is live – please grab it – make sure to check “get betas” to get this patch! (If you are already using 10.04 betas, simply go to the About box to update.) Because this is a release candidate (we think it’s good to go but we’re not sure) you must still check “get betas” – that check box should really be “get crazy experimental stuff that is not yet officially released to the world.”
I think we are reaching the point where we can slow down the rate of patches just a bit. Like past X-Plane releases, the first patches came out fast and furious, because we wanted to get critical bug fixes to as many users as possible. We are now reaching a point where we can slow down and get more done between each beta, which I think will mean better rate of improvement for the sim overall.
Of course, if we get a really huge bug fix (e.g. an ATI performance boost or a fix for NVidia start hangs) we’ll cut a tiny patch off of 10.04 immediately; our source control system lets us do an incremental update very easily.
So with the next patch we will be able to work on some bigger bugs:
- I think the next patch won’t be a full 64-bit patch, but it will contain a lot of the ground-work to get 64-bit working. We’ve already started on this code; by splitting it over two patches we can more rapidly deal with any kind of incompatibility introduced by the newer code. (Our goal is of course to make sure that any machine running 10.04 can run the 32-bit version of X-Plane when we support both bit depths.)
- Austin received a lot of feedback about the UI while we were in Mallorca, and he has already started this work, although I am not sure what will be released when.
- I will be digging into “Galloping Gertie” (our internal name for the many weird shapes the roads take when the road-placement code goes haywire). This is going to require some pretty major work to the road generation code, but of course it’s a critical fix.
- While the volume of complaints about performance have gone down, there are still one or two performance tuning changes I want to get into the sim.
There are only about four things I work on when I am in the office: performance, fixing bugs, developer tools, and the installer. Hopefully with 10.04 installer work is done for a while and I can get one major area off my plate. This should leave more time for developer tools and documents, which I try to leave a fixed amount of time for every week.
One question that people have asked me about the X-Plane developer’s conference in Mallorca: what did you guys do there (besides drink)?
The conference was attended by a number of MSFS third party developers, a few X-Plane third party developers, as well as Austin, myself, and Aerosoft management. So the audience was mostly technical people (developers) and mostly MSFS, not X-Plane developers.
The sessions covered two basic areas:
- Boot-camp, e.g. Austin and I did our best to rapidly introduce the MSFS developers to the X-Plane-specific parts of development, e.g. you know how to model, but how does a model get into X-Plane. Most X-Plane third party developers already know this stuff, but for MSFS developers the question is: where did you guys move the furniture in this new room?
- New features, e.g. we described a number of the new things X-Plane 10 can do that are new to both X-Plane 10 and flight simulation in general. Some of this stuff hasn’t made it out to the public yet, due to me not getting docs out fast enough. McPhat studios were present, and they had all sorts of nice looking camera equipment, so we may be able to get some video posted.
For me, the most useful aspect of the sessions was the interactive aspect. Being able to have artists with significant 3-d experience ask me questions about X-Plane in real time really helped me understand what parts of the scenery system are important to authors and which are not; I am working on a new set of documentation that should hopefully be much better tailored to artist’s needs.
The experience also has changed my view on the scenery tools. In my past work on WED and the ac3d exporter, the tools have provided a more or less direct wrapper around the capabilities of the scenery system. This has two problems:
- In some cases this means more work for the author, because the tools don’t provide a layer of simplification, automation and abstraction around the low level system and
- In some cases the tools make it possible (or even easy) for the author to make non-optimal scenery and models; the tools really should make it hard to do things the “wrong” (but legal) way.
For the upcoming US developer’s conference (in Columbia in April) the sign-ups so far look like a more mixed group, with some very experienced X-Plane people and some totally new developers coming over from the MSFS world. We will be tailoring the content to try to meet the needs of both audiences.
If you attend, bring your content. Austin and I are very happy to sit down with your add-on and X-Plane (perhaps with debugging) and take a look at performance.
I am back from Mallorca. I will try to post more info on X-Plane developer conferences soon, but for now I will only say: it was great to meet a bunch of developers and friends in person (indeed, one of the things I like most about flight simulation is that it brings people together, often from across continents) and any picture you see posted of me with copious quantities of beer or other alcohol are photoshopped.
We’ll be cutting an RC2 of X-Plane 10.04 pretty soon with a few minor changes – it should be ready in a day or two.
I am also looking at better memory error reporting for 10.05. The crash reports now show memory usage at the time of crash and it’s clear that almost all of the crash reports we now get are memory exhaustion. (Well, address space exhaustion, technically.)
I spent part of today looking at whether I could detect this case and put up a clean error message. It turns out it’s very hard. X-Plane uses multiple cores to load scenery, and as a result, we can run out of memory simultaneously on several cores. On my 8 core machine, it’s like having 8 race cars heading toward a brick wall at 200 mph and you only have one airbag. Often a second thread will run out of memory and explode while we’re coping with the first one.
Clearly the long term solution is 64-bit, something that we are working on. But I am hoping that for X-Plane 10.05 we can at least recover enough to put up a useful error message (e.g. “you ran out of memory”). This will help users easily differentiate when they have to turn down settings vs. when they have found a real bug.
And speaking of using up memory…Andras has posted HD meshes, and they look awesome!
Finally, one last note on ATI Windows performance – I know everyone is itchy for an update, but here’s the thing: we have NDAs with AMD, NVidia, Apple, and Intel. Therefore when we get information from them, I can’t post it here. So unfortunately whether a bug is not being looked at, being looked at, understood, being fixed, or already fixed and just waiting to make it into some kind of release, I have to say the same thing: very little. I know that that’s frustrating, but I think we’re better off having close relationships with these companies and being able to solve these problems. If there’s ever a chance to put a work-around in X-Plane, we do that.
So I can’t give you any new news on ATI Windows performance, and I am sorry about that. I can only say that it is my top priority.
I realized the other day that while Chris and I have discussed cities with a bunch of people, I left out the city plan from my series of “road map” blog posts a while back.
The picture on the right is downtown Seattle from the upcoming X-Plane 10.04 beta 7, which includes an update to the urban art assets: new facades, a bunch of lit textures, and careful library tuning by Propsman. (There’s also some clever use of spill lights next to the tall buildings.) This update is part of ongoing work to build out our new city autogen; this post will explain the road map for that work.
A Radical Approach to Cities
Before I describe some of what’s failing in the current city scheme, how it’s supposed to work, and what we’re doing to fix things, let me take a few paragraphs to describe the “big idea”. Cities in X-Plane 10 are completely different from X-Plane 9, and they’re completely different from any other flight simulator that I’m aware of.
Before X-Plane 10 there were three approaches to cities that I am aware of:
- Land class tiles. You create repeatable square orthophoto textures of cities and put matching 3-d on top of them. What’s good about this technique is that it runs on really minimal hardware, it looks good without using a lot of resources, and it’s fairly easy to code. The down side is that you will not get accurate cities. There is no way to use additional data to put roads or buildings in their actual correct places. You will always get a plausible but non-accurate city, a “city in theory”. This is the technique X-Plane 6 and 7 used, and the technique FSX uses.
- Vectors over land class tiles. You create repeatable square textures of city (just like above), but instead of attaching the 3-d to them, you build your 3-d off of real vector data. This is what X-Plane 8 and 9 did. What’s good about this technique is that it runs on modest hardware, and it puts your roads and buildings exactly where they should be. The down side is that the 3-d is completely misaligned with the textures underneath, creating a “stew” effect when viewed up close. (Some users consider this mismatch absolutely unacceptable; others seem to not care.) There’s no question that the texture mismatch is not plausible, but the shape of the city is accurate. This is the technique X-Plane 8 and 9 use.
- Fully custom cities. If you can afford custom non-repeating orthophotos of the real city and you have the matching 3-d data, you simply build the city and everything matches up. This is the ideal way to build a city, but it assumes custom data for every city; this is great for custom scenery but not scalable to a global product. It is plausible and accurate but not global.
For X-Plane 10 we wanted a way to have it all: a city that was plausible (e.g. looks good and doesn’t have weird artifacts) but also accurate (e.g. shaped like the real city, using data that’s now available – we can know where every road is, and even some of the buildings) and for the default sim, our approach has to be global; we can’t simply build a custom city for every city on the planet. And when we looked at the technology out there, it looked like it was for the first time possible.
Here’s what we came up with: in X-Plane 10, the unit of autogen is not the landclass orthophoto tile (a 1 km x 1 km square of terrain whose entire interior is defined by the texture and 3-d). The landclass tile has wonderful properties, but it’s just too big to be accurate for a real city.
Instead X-Plane 10’s unit of autogen is the city block. Each city block is an individual autogen unit, and the autogen is capable of flexing and shaping itself to fit the demands of a real road grid based on real data. This means we can have plausible autogen with tons of detail while sitting inside the real road grid of a real city.
This approach is significantly harder than using landclass tiles. Each autogen primitive needs to be able to resize and contort itself to meet the demands of real world data while still looking like it was meant to be there. The road grid has to look really good even as it is built from vector data, because we can’t just bake the pictures of the roads into the terrain. The terrain has to contain no city details in the near view (because the autogen defines where there is city), and the autogen buildings have to have “skirts” of orthophoto that they drop down to put their driveways and other ground details in place.
Then the rendering engine has to somehow take all of these disparate parts and render them at high speed!
So Does It Work?
When the new system works, it really does work and we get plausible and accurate cities with good performance on a global scale. But this system is entirely new – because it is such a radical change ,we couldn’t recycle any art assets or code from previous versions of X-Plane and thus it is very new, and frankly a bit raw. So when it fails, the artifacts are sometimes quite spectacular.
As the system ships now, a few things tend to go wrong; I will describe what’s going on under the hood and what we expect to do to fix it.
Crazy, Deformed Roads. This is the most common reported bug. (Hint: please stop reporting this. I know about it already.) We feed extremely detailed road data from OpenStreetMap into X-Plane; unfortunately due to bugs in the code, sometimes when X-Plane analyzes an overpass, it can’t figure out how the roads should stack up, and it instead creates some kind of ridiculous bridge, e.g. a road that shoots straight up into the sky or a giant 1000-foot tall arch.
The good news is: this is just a bug in the rendering engine; when I fix the bug (which I hope to do for the 10.05 patch), the artifact should go away without the need to redo any DSFs.
I brought this bug on myself by insisting that the roads be draped on the terrain. X-Plane 9 placed roads at absolute altitudes in 3-d space, which was easy for X-Plane, but meant that roads weren’t easily used in an overlay. For X-Plane 10 I wanted to make it easier to work with roads directly; once the road bridging bugs are fixed, this should be doable.
The road system is already doing a number of things we are pretty happy about though:
- Roads are made of bezier curves. Once we go 64 bit we’ll be able to crank up the smoothness factor for users who have a lot of RAM.
- The road junctions are flexible – that’s how we get clean, real, custom-looking junctions out of raw vector data. Over time we can add more junction definitions to make nicer looking overpasses.
- City roads drape on the ground, avoiding Z thrashing and some of the other ugly version 9 artifacts.
Where Are the Buildings? The second bug report we get is that cities don’t contain enough big buildings, or enough sky scrapers. Most commonly the problem is residential houses appearing downtown.
The problem here is that the new autogen requires new art assets; we couldn’t just reuse our substantial pile of buildings from X-Plane 9. So we’re building more buildings; what you see in the meantime is the existing art assets shoved into the wrong kind of library slots. Once we have more appropriate buildings for a given library slot, things should look better.
We have also discovered some cases where the placement of autogen in the DSF isn’t very good. This was the first global render where we built the new autogen and there are some clear cases where we can do better.
So when it comes to building placement, things should get better by putting more art assets in a net update; we’ll probably recut some major city DSFs to get even better use of that autogen. (One nice thing about cities: there aren’t that many of them. There are over 18,000 DSFs in the global scenery, but the top 100 cities world-wide..well, that’s probably about 100 DSFs.)
There are two aspects of the buildings that are already working right now:
- There is a ton of detail. If you get in close to the autogen, you’ll see a scene similar to what you might get in a custom scenery pack.
- It’s fast. You can max out the road and building density on today’s hardware and get 25-30 fps. Compare this to X-Plane 9 where even four years after the product shipped, “insane” objects is often not reachable.
I put Seattle from v9 on an i5 machine for testing and compared it to v10 with insane roads, insane autogen, moderate forests, and shaders. I got > 25 fps in v10 vs. approximately 5 fps in v9. We’re going to try to keep that kind of capacity for huge amounts of autogen with high performance as we add more building types.
One last note about autogen and buildings: in X-Plane 10 the autogen can optionally be “height sensitive” – in this case the buildings in the autogen block take on an AGL height based on the DFS. We only use this for appropriate autogen types. For example, a dense urban city block might be height sensitive (since the DFSs contain heights for skyscrapers) but a block of residential houses is not height sensitive – they’re always just little houses.
In most cases the height data is already in the DSF; once we get the right autogen art assets in place, they should start repsonding to the height data.
Green Terrain. The third bug report I hear about is green terrain – that is, a city is filled with nothing but a big green expanse. To understand this bug you have to understand how our autogen creates the look of the terrain.
Austin posted in one of his early rantsmanifestos that we would build the terrain from the ground up, with green grass for the ground and no cities if 3-d is not enabled. This is a massive simplification of what actually actually happens.
The terrain is a composite: the actual base terrain is indeed devoid of any recognizable city details – because they will come from overlays. The ground instead contains “natural turf” patterns that try to impart some detail while accepting overlays.
The road grid then puts down a layer on top of the ground, with sidewalks and the actual pavement.
On top of that, the autogen system puts down a ground tile for each building. (One autogen block consists of multiple tiles, and the tiles are moved around to fit the block shape.) The tiles are mostly transparent, but contain details that mask parts of the ground, such as sidewalks and driveways.
Thus the combination of three layers (ground, road and autogen ground tiles) composite together to form the kind of detail that you would normally get in an orthophoto.
So when we have green terrain, there are actually several problems conspiring to ruin this “composite orthophoto” effect:
- If we don’t have the right autogen buildings (see above) we won’t have the right ground tiles. More buildings will help fix this.
- It’s actually a bit tricky to build the ground terrain that can sit under the autogen; the only really complete climate we have supported right now is a northern climate (which works well in Seattle). So as we add more urban base terrains the bottom layer will start to work better.
- When you turn down autogen density, some autogen city blocks disappear. This is a hold-over from version 9. What we want to do is keep the base tiles but remove the 3-d when you turn the rendering settings down. This will lighten the rendering road, but the 3-layer orthophotos will still look good. (Similarly, we’d like to reduce detail but keep all roads for lower road settings.)
In other words, once we have more buildings, more ground terrain, and a better way to turn down detail, the terrain should look more like a real city under a wide range of rendering settings.
(If you’ve really watched the sim carefully, you may have noticed that the ground textures actually crossfade from “just grass” to orthophoto-like as you get farther away from them.
What Next?
Everything listed above represents incremental improvement of our cities – more art assets, bug fixes, smarter LOD and rendering settings. But that’s just the beginning. I believe that what we have on our hands is a fairly big fundamental improvement in how cities are handled: we have a way to build cities that provides a huge amount of up-close detail while working from detailed data (and representing that data), and it can run at good fps on today’s hardware.
Down the road I think we’ll be able to integrate and utilize OSM data to get even more detail into our cities and further push the envelope for how much detail, realism and accuracy we can get into cities at a global scale.
X-Plane 10.04r1 is out – this is a release candidate. That means:
- It’s not official yet. You have to check “get betas” to see this one.
- We think it’s really pretty much done. If you haven’t tried the beta and have been going “I’ll wait until they’re closer to being finished with the beta run”, well, that time is now.
Notes here. Um, here actually.
As has been mentioned, Austin and I will be in Mallorca for this, so we’ll go through bug reports against the release candidate when w get back. I look forward to seeing everyone who is coming to Mallorca, and for everyone else, hopefully this will be the first of multiple events and forums to further reach out to the developer community. I will do my best not to jump up and down and shout “Developers!” over and over like a sweaty drugged up maniac.
I just posted new versions of the installers with a few bug fixes. Let me try to clarify the situation.
The X-Plane installer and updater have been merged. The installer is now capable of updating. One app does it all now.
Therefore:
- If you bought a copy of X-Plane, use the X-Plane installer to update your copy of X-Plane. It will require a DVD or USB key, just like X-Plane does.*
- If you are just trying the demo of X-Plane, re-run the demo installer – it will update your demo.
- Or, easiest of all: just go to the “About Box” in X-Plane and let X-Plane check for updates and run the entire process. Simple!
The newest installer is here; if you haven’t bought X-Plane 10, the demo installer is here.
* Why does the full product require a DVD or USB key to update? The answer is that it will download files that are only part of the full installation, not just files that are part of the demo. This is part of our migration toward being able to update scenery and other full-sim features online.
Working on the installer is neither exciting nor glamorous work, but it is inescapably important. If the installer doesn’t do its job right, nothing else in X-Plane matters, because you can’t fly the sim until you have the sim. This is particularly true since we push out regular patches during the life of the product.
X-Plane 10.04 will complete it’s beta cycle in the next week, and this update includes a new installer with a fairly major change: the DVD installer (which also adds and removes scenery) and the net updater are being merged. (The demo installer remains separate.) The rest of this post explains what’s going on and why.
The updater and DVD installer were originally separated a few years ago so that each update/install product could do exactly one thing. We kept getting tech support calls where users tried to update and instead installed a demo copy. By having each tool do one task and only one task, it helped users to do that one task without error.
Fast forward a few years. People have faster broadband, we have more server bandwidth, and we’re using OpenStreetMap for our scenery; scenery updates are going to be part of the product. (We have already patched 5 DSFs that had major defects when originally produced.)
So now when you go to add or remove scenery, running a net update may be a necessary step to finish the scenery. That is, you need to get new scenery from you DVD, and then grab any newer patches off the web. By merging the updater and installer, we can build an installer that provides updates immediately, eliminating a second step. (This also means that when you first install the sim, we can update to latest before you run.)
The new installer/updater will also remove the “repair installation” option (which is now replaced with “update installation). The old repair option restores an installation back to the DVD version, which is almost always older than the net version, and thus undoes bug fixes (which is hardly a “repair” at all). A net update should be more useful.
Update from the about box is still supported – when you update from the about box, the sim downloads the latest installer/updater and runs it, skipping the main menu and going directly into an update.
The new installer can be found here for Mac, Win, and Linux. The demo installer is here. Links to the updater now simply grab the installer.
I’m listening to Marco Arment and Dan Benjamin debate whether a user setting could have avoided the incident where a man’s iPhone rang during a concert. I also just finished reading the Steve Jobs biography. It got me thinking about the trade-offs between a closed non-configurable system and an open, configurable one. This comes up in an X-plane context every time we poke at the cloud rendering settings and someone asks whether we can put in more settings for additional configuration. What trade-off is right for X-Plane?
Chris suggested an analogy to me a few weeks ago that I think is pretty good: tuning the rendering engine in X-Plane should be like tuning the timing in your car. It should be possible, if you are a serious expert and enthusiast, to change the tuning (and if doing so wrecks your engine, well, you were asking for it by opening up the engine), but it shouldn’t be easy. Regular users of the car expect the manufacturer to have a lot more expertise, and to set the timing to an optimal setting up front.
I think the same thing goes for X-Plane. I should use as much expertise as I have to maximize efficiency no matter what trade-off is being made; at no point should dialing in a setting result in stupid behavior inside the engine. It’s up to me to know what can be inefficient and avoid that happening.
So: the rendering settings are going to remain as simple as possible, and any time we can make them more simple, less likely to be set up wrong, or any time we can realign them to be better aligned with user constructs (“more houses”) instead of internal constructs (“higher instancing density ratio!”) we’ll take that win. It should not be the user’s job to tune the engine, and any time that is necessary, that’s a failure by me to pre-tune the sim, not a feature.
I don’t think we are even close to the kind of rendering settings that are really useful to non-expert users; one of the frustrations with performance tuning over the last few months has been the number of times a user has sent me rendering settings and it was clear that the settings were inefficient for the given hardware.
The answer is not to try to make every user into an expert in the OpenGL pipeline; the answer is to change the settings so that users don’t have to be experts. This is not easy to do; the reason it’s not done yet is that we will have to create new code and do real hard work to make the interface simple. One theme that both Steve Jobs and Jonathan Ive describe when discussing their design philosophy is that, to make something simple in its final product, you have to dig down and master the complexity that you want to get rid of. To make the rendering settings simple but still powerful, we are going to have to solve these performance-efficiency-hardware problems ourselves and encapsulate that knowledge in code, and that’s not trivial.
Underneath the settings are a whole pile of art controls, and if you really want to, you’re more than welcome to enter any value you want in there. If you discover that you’re better at tuning the engine than I am, I will be very happy to hear what you did!
This attitude is not as extreme as the “Steve Jobs” approach; Apple devices often use custom screws to ensure that normal people can’t open their devices even if they want to. We are hiding implementation out of sight, but we are not locking that implementation at all. The settings file is plain text, the art controls are viewable with a publicy available plugin (DataRef Editor).
X-Plane has always been a hackable sim, and one of the things I used to enjoy back in X-Plane 6 was just looking at all the stuff in the resources folder; the raw textures were pretty amazing to view in themselves.
My goal here is not to take that away; only to make hacking and opening up the sim something you can opt into if you are curious, instead of something mandatory to get good performance.
Announcements on the plethora of betas that went up this weekend.
Edit: beta 5 is temporarily offline while we resolve a missing scenery texture. So you’ll get beta 4 if you update.
X-Plane 10.04 beta 5 is now available. Release notes here. I think we will be winding down the 10.04 patch in the next week and going on to 10.05. There are a number of cosmetic bugs that I hope to get fixed shortly; it looks like they’ll have to go into 10.05 and not 10.04.
What’s the logic behind this patching? Basically we’re going to keep doing small patches until we get a certain set of bugs fixed – bugs that we think are important and that we can fix without tearing the sim to bits. The patch runs sometimes have to be closed off to burn new DVD masters* (this is the case with 10.04). In all cases, if you don’t want to deal with beta chaos, you can ignore the betas and get a new patch every few weeks with a little more performance and a few more bugs fixed.
I spent some time yesterday retuning cloud performance and the effect of the cloud rendering setting. Besides trying to find a better performance-quality trade-off point, the old slider had way too much range on it. It defaulted to 50%, but that 50% point really represented “a good setting for a high end gaming desktop”; going past that would put you into the range of “this isn’t ever a good idea, but the engine can do this without crashing”. The side effect was that running at 25% clouds (a very reasonable setting) would feel demoralizing because the setting scale was so vast.
The new scale marks a “100%” point at about where I would put it for high quality if you’re just sitting on GPU power (e.g. small screen, nice GPU). You might still want to turn it down for performance reasons. The range from 100%-150% is highly experimental land…if you want to turn it up to eleven, we’re not going to stop you.
WorldEditor Developer Preview
There is now a binary build of WED 1.2 (download here). This is a developer preview: WED is still in environment and doesn’t meet the criteria for beta software. Basically: do feel free to poke around with it, but don’t use it on your real work. Don’t edit your scenery packs with it – make a copy first. WED 1.2 has not been tested enough to use for production yet!
The main feature of interest is ATC taxiway editing: WED 1.2 contains complete support for taxiway networks, as well as “flow” information to control the direction of operations. A number of weird ATC bugs actually stem from the automatically generated taxi layouts that X-Plane produces when the apt.dat layout contains no taxi information. By providing a custom layout, you can get the AI planes to behave quite a bit better.
* We periodically re-burn the disc 1 master to put the latest sim on it whenever we are preparing new DVD inventory. If you have an old DVD, please do not panic! Once the updater is run, the sim looks the same no matter which DVD you have.
Chris and I bashed at the developer site (that is, this site today); sorry about any dead URLs – things should be more stable now.
I have (finally) posted a new build of XPTools; there are two items of note:
- The new build of DSFTool, which can read and write X-Plane 10’s DSF raster data layers.
- The new DDSTool can write gamma 2.2 DDS files for X-Plane 10. This should provide better color fidelity for content that targets only X-Plane 10.
