With X-Plane 8 and 9 we had a perpetual problem: the airport layouts ship with X-Plane and are updated frequently by Robin; if an airport layout is changed significantly or a new runway is installed, the airport layout may conflict with the base terrain and autogen from the underlying DSF that was cut when the sim was first shipped.
This resulted in a huge amount of stress for users who were building airport layouts. When is the cutoff for new layouts? What if I don’t make it? It will be years before the DSF is fixed.
I think the solution for version 10 is actually quite simple: we will recut individual DSF tiles and “push” them in sim updates.
In fact, we are already doing this. Five tiles, including the tiles containing Paris and Denver, rolled off the production line broken and we didn’t detect the problem until after we burned the DVDs. The patch is already online, and if you are running with current betas and installed Denver and Paris from DVD, you should have the corrections.
I think we’ve reached a point in X-Plane’s development where we need to evolve the global scenery from a static product that never changes to an ongoing one that can be updated in pieces, on the fly.
The History
I have been involved in X-Plane since version 6; for as long as I’ve been around, the product has been updated frequently with frequent patching. Starting with X-Plane 8 we developed a proprietary installer that let us easily push patches to three operating systems, with users downloading only the changed bits of the sim.*
The problem with the update strategy was that the global scenery was always too big. And what we found was that if we perpetually recut the global scenery, it made everyone who had already bought the scenery crazy mad. The last thing you want to do is buy the sim and discover it would have been better if you waited.
The idea of the “update” strategy is that you can buy into a major version run as soon as you want or late, and if you jump in early you get all of the future updates for the version run for free over the net. So we would update the sim and leave the global scenery alone. Thus there was never an update to the version 9 global scenery.
The Brave New World
The problem with not updating the scenery is that it takes us years to fix bugs, share improvements with users, and integrate changes from users (e.g. better airport layouts). Using OpenStreetMap raises the stakes: anyone can improve raw vector data using OSM at any point. How long do we wait to ship the improvements to everyone?
So I think it’s time to start recutting tiles and shipping them in updates.
Now to be blunt: I have no idea how we are going to go about doing this. So far recut tiles come down in free updates to users who have them installed, but this is just a beginning; we don’t have scalability problems yet because we have recut only 5 tiles.
Airport Anomolies
In the long term I don’t know how many tiles we will recut, when we will recut them, or what percent of the Earth we might recut in a given version run. We are not promising to recut any particular area or set of tiles. Please read that sentence twice. Don’t come back to this post and say “but you promised me free global scenery.” I don’t know how much recutting we can do or what the terms will be. I just know that it is the direction we need to investigate and move in.
With that disclaimer in mind, I do think we will be able to recut tiles that have airport anomalies. For example, I have two bug reports on class B airports. KSFO has an incorrect coastline, and KORD has a collision between autogen and runways. In both cases, I don’t know what went wrong and will need to investigate. And in both cases, pushing the fixed tile in an update seems like a reasonable solution.
* As far as I know, no commercial solution would give us three operating systems, deltas from any past version, while making it quick to cut a patch. Since we use the installer for every beta, the initial development time was worth it to make patching quick for users and for us.
My preferred way to fix X-Plane problems is to fix the problem first, then post second. That way there is no question that what we ship is what we announce.
But there’s a bunch of issues floating around with X-Plane 10, some of which will either take a while to fully fix, or which will be fixed in steps, with some steps happening now and some happening over a much longer time frame as the sim evolves. So I’m going to write some of them up now and get the information out there. A lot of users have taken the time to send in good bug reports, so it’s not fair to then sit on the plan.
A lot of these plans are just that: plans. Plans change. I believe that one of LR’s strengths is that we can change the plan quickly – if we come up with a plan and it turns out to be dumb we don’t have to just go through with it, we can come up with something better.
With that in mind, I had the following conversation with Austin yesterday – over a rather poor cell phone connection:
Me: Heya! Do you want to talk about BAD ALEX?*
Austin: Wait, what? What did Alex do???
Me: Oh…no no. Bad ALLOCs! A-L-L-O-C-S. Running out of memory…
A number of you have seen this error. When X-Plane runs out of memory one of three things usually happens:
- Some low level part of the sim seg faults because the request for memory failed and wasn’t properly detected.
- The OpenGL driver blows up because it can’t function without memory.
- The sim tries to allocate memory and throws an “uncaught exception: std::bad_alloc” error.
So first, let’s make one thing clear: we are running out of virtual address space, not physical RAM or VRAM. X-Plane is a 32-bit process, and thus it can only access 2-4 GB of virtual memory (address space) regardless of what is in your computer. The specific limits:
- 32-bit Windows: 2 or 3 GB, depending on OS settings.
- 64-bit windows: 4 GB
- OS X: 3.5 GB
- Linux: 3 GB – I think.
This address space has to contain a few big things:
- A copy of every texture X-Plane is using. (Most OpenGL drivers will “back up” textures in virtual memory at least some of the time if not all of the time.)
- Every mesh that will be drawn for all scenery, planes, etc.
- The weather system (physical and drawn).
- The physical mesh that is collision checked.
- The sim itself.
- etc.
So if you increase the combination of screen size, texture res, turn on HDR (which allocates a bunch of textures to render to), crank up the autogen, the sum can exceed X-Plane’s virtual address space limit.
The Short Term: Surviving With 3 GB
In the short term there are a few things that you can do and a few things that we can do. If you are seeing crashes or bad_alloc errors:
- Turn down rendering settings. The major ones are: airport detail (set to default), forests (set to something moderate if you are using autogen too), and texture res (first run with compressed textures). Don’t use 4x SSAA when in HDR mode – use FXAA instead.
- If you are on 32-bit Windows, consider moving to 64-bit Windows, at least in the long term.
- If you are on 32-bit Windows with 2 GB per process, increase that limit to 3 GB as shown here.
Note that buying more RAM will not help! We are out of virtual address space, not physical RAM chips!!
In the short term we can:
- Improve the error reporting to make it more obvious what has happened.
- Attempt to detect a lack of memory earlier and give warnings.
- Tighten our memory belts – I need to go on another memory hunt. Unfortunately we went on a pretty serious one before we shipped so there may not be any quick wins.
- Detect 32-bit Windows that are running without the 3 GB warning and flag this for users clearly with help instructions. If you’re running with 2 GB on 32-bit Windows it’s hard to see this in the OS itself, but getting that extra GB makes a big difference!
The Long Term: 64 Bits
The long term solution is clear – we need to migrate X-Plane to 64 bits. This is not going to be particularly easy or fun.
- On Mac, we have to port all of our platform specific Window/UI setup code from Carbon/C++ to ObjectiveC/Cocoa because Apple doesn’t provide our APIs in a 64-bit variant. Thanks Steve.
- On Windows we have to change compilers, as the one we use now is old and doesn’t build x64 apps.
- Sandy and I will have to make a 64-bit variant of plugins.
- 32-bit plugins won’t run in a 64-bit app. As far as I know none of the three operating systems provides a way to bridge DLLs like this, so plugin authors will need to create 64-bit compiled versions of their plugins.
It’s not a trivial amount of work. But it is what we need to do. It’s crazy to have a video card with 3 GB of VRAM and an app with only 3 GB of address space – in that situation by definition an app could only fill VRAM with textures if it had no code (let alone data). For years the cards, CPUs, buses, everything has been getting faster, bigger, more, except for virtual address space limit, which we have finally smashed into face first.
There have been a bunch of posts in which I have said “we know we need to do 64 bit” but now we’re saying “it really needs to be on the next train that leaves the station.” We will probably start the porting work once we get the 10.0.x series of patches stabilized.
X-Plane 10.03 beta 4 is out – just a few unrelated notes:
- Legacy scenery packs should work – I’ll write that up in detail in another blog post.
- If you have Paris or Denver installed, you need to update using the newest updater (version 3.02). If you update from the About Box, you get this version automatically. But if you have a version of the updater already on your hard drive, get a new one here.
- If your third party add-on worked in v9 and is still not in working in v10, please file a bug if you have not done so already. If you did file a bug, please bear with us – we’re killing off compatibility bugs with each patch, but there are still some major ones out there. (For example, I have on my plate to fix some bugs with OBJ materials that affect a lot of airplanes.)
And I meant to link to this: everything you ever wanted to know about global scenery but were afraid to ask.
I have a number of technical issues to cover relating to v10 scenery, so I’ll try to increase the blog post frequency to get through it all over the next week or two.
The short answer is: I think it will be fixed for everyone in 24 hours, but less if you want to get it by hand. Here’s the whole story:
The Paris DSF rendered incorrectly when we cut the global scenery. We discovered this after we cut the global scenery DVDs, but before they shipped out to users. I found the bug in the DSF generator (it was an error in airport processing) and fixed Paris and 5 other tiles. This was a top priority fix – it’s been fixed for probably two weeks now.
The Paris DSF is included in the X-Plane 10.02 beta series – that is, it has been available for over a week!
But here’s where things go sideways: since we’ve never updated global scenery before, you need a new updater to actually get the file. And what I am just realizing now is: while you need the X-Plane 3.02 updater to get the file, everyone is using the 3.01 updater, which doesn’t know to grab the Paris fix if you have Paris installed from DVD.
Hang on one second while I bash my head on my desk.
Okay. So there are two ways you can get your Paris fixed.
The Fast Way
If you want Paris fixed now, do this:
- Get the X-Plane Updater from here.
- Run the updater on your copy of X-Plane.
Paris should be fixed, at least I think.
The Easy Way
If you have already installed global secnery for Paris and you do nothing now, but get beta 4 when it comes out, X-Plane will use the latest updater to get beta 4 and you’ll get Paris anyway. I expect beta 4 tomorrow…maybe Monday at the latest.
If your joystick throttle axis won’t move the throttle (but will move in the axis tab) it could be that the autothrottle is on – autothrottle state appears to be saved across sim runs and airplanes in X-Plane 10. I am investigating this and we should have a fix soon, but in the meantime if you’re stuck, you may have to open a plane with an autothrottle on the panel (or use a key command) to turn it off.
Posted in News
by
Ben Supnik |
A number of users have reported “ghosting” with NVidia Windows hardware and X-Plane 10. The artifact typically looks like this:
What you’re seeing there is the scene from last frame being left around where the sky was supposed to be drawn, but was not. As the camera moves, we get a trail of artifacts.
For a few days I couldn’t see this bug on my own system, and this afternoon I tried to reproduce it based a report from a user with the same hardware as my development machine. So after a bunch of failed tries and scratching my head at the German version of the NVidia control panel, I was overjoyed to see this:
I mean, how cool is that! That could be our new startup screen! It’s like a giant 737 is going to pop up out of the screen and eat us all. (Hint: if you are in the pit of a 747 and an airplane appears over the horizon that looks bigger than you from 20 miles away: run!)
It appears that this artifact is caused by turning on full screen anti-aliasing in the Nvidia control panel (with “override application settings”) but not in X-Plane. My work-around: use “application controls” for the control panel and set FSAA in X-Plane itself.
Here’s another fun one:
This shows up with moderate global shadowing. This shows up on the 285.62 NVidia drivers; the 280.62 and (for the very brave) 290.36 drivers don’t have this problem.
I don’t think there’s a moral of the story here; we’re still working through bugs, but I did find the bottom of my in-box, which I hope is an indication that the patches are helping at least some users. There are a lot of different card/driver/OS combos, so working through all the combinations (as Windows X1000-users are finding) will take some time.
A new beta is out – 10.03 beta 1 – release notes here.
I guess people don’t really have .plan files anymore, but here’s what would be on my short list if I had one, now that 10.03 beta 1 is out:
- Crash bugs and artifacts. We’re still seeing some of these, particularly on old hardware; I am still working my way through these.
- Third party compatibility. 10.03 beta 1 starts to fix some problems with older planes, but it’ll take a few more betas to work through these – I still have a long list! Thanks to the aircraft authors that have sent bugs in.
- Performance. I haven’t had a chance to dig in hard on this yet, but I have been logging bugs with some of the complaints that have come in. I hope to be on performance in a big way in the next week.
Austin, Ben and I are still working on cleaning out the remainder of Ultra-High priority bugs right now which essentially consist of crashes or other things that impact large groups of people. Only once these types of problems are taken care of will we be able to start looking into figuring out what can be optimized in the sim.
Many of you are getting really decent performance. Many of you are getting “ok” performance and others are getting really poor performance. This is not a surprise because there is a huge spectrum of hardware out there. Moving forward, we have some tools at our disposal that we can use to collect some data see what’s impacting performance the most. Then we can optimize it. Then we can see what the second biggest problem is, and then optimize that….and around and around we go.
In my opinion, one major contributor to performance loss right now is having your settings wrong. X-Plane allows you to configure a LOT of settings to get optimal performance. We’ve given you more than enough rope to hang yourself with, and many of you are doing just that. X-Plane’s rendering engine is very flexible. It has to be because it has to run on a huge spectrum of hardware. Like any large system, it needs to be tuned properly for its intended use. Ben’s already told you that you need to hit rock bottom to tune X-Plane properly. What that means is that you need a baseline. Think of how you tune a guitar or any instrument for that matter. You need a reference pitch such as a tuning fork. You match one string up perfectly to that pitch and then you adjust ONE string at a time until it’s in tune with the previous string. Imagine trying to tune a guitar without a reference pitch. You would have no idea whether you were making progress or not!
The “tuning fork” of X-Plane is starting from zero. This is your reference. Turn off EVERYTHING. Turn of AI aircraft, set your clouds to clear, set your visibility to something reasonable like 20-30SM, turn off rain and storms. Go to your Rendering Options and set EVERYTHING as low as it will go. If you run X-Plane in fullscreen mode, turn down your resolution. If you run X-Plane as a window, reduce the size of the window. The item that says “Framerate-lock to monitor” should be set to “Do Not Lock” (more on this in a moment). Take a look at what kind of framerate you get in the sim. Don’t just sit on the runway, fly around, take note of the range that you see. Also, see what you’re system’s doing as well as you can. Take a note of CPU usage, Memory Usage (mainly how much is left to be used), and GPU usage if you have a fancy driver that lets you see this information. At this point, you should have a rough guess as to what your weakest link is. If your CPU is almost maxed out right now, it’s going to be your limiting piece of hardware. If your almost out of memory, that will be your limit etc.
(A quick detour on CPU cores…more is better-ish…to a point. 9 women can get together and do 9 different tasks in 1/9th of the amount of time it would take 1 woman, but 9 women cannot have a baby in 1 month! The point is, if X-Plane has multiple tasks going on (like AI traffic, scenery pre-loading etc), then multiple cores helps get the tasks done quicker. But if there’s only one task going on (like rendering), your other cores will NOT help you do it any faster.)
Back to tuning…Start with Rendering Options. Now start turning things up ONE setting at a time. Start with things that are most important to you. Each time you change something, go back to the sim and see how it impacted you. Take note of your hardware utilization again. Has CPU changed much? Has memory changed much? Don’t be fooled by large jumps in FPS. Remember, going from 100fps to 90fps is a difference of 10fps (“wow” you’re thinking)…but that’s only 10%! That same 10% reduction at 20fps would bring you down to 18fps. You should care about the PERCENTAGE change, not the FPS change. If you notice that one setting in particular causes your framerate to lose a huge percentage, then you’ve probably figured out what the weakest part of your particular hardware is. This is the setting that you’ll either have to live without or the one you’ll have to be VERY careful about touching. If this setting is important to you, perhaps there is something else that you can sacrifice to get some performance back for this setting. Or if you’re able to upgrade your hardware, that’s the thing that you should invest in.
At some point, hopefully you’ve found yourself with a configuration that you like. Again, note your average FPS in various situations. NOW you can go back and set the weather how you like it. Turn the clouds up, turn the visibility up if you want but be careful. Visibility can be a huge factor in performance. The farther you can see, the more things the system has to draw. I would not recommend going over 50SM unless you find yourself with a ton of performance left over. Now you can turn on the number of AI planes that you like flying around with. You may have to go back and retune some things to get the system balanced with the new weather and AI. Make SMALL incremental changes. You don’t want to de-tune it severely. When everything’s balanced, now go back into Rendering Options and note at the very bottom, it will tell you how much VRAM is being used. Turn the Texture Resolution up until the VRAM usage gets near the amount of VRAM you have on your GPU. Then fly around and test it. You should NOT notice any difference in performance. If you do, go back and turn your Texture resolution down a notch and try again. Your goal is to get your texture resolution as high as you can without impacting performance or going over your VRAM limit.
Lastly, a note on VSync. I told you earlier that the “Framerate-Lock to Monitor” setting should ALWAYS be on “Do Not Lock”. This is a tiny white lie. If you’re an advanced user, if you know what VSync is, if you know how it works and if you have enough performance to spare and you want to stop scene tearing, go ahead and turn it on. Even then, don’t turn it on until you are done tuning framerate, because VSync will obscure changes in fps, making it impossible to tell what effect a rendering setting has on your hardware. If you don’t know what it is or what problem it’s meant to solve…please leave it off and forget it exists. It will NOT improve your performance. If you’re like me and just want to learn about it anyway, read this but leave the setting off anyway.
| Feature |
Affected Subsystem |
| AI Aircraft |
CPU/VRAM/Bus Bandwidth |
| Cars |
CPU |
| Clouds |
GPU |
| HDR |
GPU |
| Objects |
Bus Bandwidth (CPU on old cards) |
| Roads |
Bus Bandwidth |
| Forests |
Bus Bandwidth |
| Screen Resolution / Window Size |
GPU (usage proportional to resolution) |
| Shadows |
Bus Bandwidth (this amplifies all other bus bandwidth use a lot!) |
Table 1. Feature vs. Affected Subsystem
***EDIT*** I didn’t want to have to do this but I’m going to be more heavy handed with the comment moderation because inevitably, the comments end up getting WAY off topic. Feel free to ask questions or post comments directly related to the topic but this is not the forum for other discussions. I am also not going to be providing one-on-one tuning at this time so please do not post your specs unless asked. We have not begun diving into performance issues yet. The purpose of this post is to educate you on how to setup your own system in the most optimal way possible right now. Thank you for your understanding!
A number of third party authors (bravely) promised X-Plane 10 updates to their airplanes. And I believe that a tune-up to be X-Plane 10 compatible isn’t going to represent a lot of man hours.
That is, unless you try to do this job now.
I have a number of open bugs where version 9 airplanes don’t load quite right in X-Plane 10. If you have an X-Plane 9 airplane and you try to work around these bugs, a few things will happen:
- You will only be able to work around some of the bugs, as others are pretty hard-baked into the sim.
- When I actually fix the bugs (in the next weeks) your airplane will be “broken” yet again, since “fixing” the bugs now means trying to make a right with two wrongs.
So third party authors: please do file bugs if you haven’t already, and give us a little time to work through them. Please do not try to work around these bugs, only to have your airplane become “re-broken” when we get the sim corrected.
And users: please be patient with your third party airplane authors. They can’t make their plane v10 compatible until we fix some bugs, and if they try they’re just going to get thrashed.
There are some things that do need to be reworked for version 10, particularly for HDR mode. But a lot of the reports I get are just things that are funky in the sim.
How to File an Airplane Bug
Since I am getting deluged with bug reports, support requests, questions, etc. I want to describe the best way to get an airplane compatibility bug to us. By following these guidelines, you’ll make it easier for us to kill off compatibility bugs fast.
- Please file the bug only once. If you have filed the bug and haven’t heard that it’s fixed, you do not need to tell us in every new beta that it is still broken.
- File bugs via http://dev.x-plane.com/support/bugreport.html. We can route this form to whomever we think is best suited to handle the bugs. Please do not just email the last person you conversed with directly.
- Please only file bugs if an airplane looks different in the latest X-Plane 10 build and X-Plane version 9.70. X-Plane 9.70 is the version 9 release that we are targeting – no older version!
- Please get us reproduction materials – preferably a complete ACF pack, and preferably a cut down one if it can be simplified.
- Send us the v9 .acf file, before any modifications. We want to see what your customers would have seen if they just tried to use the plane. If you send us a version resaved in X-Plane 10, we don’t know what happened.
- Please provide illustrations of how the plane should look in version 9 vs how it does look in version 10. We need a reference point.
- Please try to keep reproduction steps as short as possible. If we have to make a 2 hours flight with 400 waypoints to see a bug, that’s a time sink for us.
A number of you have already sent us good bug reports – we will get to them as quick as I can. If all goes well tonight my fires will be out and I’ll be able to jump into this shortly.
If I could have a nickel for every email I see from a user who says “I have installed X-Plane from scratch 6 times, and it still does not work”, well, I’d have enough money to buy more servers to handle the download bandwidth.
If your copy of the X-Plane 10 demo is not working, please do not do a full reinstall. It will not help.
You can run the X-Plane updater at any time. The X-Plane 10 updater checks every single file in the install and replaces any file that isn’t exactly what it should be. When the updater is done, you basically have a clean install, but at a fraction of the download.
The demo servers are starting to show lower load – please don’t re-download and kill the servers all over again.
Posted in News
by
Ben Supnik |
