To those who have sent performance info: thank you, but you probably won’t hear for me for a week. I’m up to my eyeballs in reports and it’s going to take a while to get through them.
I finally found the code that allows X-Plane to turn off V-Sync. This should help nVidia users who are having framerate problems.
The basic idea is this:
- X-Plane tells the graphics card to draw a lot of stuff.
- The video card accumulates this “todo” list and works on it while X-Plane runs.
- X-Plane indicates that the entire frame to be seen is done and tells the card to show the results.
- Eventually some time later the card finishes the todo list and then shows it to the user.
V-Sync relates to the question of when this last step happens. When V-Sync (vertical sync) is off, the card shows the results as soon as it is done drawing.
But when V-Sync is on, when the card finishes drawing the world, it then waits until the monitor is done drawing its frame, and then shows the results. Without V-Sync we can have a situation where the top half of the monitor is showing a new frame and the bottom half is showing an old frame. (This is called “tearing”.)
So normally V-Sync is good because it prevents tearing. But the problem with V-Sync is that a frame can only be shown when the monitor refreshes. The video card has to wait until this happens, and this slows our framerate down.
In particular, most users have their monitors set to 60 hz. If X-Plane can only produce frames at 50 hz, the video card will have to further slow the framerate down to30 hz (one x-plane frame for every two monitor refreshes). If X-Plane falls below 30 hz, we end up with 20 hz (one X-Plane frame for three monitor refreshes), and if X-Plane goes below 20 hz, we would clamp at 15.
So when monitor refresh is on, there can be large framerate hits for small losses of performance in the sim, and a real risk of getting locked around 20 fps.
(The minimum framerate in X-Plane is intentionally set to 19 so that we won’t fog up if the video card clamps us at 20 fps.)
So when beta 11 comes out, you may get some framerate back if you haven’t already hacked your graphics card’s control panel settings. If you still want v-sync, you can always set it this way in the control panel. But most users I’ve talked to are happy to have it off.
In an only vaguely related note, one of the reasons to have high frame rate is to have a smooth flight model. But Austin has now put a new setting in the operations-and-warnings dialog box: you can pick how many times per graphics frame the physics run. The normal ratio is 1:1, but for fighter and acrobatic pilots, you might find that you can get a nice feel at lower fps (20-30) by setting a higher ratio.
Random thoughts:
Update your drivers! Version 9 uses driver features that version 8 does not. Just because version 864 works doens’t mean that your drivers are up to date and bug free! First thing to try when weird things happen in version 9 is a driver update.
If you have 60 fps and a rendering setting cuts it to 30, you probably have vsync on – that is, your graphics card can only run at an even divisor of your refresh rate. The next hit will be 20, then fog. Change your monitor refresh rate to 75 or 80 hz. If the framerates all change (to 80, 40, 20, etc.) it must be v-sync. Turn v-sync off for better framerates under heavy load. Nvidia users, you need to turn v-sync to off, not application controlled.
X-Plane 900b7should be able to put sustained load on three cores – if you’re recording a QuickTime movie, one core draws the world, one compresses QuickTime frames, and one rebuilds 3-d as you fly. So…I guess we’ve already hit a point where a quad-core machine has some benefit over a dual-core machine. (I think we’ll start to see more central features use more cores during the v9 run.)
The new forests rebuild 3-d very frequently – dual core users who run on “tree hugger” should see utilization of 75% or higher on both cores, depending on video card power. (If your CPU usage isn’t 100% then probably your video card is holding you back – turn down FSAA.)
I think we’re reaching the point where I can start looking at version 9 performance in detail. Over the past few betas we’ve fixed some very general and global performance problems and also some crash bugs. Now we’re reaching the point where some users have fast performance and others don’t, and it doesn’t quite match up to the hardware the way we’d expect.
The biggest single complaint I’m hearing is “pixel shaders slow may machine down”. This is expected, but the level of slow-down we’re hearing about (2x, 3x, 4x fps loss) is a lot higher than we’d expect.
Here’s a very basic test case you can try:
- Start with shaders off, no FSAA, 4x anisotropic filtering, and a texture res that doesn’t push your card too far (“very high” for 256 MB cards). Tune objects to get between 30-50 fps all the time. Keep forests to “none” for now – they use the second core, so better not to bring in another variable for testing. Measure fps while on the ground somewhere, so that the camera isn’t moving around. Pick a view that has no panel at all, like shift-w (forward-no-HUD view). Use the default 747.
- Now turn shaders on. Turn water reflection to none, turn volumetric fog off. What’s the new fps (keeping the same forward-no-HUd stationary 747)?
- Now turn volumetric fog on (water is still “none”). What are the fps now?
- Finally turn water reflections to “default”. What are these fps?
I’d like to know these four fps numbers because they show the incremental degradation to performance as settings are slowly brought in.
Now if these numbers come out okay, but some real configuration that you tried was terrible, tell me, what’s the delta? (For example, did you have FSAA cranked up when you were testing? What was the texture res?) That will tell me the delta between controlled situations and real-world usage cases.
It will probably take me up to a week to get back to performance reports. Best venue right now is probably: send me an email with attachments for your log.txt file. Please don’t post performance to the blog comments section, I think the thread will get unreadable.
It’s too soon to evaluate whether you need a new video card. I’m hearing reports of GeForce 8800 users with low performance – and yet I have good performance on an X1600. This implies that there’s a driver call we make that isn’t fast in some cases that we need to replace with something better. Let’s get that worked out before you go out shopping!
I’ve posted the first beta of the AC3D X-Plane export plugin version 3.1 on the scenery web site. This plugin contains support for key frames and panel regions, two new version 9 features.
Generally, the scenery tool code is not part of the X-Plane code. There aren’t version 9 and version 8 scenery tools. There will always be one set of scenery tools that let you optionally use the v9 features or not.
An example is the AC3D plugin. Everyone can use the new plugin (and a few small features apply to X-Plane 8 too, like a less sluggish animation UI when editing large objects). If you use key frames, your object will only work with v9 – if you don’t, it will work with v8.
More specifically on key frames: a traditional v8 animated object is really just a key framed object with only two key-frames. So the UI has changed very little – basically where you only had two key frames (points of correspondence between the dataref and animation) in the old plugin, you can now use the “add” button to add more.
I will try to add features to the scenery tools to warn when v9 features are being used, so that if you want your scenery to run on v8 and v9, you can check for v9-only feature usage.
I woke up this morning to one of those funny coincidences that defines the experience of working on X-Plane: two users had emailed me. One was asking whether we’d be extending water reflection technology to airplane fuselages (like some other programs have) and the other made the case that such an extension was not necessary. The two emails arrived in sequence! (Perhaps there was a forum debate on the subject somewhere.)
First, I can tell you that if we ever have reflective airplanes, it won’t be that soon. I have a number of features for version 9 that are in progress and need to be finished off before I can start anything new.
Reflective airplanes are on my “investigation list”…that is, a feature where we want to do the initial research to see if it could be implemented in a way that makes sense (for X-Plane this means: would it look good and not kill fps too badly for at least some segment of our users).
I believe the X-Plane 9 version run will start to contain more features for serious 3-d modeling of airplanes. Version 9 already features 3-d lighting in the 3-d cockpit, key frames for animation, and a ton of new datarefs to drive that animation. We’re going in the direction of being able to model the plane in absurd detail.
We’re also looking at the lighting model in X-Plane. We’ve only started this work for version 9, but consider pixel-shader-based water. Even in the “no reflections” case, the pixel-shader based water is a reflection of the real sky (as rendered) with a procedural texture to create waves. When you compare this to the version 8 water, you can see how having really close alignment of the coloring scheme for all parts of the sim creates more of a sense of realism.
So reflective airplanes are at least on my list of things to try. I have seen users do wonderful amazing “metal” textures on airplanes, but the one thing that I think holds them back is that all metal airplanes have some kind of tinting assumption to them based on the reflection used…typically these are “blue-based” (meaning they look right on a sunny day) or “gray based” (meaning they look right on a cloudy day). But if you put the plane in the other environment the texture looks a lot less convincing. Reflective textures would let authors really use the real sky color on the plane, for consistent lighting (especially when the plane’s orientation changes and the blue side isn’t up anymore).
On the other hand, reflections are expensive. Planes reflect light from all sides, so we would need to take reflections from all angles (the water always reflects up, which is a huge savings). For low-quality settings for water, we drop the terrain, and since the terrain only reflects at the water’s edge, this is a pretty tolerable omission. An airplane reflection with “sky on the bottom” would look absurd. (Similarly, the water tends to only reflect things that aren’t on camera, so the total rendering load of water + the world tends to be static. The plane would pick up a lot of 3-d objects even in orientations where they don’t do much good, so plane reflections would become expensive.) And the plane reflection isn’t usable for any other plane…do we build them for all planes or just the user’s plane?
Certainly right now it’s still too soon to tell. Not only have I not done the research into this feature, but we still don’t have comprehensive performance data on the water across lots of hardware. A number of users are reporting huge framerate loss on the lowest water settings. This implies that our “render-to-texture” code is slow on some hardware but not others. (The fps loss on my laptop with the lowest water setting is less than 4%.) Render-to-texture is new to v9 and used heavily, so we need to understand how it scales for all users before we go further.
Finally, there is a whole area of 3-d techniques that X-Plane does not yet use that could make sense for airplane modeling: artist controlled fake lighting.
For example, imagine if the airplane contained a single “reflection” texture – this texture would contain a fake ground texture and alpha transparency where the sky color goes. X-Plane could then fill in the sky color (where there is transparency) only when the weather conditions change, and then apply the texture keeping the plane’s orientation in mind. Such a proposal would give the plausibility of reflections (correct coloring on all parts of the plane across lighting, orientation and weather conditions) for a fraction of the cost of “real” reflections. I’m not saying this is the best idea, just that there’s a lot of intermediate ground between “full reflections” and “make a static texture”.
Lori and I are hooked on the TV show “House”, where Hugh Laurie plays a really grumpy doctor who solves bizarre medical cases more or less by ESP. The characters are well written, but the medical plot line is somewhat predictable: there are four quarters to the show – in each one House except the last, House will make the wrong diagnosis and the patient will get worse right before the commercial break. (Usually this involves massive bleeding or cardiac arrest going into the long commecial block at 0:30.) None of the symptoms fit until the very end when House finds the simple right explanation they just couldn’t see.
This set of nvidia crash bugs felt a lot like that – we had multiple attempted fixes, some of which didn’t help at all, until finally after multiple tries I found a bug that explained all of the otherwise completely weird behavior we were seeing.
But I must admit – I have brought shame on the house of X-Plane…the buggy code was mine and the mistake was really stupid. Why nVidia on Windows? As far as I can tell the optimizers present in most OpenGL engines can change whether (and how) the bug manifests itself – different OS/driver pair: different engine with different optimizers.
Now that (at risk of massively jinxing ourselves) we have the crash bug fixed, I will resume performance work. Once we get a build done with all of the immediate performance items I want to cover, we’ll start collecting user reports on in-field performance. So I should have more specific instructions on what you can do to help us isolate performance problems in the next few days.
X-Plane 9 introduces a new OBJ feature: panel regions. The basic idea is this:
In X-Plane 8 you could use the 2-d panel as a texture in your 3-d cockpit. This allows a plane to have a moving map or glass display in the 3-d cockpit. However, the panel texture is expensive, particularly in v9 for a few reasons:
- You have to take the entire panel, even if you don’t use it all. For example, consider all of the “wasted space” from the windows.
- The real texture is rounded up to a power of 2 (and in x-plane 9 that could mean 2048×2048 for a a 1600×1600 panel.
- The texture has an alpha channel, which probably isn’t usfeul (model your 3-d cockpit windows in 3-d, not using transparency). The alpha channel increases VRAM use by 33% and requires some pixel shader gymnastic in v9 that slow things down.
- In X-Plane 9 this is all twice as painful since we have a panel day and and lit texture.
Panel regions address all of these problems. Here’s how they work:
- A panel region is a rectangle within your 2-d panel. It can be placed in any location as long as it is (1) fully within the 2-d panel and (2) its dimensions are a power of 2.
- The cockpit object declares up to four panel regions it wants to use.
- A new attribute lets you use each of the four panel regions as a texture (alpha is not provided – the regions act opaque).
This does exactly what you might expect: it creates between one and four smaller power of 2 textures rather than one huge one and manages those smaller textures. We do have more textures, which is usually bad but we get some big wins:
- Better VRAM use. The panel texture, being a dynamic texture, puts a lot more pressure on VRAM than regular scenery textures. Without this optimization, we could be paying 25 MB of permanent VRAM use just for the 3-d cockpit. Now we don’t have to pay to round up to a power of 2.
- Faster updates of the 2-d panel into the 3-d texture, since we have to process a lot fewer pixels.
- Efficiency – a clever author can cut down the panel use to only the parts that really matter, which might only be the EFIS at 256×256 pixels.
I will try to provide detailed documentation on this in the near future. There will be a new ac3d plugin coming out (hopefully in the next week) that will provide both editing capabilities for key frames and for panel regions.
(Probably I’ll be blogging a lot today…the load/change-planes/crash/recompile cycle I am going through while working on the crash bug is a slow one – my old Dell is long in the tooth…it leaves me a lot of time to post.)
Beta can be frustrating for both users (why don’t they fix the bugs I reported) and programmers (I need more details in my bug report). Here are a few thoughts on what makes an initial bug report useful:
- Precision of reproduction. This is probably the most important thing – we get a lot of “open an airplane”-type instructions. Which airplane? It turns out that a lot of bugs depend on the particular content being used. So if you know how to make a bug happen, please describe it in the most painfully precise steps possible!
- Short is beautiful. We must know precisely how to reproduce a bug, but a procedure that takes two hours kills our productivity. So please try to determine how to reproduce the bug with the minimum number of precise steps.
- Clean system. Bugs that involve only the default content shipped with the sim are more useful for us because they’re quicker to find and more likely to be due to a bug in the sim itself.
- Nuke the prefs. Bug reports that start with “delete your preferences” are good because it means the bug procedure starts from a known state (the sim defaults). We get bugs that we can’t reproduce because something is subtly different in our system. Killing prefs is the quickest way to eliminate this case.
As an example, the cleanest, simplest version of the nvidia crash bug would be:
1. Delete prefs.
2. Start the sim.
3. Open the C172 using the “open aircraft” dialog box.
Result: unexpected program termination before the terrain is visible.
We still haven’t fixed the crash bug in X-Plane 9 yet, but we have narrowed it down to:
- Windows only
- nVidia only
- Pixel shaders only
We actually know partly what happens, but we don’t know why, and we don’t know how to fix it yet (short of disabling VBOs or pixel shaders, both of which are unacceptable workarounds). Austin’s shipped b4 with shaders off, just to get people running while we work on this.
So I went to BestBuy last night and bought a 6200 (out of desparation) – I can now see the crash bug for myself. My framerates are also terrible. My guess is I can tweak the drivers, but having experience now with the 5200, 6200 and 7300, I can safely say: avoid the lowest end nVidia cards. In particular:
http://en.wikipedia.org/wiki/Comparison_of_NVIDIA_Graphics_Processing_Units
The important thing to look at here is “core config” (although bus width and clock speeds will always be higher in the higher end models). For the last four generations, nvidia has made three levels of cards: a hig end card (with a number like x700, x800 or x900), a mid-range card (usually x600) and a low end card (x200, x300). So for the GeForce 6 series, the 6800 was the flagship card, the 6600 was the mid-range card and the 6200 is the “budget” card.
Each time they go down a level, the number of shaders and other parallel hardware resources gets cut down. This translates directly into lower framerate, particularly in X-Plane 9 where we’re shader-bound when you use the shader features. I often recommend the mid-range cards as a good value, but the low end cards aren’t worth it – they’re too stripped down for not much price savings.
Back in the good old days (that would be X-Plane 6), X-Plane’s framerate would suffer in two ways as you cranked up the rendering options:
- For most features (more visibility, more autogen) as the CPU and GPU became more heavily loaded, the framerate would gradually decrease.
- If you ran out of VRAM (that is, the working set of textures needed per frame was more than your card’s VRAM) framerate would really die fast – think 2 fps.
The reason for this second behavior was that computers couldn’t shuffle textures between main memory and VRAM fast enough to render a frame in a 30th of a second.
As computers have gotten faster, this second behavior has gone away – modern cards, with fast PCIe 16x busses, can transfer textures from system memory to VRAM pretty fast – fast enough to have the working set be (slightly) larger than VRAM and still fly. So as texture memory increases, framerate decreases more gradually.
However, a new behavior has emerged: “glitching”. You may have noticed that when you’ve got your computer set to the ragged edge of the rendering settings, as you turn the camera, the framerate will stutter for a few frames, then return to a relatively high rate (40-50 fps).
What’s happening is: the working set of textures and geometry needed by X-Plane just barely fit in VRAM. But when you turn your head, a different set of textures and geometry are needed. While the card sorts out what is needed and what isn’t, it spends some time needlessly shuffling textures, and eventually reaches stability, with only what’s needed in VRAM, and framerate stays high.
Glitching has emerged as a mode of performance degredation because over time we’ve cut down the amount of “stuff” (textures and geometry) x-plane needs to draw a frame to only what’s really absolutely needed. This means there is less intersection between the working set in one view and another, and it also means you can get closer to the edge of your hardware.
So my view on glitching is basically “too bad”. If the working set weren’t as carefully trimmed, you wouldn’t have glitches, but the framerate would be entirely low, not entirely high. The only solution is to turn down settings that increase the working set (object density, world LOD, tex res, forests…) until the computer can run without glitches.
An even stranger variant of this: users sometimes report framerate getting “stuck” at 19 fps and then coming back when they change apps. The problem is that the driver doesn’t know exactly what the best order of textures to keep in VRAM vs. shuffle is…as the view changes, sometimes the driver ends up with a non-optimal decision about what stays in VRAM and what goes, causing framerate to drop. Changing which app is in the foreground fixes this by temporarily pushing a lot of items out of VRAM, at which point the driver makes a different decision by luck.
Again, the solution is simple: turn down rendering settings to get the working set smaller than VRAM.
Basically, if the working set is smaller than your amount of VRAM, you should have even framerates, proportional to rendering load.
If the working set is greater than VRAM, the driver may find an optimal way to shuffle things and only decrease fps a little, or it may find a non-optimal way to shuffle and you’ll get terrible fps, and that shuffle can change on the fly, causing framerate to fluctuate all over the place.
