Category: Hardware

Clouds, Screws, and Settings

I’m listening to Marco Arment and Dan Benjamin debate whether a user setting could have avoided the incident where a man’s iPhone rang during a concert.  I also just finished reading the Steve Jobs biography.  It got me thinking about the trade-offs between a closed non-configurable system and an open, configurable one.  This comes up in an X-plane context every time we poke at the cloud rendering settings and someone asks whether we can put in more settings for additional configuration.  What trade-off is right for X-Plane?

Chris suggested an analogy to me a few weeks ago that I think is pretty good: tuning the rendering engine in X-Plane should be like tuning the timing in your car.  It should be possible, if you are a serious expert and enthusiast, to change the tuning (and if doing so wrecks your engine, well, you were asking for it by opening up the engine), but it shouldn’t be easy.  Regular users of the car expect the manufacturer to have a lot more expertise, and to set the timing to an optimal setting up front.

I think the same thing goes for X-Plane.  I should use as much expertise as I have to maximize efficiency no matter what trade-off is being made; at no point should dialing in a setting result in stupid behavior inside the engine.  It’s up to me to know what can be inefficient and avoid that happening.

So: the rendering settings are going to remain as simple as possible, and any time we can make them more simple, less likely to be set up wrong, or any time we can realign them to be better aligned with user constructs (“more houses”) instead of internal constructs (“higher instancing density ratio!”) we’ll take that win.  It should not be the user’s job to tune the engine, and any time that is necessary, that’s a failure by me to pre-tune the sim, not a feature.

I don’t think we are even close to the kind of rendering settings that are really useful to non-expert users; one of the frustrations with performance tuning over the last few months has been the number of times a user has sent me rendering settings and it was clear that the settings were inefficient for the given hardware.

The answer is not to try to make every user into an expert in the OpenGL pipeline; the answer is to change the settings so that users don’t have to be experts.  This is not easy to do; the reason it’s not done yet is that we will have to create new code and do real hard work to make the interface simple.  One theme that both Steve Jobs and Jonathan Ive describe when discussing their design philosophy is that, to make something simple in its final product, you have to dig down and master the complexity that you want to get rid of.  To make the rendering settings simple but still powerful, we are going to have to solve these performance-efficiency-hardware problems ourselves and encapsulate that knowledge in code, and that’s not trivial.

Underneath the settings are a whole pile of art controls, and if you really want to, you’re more than welcome to enter any value you want in there.  If you discover that you’re better at tuning the engine than I am, I will be very happy to hear what you did!

This attitude is not as extreme as the “Steve Jobs” approach; Apple devices often use custom screws to ensure that normal people can’t open their devices even if they want to.  We are hiding implementation out of sight, but we are not locking that implementation at all.  The settings file is plain text, the art controls are viewable with a publicy available plugin (DataRef Editor).

X-Plane has always been a hackable sim, and one of the things I used to enjoy back in X-Plane 6 was just looking at all the stuff in the resources folder; the raw textures were pretty amazing to view in themselves.

My goal here is not to take that away; only to make hacking and opening up the sim something you can opt into if you are curious, instead of something mandatory to get good performance.

Mobile GPUs and Fill Rate

Fill rate: when we talk about fill rate, generally what we’re talking about is your GPU’s ability to fill in individual pixels to make the final image you see while flying.  (The actual components of GPU throughput get a lot more complex, but we’ll keep things simple for now.)  Two key things to note about fill rate:

1. It only comes from your GPU – not your CPU.
2. The bigger the size of X-Plane’s window (or the higher the full screen monitor res), the more of it you use.

Therefore I can offer this very simple test as to whether you are fill-rate limited:

If you are running in windowed mode and making the window smaller improves frame-rate, you are fill-rate limited.

(There is actually one important exception to the above test: if you are running out of VRAM, making the window smaller uses less VRAM for the main window + off-screen buffers, which can help fps.  So be sure to see whether cutting your texture res helps fps – if it does, you are out of VRAM.)

Fill rate gets used up by:

• A larger rendering window.  This is the number one consumer of fill-rate.  Double the size of the window in each dimension, you use 4x the fill-rate.  That adds up quick!!
• Clouds tend to be fill-rate intensive.
• HDR mode is significantly more fill-rate intensive than non-HDR mode.
• The 3-d cockpit is more fill-rate intensive than 2-d and external views, particularly with HDR enabled.
• Shadows, to a limited extent.

X-Plane 10 contains a stat called “cpu load” in the framerate output data line; I will describe its full meaning and implication in another blog post, but generally low CPU load (e.g. < 0.7) means you are fill-rate limited.  If your CPU use for one core is 100% (this means 25% in the task manager for Windows users on a 4-core i5, but 100% in top for Mac/Linux users no matter what the number of cores) then you are almost certainly not fill-rate limited.

Optimizing Fill Rate

X-Plane 10.03r1 will post in a little bit – this is our first attempt to finalize X-Plane 10.03.  It is not the end of the bug fix and performance tuning work for X-Plane 10.  But if we can get a stable 10.03 we can use it as a baseline and limit betas to those who check the “get betas” check-box in the installer.

X-Plane 10.03r1 contains fill-rate optimizations for the 2-d and forward-no-HUD views; we’ll have to see how much they help people, but they should help you a bit if you are fill rate limited, flying a plane with a 2-d cockpit, and are seeing low fps with clouds and/or HDR.

Once 10.03 is final, we’ll start 10.04 and I will attempt to make a similar set of optimizations for the 3-d cockpit.  Note that the 2-d optimizations do not apply if you have a plane that uses the 3-d cockpit for the 2-d view.  (Since all of the new X-Plane 10 default planes do this, we’ll need the 3-d optimizations in 10.04 to see the performance benefit with the 747, Baron, and other new default planes.  The Cessna, Cirrus Jet, and Piaggio from X-Plane 9 should all benefit immediately.)

Your Mobile GPU Is Light But Not Fast

Here is a table I culled from Wikipedia: it is a table of GPU performance for the “top end” cards for ATI’s last 4 generations of hardware.

GP = gigapixels/second of fill, GT = gigatexels/second of texture, BW = GB/second of memory bus bandwidht, and GFLOPS = gigaflops/second of MADD.  At least I hope – see the original table.

Now I have inserted one mobile GPU (the 6970M) into this chart.  So the first thing you can see is: the 6970M provides in a laptop the power that was available in a desktop about two years earlier, more or less.  That’s a big step back in time.

The reason is simple: mobile GPUs are typically cut down in their core configuration – there’s no way you can run a 150W full power GPU with those 3 fans in a laptop.

If you have an iMac, you have a mobile GPU.  Even the older iMacs which didn’t have the “M” designation on their chip, those were mobile too.  In fact, you’ve got a mobile GPU on a huge screen.  (Super-size to the 27″ iMac to get the 4x fill rate boost and you pick up 1.7x the pixels; if you get the big iMac with the cheap GPU you have a mid-range mobile GPU on a 2560 x 1440 screen!)

Some Kind Of Summary

I’m not entirely sure what my main point here is…perhaps the executive summary is:

• We (LR) are working to optimize fill rate in X-Plane 10.  Fill rate is not the only performance bottleneck we are working on, but it is a very important one.  Some of this is in X-Plane 10.03r1, some will come in later updates.
• You can determine whether you are fill-rate limited by resizing your window.
• If you have a mobile GPU in a laptop or iMac you may have less fill rate than you would expect.

What’s This Whole PCIe Thing About, Anyway?

I’ll try to summarize some of our hardware findings for X-Plane 10 over the next few posts.  But in my previous post I mentioned that the new MacBook Pros have only an 8x PCIe connection to the discrete GPU (that is, the nice GPU that isn’t built in to the CPU, the one you want to fly X-Plane with) and this got a bit of attention.

So it begs the question: what is this PCIe bus and why do we need to  care all of a sudden?

The PCIe bus is the connection between the CPU/main memory and your graphics card (with its memory and GPU).  It is the bottleneck through which all communications must flow – sometimes every frame, sometimes every now and then.

PCIe slots are named by the number of lanes (e.g. 16x means 16 lanes) – each lane has fixed capacity (which is doubled in PCIe 2.0).  So a graphics card in a 16x slot drink data from your computer at double the rate of one in an 8x slot – it’s an extra wide straw.

(Nerds: I realize this is about the worst description of the PCIe bus you will ever find.  Go read Wikipedia!)

What Do We Use the PCIe Bus For?

X-Plane needs the PCIe bus to:

• Send the instructions to draw each frame to the GPU.
• Transfer any textures, new OBJ meshes, and other data that will be held in VRAM.  The data is born on the CPU, goes over the PCIe bus once, and then lives in VRAM.
• Send to the GPU anything that changes every frame to the GPU.  For example, smoke puffs and car headlights have to go over the PCIe bus every frame because they are constantly changing.
• Send to the GPU mountains, forests and other non-repeating geometry.  This data gets sent every frame.

If the sum of all of the stuff on that list gets too big, your framerate drops as the CPU and GPU both wait data to make it over the bus.  In other words, the bus can at times be the bottleneck in terms of framerate.

If you set your rendering settings near the maximum that your computer can handle and get the occaisional stutter, that may be X-Plane running out of PCIe bus bandwidth.  As you fly to a region with new textures that haven’t been used before, the OpenGL driver will transfer our textures over the PCIe bus from system RAM to VRAM.  If the PCIe bus is already nearly maxed out, the extra traffic of those textures is going to temporarily hurt framerate – sometimes in the form of a stutter or pause.

Are You Sure You Know What You’re Doing?

At this point those of you who know some things about 3-d graphics are shouting at your monitors: why are you guys transferring the mountains and forests over the PCIe bus every frame?  Why not just put them in VRAM, since they don’t change?

That’s a good question and if you have a better solution than the one we use, I’d love to hear it.

The problem is this: OpenGL doesn’t give us a good way to prioritize which meshes (VBOs) stay in VRAM and which ones are purged out when we run out of VRAM.  If we put every mesh in the sim into VRAM, framerate gets better (because we aren’t using the PCIe bus) right up until we run out of VRAM.  At that point the OpenGL driver freaks out and starts throwing out textures to make room for meshes, and then the textures have to be sent back over the PCIe bus, and we end up in a world of hurt.  We end up in a state of texture thrash as we have too much “stuff” for VRAM and framerate falls off of a cliff.

The real problem is this: X-Plane has no idea how much VRAM is available for its own use.  Sure the card might have 256 MB, but how much is being used by the OS window manager for those translucent window effects, or by other applications?  We can’t even add up how much VRAM we use with ultimate precision because we don’t know the granularity of allocation on the video card (there’s real overhead for VBOs being rounded up to the VM page size, for example) or whether side buffers like a hierarchial Z buffer have been allocated.

X-Plane works around this with a simple rule: all OBJs go to VRAM, because their geometry is likely to be repeated, and non-repeating geometry, like forests and mountains, stay in system RAM and go over the bus.

This heuristic actually works pretty well in X-Plane 9 – we have enough bandwidth to transfer all of that “stuff” once per frame, and we tend not to run out of VRAM and thrash.

Why Does X-Plane 10 Want more PCIe Bus Bandwidth?

X-Plane 10 is hungrier for bus bandwidth for three reasons:

1. The OBJ engine’s performance has been improved a lot.  In the past, we’d run out of CPU capacity (to draw OBJs) long before we ran out of bus bandwidth.  This isn’t always the case with X-Plane 10.  The graphics are always held back by their weakest link.  If you have a strong GPU (and low effects settings) and the OBJ engine is efficent, the PCIe bus is the weakest link.
2. The art assets are more detailed and thus contain more vertices.
3. Shadows.

When shadowing is on we have to draw the entire world multiple times, once to build shadow maps and once to draw the real world.  So shadowing can double (or even triple or worse) our bus bandwidth usage.  We didn’t have that kind of free capacity on the bus in the first place.

We’re still working on the engine, art assets, and performance, so my hope is that we’ll find ways to cut down bus use (especially with shadows).  And there has to be one slowest part of the system – as of this writing, the PCIe bus is often it.

X-Plane, SLI and CrossFire

In my previous post I described some of our findings for X-Plane 10 with respect to GPUs.  This begs the question we get asked all the time: what about SLI/Crossfire?

For those of you not familiar with SLI and CrossFire, they are technologies (from NVidia and AMD respectively) that use two graphics cards to share the load of drawing a 3-d scene.  The idea is to double the shading power of your graphics system by having each card render every alternate frame.*  You can get SLI/CrossFire with two GPUs and an appropriate motherboard, or by buying one of the monstrous “x2” GPUs.

Now to try to answer some often-asked questions:

Can X-Plane take advantage of Crossfire/SLI?

No.  Neither X-Plane 9, nor 10.0, will be capable of running with Crossfire or SLI.  From what I can tell, there are a few sites in the code that need to be reworked a bit to be ready for these technologies.  If you have an SLI/Crossfire setup, I would expect you to get the same framerate with only one of the GPUs.

Will X-Plane 10 ever take advantage of Crossfire/SLI?

Maybe someday.  If we can get our code clean enough to work with these technologies, I will post an update on this blog.  But I can’t promise anything – for all I know we may someday hit some horrible show-stopping problem.

Would SLI/Crossfire be useful for X-Plane?

Not for X-plane 9.  X-Plane 9 can’t even max out a single GPU from the previous generation’s top end cards, so there’s really no need for two of them.

For now there are resolutions where two GPUs would be necessary to get better frame-rate.  If we can run 2560 x 1440 at 20-30 fps on one card, it would in theory be nice to run at 40-60 fps on two.

But the next generation of GPUs is on the way, so it may be that the next generation of cards will be fast enough on their own.

Should I Buy an SLI/Crossfire setup?

Not for X-Plane – we can’t take advantage of it right now.

Here’s another way to look at it.

• If you wanted 2400 GFLOPs on your GPU in August of 2008, you could buy the Radeon HD 4870 x2, which is basically two 4870s jammed together on a single card via Crossfire.  This would set you back $549.
• 13 months later, in September of 2009, AMD released the 5870, which could put out 2720 GFLOPs as a single GPU for $400.

The performance curve for GPU power is really quite steep, and a dual-GPU system typically costs at least twice as much as two of the single-GPU form.  That’s a lot of money to pay for fill rate that will be available in about 18 months in single-card form.

So the short version is: we don’t support SLI/Crossfire yet.  Someday we may work with these technologies, but even if/when we do, they’ll only make sense if you really like high resolutions and framerates and money isn’t an object.

* SLI can operate in “split-frame” mode where each video card takes half of a frame, but this is not so good because a full frame of geometry must be sent to each card for each frame, effectively doubling PCIe bus use.  In alternate-frame mode, each card only sees every other frame, and thus only needs half the bandwidth – both cards together use the same bandwidth as one card would have.