Per-pixel lighting is something I hope to have in X-Plane soon. A number of other features will take longer, and quite possibly might never happen. This is the “pie in the sky” list – with this list, we’re looking at higher hardware requirements, a lot of development time, and potential fundamental problems in the rendering algorithm!
High Dynamic Range (HDR) Lighting
HDR is a process whereby a program renders its scene with super bright and super dark regions, using a more detailed frame-buffer to draw. When it comes time to show the image, some kind of “mapping” algorithm then represents that image using the limited contrast available on a computer monitor. Typical approaches include:
- Scaling the brightness of the scene to mimic what our eyes do in dark or bright scenes.
- Creating “bloom”, or blown out white regions, around very bright areas.
Besides creating more plausible lighting, the mathematics behind an HDR render would also potentially improve the look of lit textures when they are far away. (Right now, a lit and dark pixel are blended to make semi-lit pixels when far away as the texture scales down. If a lit pixel can be “super-bright” it will still look bright even after such blending.)
Besides development time, HDR requires serious hardware; the process of drawing to a framebuffer with the range to draw chews up a lot of GPU power, so HDR would be appropriate for a card like the GeForce 8800.
While there aren’t any technical hurdles to stop us from implementing HDR, I must point out that, given a number of the “art” features of X-Plane like the sun glare, HDR might not be as noticeable as you’d think. For example, our sun “glares” when you look at it (similar to an HDR trick), but this is done simply by us detecting the view angle and drawing the glare in.
Reflection Mapped Airplanes
Reflection maps are textures of the environment that are mapped onto the airplane to create the appearance of a shiny reflective surface. We already have one reflection map: the sky and possibly scenery are mapped onto the water to create water reflections.
Reflection maps are very much possible, but they are also very expensive; we have to go through a drawing pass to prepare each one. And reflection maps for 3-d objects like airplanes usually have to be done via cube maps, which means six environment maps!
There’s a lot of room for cheating when it comes to environment maps. For example: rendering environment maps with pre-made images or with simplified worlds.
Shadows
Shadows are the biggest missing feature in the sim’s rendering path, and they are also by far the hardest to code. I always hesitate to announce any in-progress code because there is a risk it won’t work. But in this case I can do so safely:
I have already coded global
shadow maps, and we are not going to enable it in X-Plane. The technique just doesn’t work. The code has been ripped out and I am going to have to try again with a different approach.
The problem with shadows is the combination of two unfortunate facts:
- The X-Plane world is very, very big and
- The human eye is very, very picky when it comes to shadows.
For reflections, we can cheat a lot — if we don’t get something quite right, the water waves hide a lot of sins. (To work on the water, I have to turn the waves completely off to see what I’ m doing!) By comparison, anything less than perfect shadows really sticks out.
Shadow maps fail for X-Plane because it’s a technology with limited resolution in a very large world. At best I could apply shadows to the nearest 500 – 1000 meters, which is nice for an airport, but still pretty useless for most situations.
(Lest someone send the paper to me, I already tried “TSM” – X-Plane is off by about a factor of 10 in shadow map res; TSM gives us about 50% better texture use, which isn’t even close.)
A user mentioned
stencil shadow volumes, which would be an alternative to shadow maps. I don’t think they’re viable for X-Plane; stencil shadow volumes require regenerating the shadow volumes any time the relative orientation of the shadow caster and the light source change; for a plane in flight this is every single plane. Given the complexity of planes that are being created, I believe that they would perform even worse than shadow maps; where shadow maps run out of resolution, stencil shadow volumes would bury the CPU and PCIe bus with per-frame geometry. Stencil shadow volumes also have the problem of not shadowing correctly for alpha-based transparent geometry.
(Theoretically geometry shaders could be used to generate stencil shadow volumes; in practice, geometry shaders have their own performance/throughput limitations – see below for more.)
Shadows matter a lot, and I am sure I will burn a lot more of my developer time working on them. But I can also say that they’re about the hardest rendering problem I’m looking at.
Dynamic Tessellation
Finally, I’ve spent some time looking at graphics-card based tessellation. This is a process whereby the graphics card splits triangles into more triangles to make curved surfaces look more round. The advantage of this would be lower triangle counts – the graphics card can split only the triangles that are close to the foreground for super-round surfaces.
The problem with dynamic tessellation is that the performance of the hardware is not yet that good. I tried implementing tessellation using geometry shaders, and the performance is poor enough that you’d be better off simply using more triangles (which is what everyone does now).
I still have hopes for this; ATI’s Radeon HD cards have a hardware tessellator and from what I’ve heard its performance is very good. If this kind of functionality ends up in the DirectX 11 specification, we’ll see comparable hardware on nVidia’s side and an OpenGL extension.
(I will comment more on this later, but: X-Plane does not use DirectX – we use OpenGL. We have no plans to switch from OpenGL to DirectX, or to drop support for Linux or the Mac. Do not panic! I mention DirectX 11 only because ATI and nVidia pay attention to the DirectX specification and thus functionality in DirectX tends to be functionality that is available on all modern cards. We will use new features when they are available via OpenGL drivers, which usually happens within a few months of the cards being released, if not sooner.)
The 921 patch is now available. This fixes the bad-alloc problems relating to complex airports.
Please note: if you have a bad-alloc crash, it could be because you are out of memory. Make sure you have virtual memory turned on, your page file is large enough, your disk isn’t full, etc. If you have a bad alloc error, try the sim without third party add-ons to see if you really are running out of memory. If you are running Vista or XP, use /3GB or the BCD – see this for more info. Basically when you are running out of memory, you either can crash on “bad alloc” if we need memory for the CPU or “we ran out of video card memory” if we can’t map geometry into virtual memory.
If you get a crash with 921 on OS X, please let me know by email! I’ve seen one of these reports.
I also have cuts of the new installer/updater suite, version 2.05 – if you are going to update, tryo ne of these:
http://dev.x-plane.com/update/installers9/stage/
The main features of the new installer are:
- Support for French, Spanish, Italian, German, and Russian.
- Clearer colors on the world map.
- Scanning the global scenery folder to add/remove scenery is much faster.
I recommedn the new installers for that last point alone.
The triangle is at the heart of 3-d modeling – but before we discuss what might become of the triangle, we need terminology.
- Per-vertex lighting. This means that the brightness of the model (a function of the sun and camera position, etc.) is calculated for each vertex in the model, and then crudely interpolated between the vertices to light the pixels.
- Per-pixel lighting. This means that the brightness of the model (a function of the sun and camera position, etc.) is calculated for every pixel on the screen separately.
- Tessellation. This is the process of splitting a triangle into a number of smaller triangles, increasing the number of vertices in a model.
- Specular lighting. The specular lighting component is an extra amount of brightness that you get when the angle from the sun to the model to your eye is very small. (That is, if the model was a mirror and you could see the sun by looking at a certain location, then that location would have a bright “specular hilite”.)
- Normal map. A normal map is a texture that describes the way light bounces off a surface. This is one way to do “bump mapping”. This tutorial shows a pretty good example of how normal maps work. (The earth orbit textures in version 9 use normal maps to create “bumpy” mountains when pixel shaders are in use.)
- Specular map. A specular map is a texture that describes how strong the specular component of the lighting model appears for a given textured location. Here’s another tutorial that explains it.
- Environment Map. An environment map is a texture that represents the world around an object, used to simulate reflections. Here’s another blender tutorial that explains it better than I. (The reflective water in X-Plane 9 is effectively using a dynamic environment map created by taking a picture of part of the sim’s world every frame,)
- Material attributes. These are OBJ attributes that change the lighting model. For example, ATTR_shiny_rat changes the lighting model so that specular hilites appear.
- Batch. A batch is a single set of triangles sent to the graphics card without any change of mode. Basically every TRIS command in an OBJ becomes a batch; submitting a batch requires the CPU, but submitting a bigger batch (more triangles) does not require more CPU.
That’s enough vocabulary to describe just about everything that is happening now, will be happening in the future, as well as some pie-in-the-sky stuff. 🙂
Traditionally, a pilot’s priorities are: aviate, navigate, communicate.
But that might not be true for X-Plane for the iPhone.
It’s real! And it pretty much is X-Plane – there really are OBJs and DSFs in there, as well as an ACF model, all tuned for the iPhone.
In the next few posts I’ll blog a little bit about the impact of doing an iPhone port on scenery development. The iPhone is an embedded device; if you go digging for system specs you’ll see that it’s a very different beast from the desktop. The porting process really helped me understand the problems of the rendering engine a lot better, and some of the techniques we developed for the iPhone are proving useful for desktop machines as well.
Yesterday I went off on a rant about how the OpenGL 3.0 spec doesn’t matter a lot because OpenGL grows via a la carte extension. And I mentioned that this creates a dilemma for anyone designing a rendering engine that has to run on a wide range of hardware, with very different underlying capabilities.
Back in the old days of X-Plane 6, there wasn’t a lot of variety in hardware. Some cards were faster than others, but the only real difference in capability was how many texture units you had. X-Plane’s job was simple…
- First we have a runway texture.
- Still got a second texure unit? Great! Skid marks!
- Still got a third texture unit? Great! Landing light!
- Got another one? Etc…
Other than texture stacking, there wasn’t much to do.
Since then the rendering engine has become a lot more complex, as have OpenGL cards. To try to keep the combinations down, I tried to use a “bucketing” strategy for X-Plane 9. The idea of bucketing is to group cards into major buckets based on whole sets of common functionality, so that we only have to test a few configurations (the “low end” bucket and “high end” bucket), rather than having to test every single card for its own unique set of features.
The obvious bucketing choice was pixel shaders – given a card with shaders and a few other features, we can render all of the new effects. A card without shaders basically gets X-Plane 8.
So what went wrong? Driver compatibility, that’s what. Ideally we don’t want to allow every single underlying rendering engine feature to be turned on or off individually because the combinations are uncontrollable.* But in practice, being able to turn features on and off is necessary to trouble-shoot drivers that don’t always do what we expect them to.
With the GeForce 8800 and Radeon HD cards, there is a potential third bucket for DirectX-10 capable cards, capable of significantly more advanced pixel shading effects. But time will tell whether we can actually make a bucket, or we have to look at each feature individually. My suspicion is that even if we organize the shipping features into buckets, we’ll have to support a lot of combinations under the hood just to trouble-shoot our own application.
*Example: cross a standard vs. oversized panel with the presence or absence of advanced framebuffer blending, crossed with whether render-to-texture works. That’s 8 possible ways just to render the 3-d cockpit panel texture. Cross that with panel regions and 3-d cockpits and the new panel spotlighting and you have 64 configurations. Ouch!
A few people have asked me about OpenGL 3.0 – and if you read some of the news coverage of the OpenGL community, you’d think the sky was falling. In particular, a bunch of OpenGL developers posted their unhappiness that the spec had prioritized compatibility over new features. Here’s my take on OpenGL 3.0:
First, major revisions to the OpenGL specification simply don’t matter that much. OpenGL grows by extensions – that is, incremental a la carte additions to what OpenGL can do. Eventually the more important ones become part of a new spec. But the extensions almost always come before the spec. So what really matters for OpenGL is: are extensions coming out quickly enough to support new hardware to its fullest capacity? Are the extensions cross-vendor so that applications don’t have to code to specific cards? Is the real implementation of high quality?
So how are we doing with extensions? My answer would be: “okay”. When the GeForce 8800 first came out, the OpenGL extensions that provide DirectX 10-like functionality were NVidia-specific. Since then, it has become clear that all of this functionality will make it into cross-platform extensions, the core spec, or some of each. But for early adopters there was a difficult point where there was no guarantee that ATI and NVidia’s DirectX 10 features would be accessible through the same extensions.
(This was not as much of an issue for DX9-like features, e.g. the first generation of truly programmable cards. NVidia had a bunch of proprietary additional extensions designed to make the GeForce FX series less slow, but the basic cross-platform shader interface was available everywhere.)
Of more concern to me is the quality of OpenGL implementations – and for what it’s worth, I have not found cases where a missing API is standing between me and the hardware. A number of developers have posted concern that OpenGL drivers are made too complex (and thus too unreliable or slow or expensive to maintain) because the OpenGL spec has too many old features. I have to leave that to the driver writers themselves to decide! But when we profile X-Plane, we either see a driver that’s very fast, or a driver that’s being slow on a modern code path, in a way that is simply buggy.
Finally, I may be biased by the particular application I work on, but new APIs that replace the old ones don’t do me a lot of good unless they get me better performance. X-Plane runs on a wide range of hardware; we can’t drop cards that don’t support the latest and greatest APIs. So let’s imagine that OpenGL 3.0 contained some of the features whose absence generated such fury. Now if I want to take advantage of these features, I need to code that part of the rendering engine twice: once with the new implementation and once with the old implementation. If that doesn’t get me better speed, I don’t want the extra code and complexity and wider matrix of cases to debug and test.
In particular, the dilemma for anyone designing a renderer on top of modern OpenGL cards is: how to create an implementation that is efficient on hardware whose capabilities is so different. I’ll comment on that more in my next post. But for the purposes of OpenGL 3.0: I’m not in a position to drop support for old implementations of the GL, so it doesn’t bother me at all that the spec doesn’t drop support either.
The real test for OpenGL is not when a major revision is published; it is when the next generation of hardware comes out.
I just received a series of reports today that certain converted scenery will cause X-Plane to crash with a “bad alloc” error. Basically, this couldn’t have hit us at a worse time. The final 920 was cut a week ago. We physically can’t recut; Austin is on the road, and I am knee deep in it. But there is a possible work-around, and there will be a patch. Here’s the whole situation.
What is a Bad Alloc?
A bad alloc error is an error that comes up when X-Plane runs out of memory. This can happen for two reasons:
- We have run out of address space – that is, there is no more virtual memory left, or
- We have run out of page file/physical memory – that is, we can’t back that virtual memory.
The first case is by far the most common – you’d only hit the second if you are on Windows with a fixed-size (but small) page file. (Hint: if you have a fixed size page file, make it big!)
X-Plane can run out of memory for many reasons – everything that runs in the sim uses memory, and the amount used depends on what area you are in, what rendering settings you pick, and what third party add-ons you use. While I’d like to someday reach a point when the sim tells you gracefully that it’s out of memory, it will always be a fact of life that at some point (hopefully an absurdly high one) the amount of stuff you’ve asked X-Plane to do will exceed how much memory you have.
(If you are thinking 64 bits, well, that will just change the problem from a crash to a grinding halt when we run out of physical memory.)
We see bad allocs when there are too many third party add-ons installed (XSquawkBox is a particular pig because it loads every CSL on startup), too complex scenery, and it can also be caused by drivers not efficiently using memory. (This is particularly a problem on Vista RTM.)
The Bug
When X-Plane creates a curved airport taxiway, it allocates a temporary memory buffer to hold the intermediate product of the pavement. The size of that buffer depends on the complexity of the curve it is processing and a constant, based on the maximum curve smoothness.
In 920 I provided an option to crank up the curve smoothness in X-Plane. In the process, I increased that constant factor by 4x, which causes X-Plane to hit its memory ceiling on layouts that used to be acceptable. You’ll see this problem more often on:
- Bigger, more complex layouts.
- Configurations that were already chewing up a lot of memory.
- Machines with less address space (Windows without /3GB, older Mac OS X operating systems.)
What really suckered us about this bug was that it comes in a form that looks almost the same as a driver issue we’ve seen with ATI drivers on Windows — we’ve seen strange forms of memory exhaustion on ATI when shifting scenery with high rendering settings. So we didn’t realize that this was something new until G5 users reported the bug (making us realize it wasn’t a driver thing).
What To Do
The bad news is that we can’t do an RC5 – we’re out of time. But – there will be a patch – relatively soon. This bug is on the short list for a patch to fix 920.
In the meantime, there is actually a work-around. By coincidence, some of the internal rendering engine constants are viewable via the “private dataref” system — basically a series of datarefs in the sim/private/… domain that I use for on-the-fly debugging. The dataref that matters here is:
sim/private/airport/recurse_depth
If you load up DataRef Editor you’ll see it has a value of 12 . That’s too high. Changing it to 10 will allow otherwise problematic airports to load.
I will try to post a plugin in the next 10 days that sets this dataref to 10 on startup, effectively patching the problem. This will also limit the maximum smoothness of curves – but my guess is that if you see the crash (not all users do) then you can’t run on the max airport curve setting anyway.
Of course the next patch will contain a real solution: a more efficient memory allocation scheme!
Some users reported during 920 beta that X-Plane would sometimes not detect its DVD – a condition that would come and go. Tonight I figured out what is happening.
- In order to validate the DVD, X-Plane decompresses part of its contents into the preferences folder. Why preferences? There is no good reason – it’s historic.
- X-Plane will create a preferences folder if there is not one. But it does not do that until you quit.
- The X-Plane installer will not make directories unless they contain files.
So put these three things together: on the first run of a new install, there is no preferences file, so the DVD check fails since the directory that will contain some temporary files is missing. Run a second time, and the directory is there and the DVD check succeeds.
The next patch of the sim will fix this, but in the meantime, if you delete your preferences, leave the empty directory in place!
Two random and unrelated notes:
First, RC4 is going out as is, despite the engine modeling changes being incomplete. Basically we now have a more sane approach to the engines themselves, but no FADEC control. FADECs are on the short list for the next update. Sometimes we just run out of time – not every release can have everything.
Second, a note on autopilot customization – I am party to a fair number of questions about whether the plugin system can be used to make subtle changes to the autopilot logic. The answer is of course: no. If you really want something different for an autopilot, you’d have to replace the entire “top-half” set of logic and drive the flight directors yourself – in this situation you are responsible for:
- All modes and mode changes based on conditions.
- The actual selected flight envelope to achieve the desired AP setting.
But you are not responsible for driving the trim and yoke, which are done by you setting the flight director.
Why can’t you just override one specific behavior? It’s an issue of infrastructure.
Fundamentally, the autopilot only does a few certain tricks. If it were capable of doing customized behaviors, you’d already see it, in the form of a dataref or (more likely) a Plane-Maker setting. Basically there is no generality to the autopilot that we secretly have inside the code but don’t expose.
Will there be a more general autopilot someday? Maybe – I don’t know, I don’t work on that code. But the plugin system has always aimed to make it possible to do anything, but not necessarily easy. In particular, the plugin system doesn’t aim to make your development easier by recycling the simulator itself as a convenient library of lego bricks. In the end of the day, X-Plane is an application, not a library. If it were a library, that would be lots of fun for third parties, but it is not.
I believe I am getting close to a possible solution for the dreaded “Framebuffer Incomplete” errors – these error messages pop up when X-Plane starts, and you end up quitting.
If you meet these criteria, please contact me:
- You have an ATI card that has shown this error in the past.
- You can put on the latest Catalyst drivers. (I know a lot of you have put on older drivers to work around this.)
- You can run X-Plane 920 RC2.
If you’re in this crew, please email me at my XSquawkBox email address!
The rub is: despite having four machines with ATI cards, I never see this error. So I need to send you a build to get close to a fix!!! Let’s swat this bug for real!
