“We eat our own dog food” is a term in software development for a company that uses the product it sells for its own operations. The idea of “dogfooding” is that developers who use their own software will have much stronger incentives to make the software usable, productive, and reliable.
Laminar Research practices a form of dogfooding with the scenery tools:
All of the scenery tools are open source, so we don’t have any “special secret magic tools” that you don’t have.
Our art team uses the same set of tools that third parties use.
Dogfooding does have an effect on development: when the artists get on my case about the usability of a scenery tool, it gets my attention. We pay those guys to churn out a lot of artwork; if they aren’t productive, we can’t get the sim out.
However, there is an Achilles heal to dogfooding: you can only dogfood software that is useful for your company’s business. We have this problem with MeshTool: since we don’t make our own orthophoto sceneries, MeshTool doesn’t get heavy internal use, and as a result I think it’s been a lot slower to become usable. The MT users who have filed bugs have been very helpful, but I feel for their suffering with a very raw tool.
Dog Food and Version 10
For version 10, we are producing art assets that use new rendering technology for the very first version. This means we’ve had to build the tools to support new version 10 technology early on, and that’s good news for third party developers. When version 10 is released, I’ll package up the various version 10 tools we’ve created for public consumption.
In most cases, the tools need additional polish – our artists often use tools in very early, buggy, raw form. But what we already have is a big head start, which should shorten the time from version 10 release to a full set of tools.
I’ll post more over the next few days regarding some of the specific tools we have and what we will do with them once version 10 ships.
The billboarding scheme for cloud puffs has been rewritten for X-Plane 10; for the most part you should not see puffs spinning or reorienting themselves as you move the camera. However, while poking at the cloud puff shader, I managed to induce this picture on the left. The weather system makes some of the best bug screenshots.
(The post was originally labeled “bug or tornado”…I changed it because it seemed obnoxious after what happened in Joplin, MO – just me being a jerk from the East Coast. A few days later we were hit by a series of Tornadoes fairly close to home.)
Bad preferences have become a superstition in the X-Plane community. If anything goes wrong, the first thing anyone says is “delete your preferences”. This often “fixes” the problem – in that it undoes all of your careful settings and changes about 100 different things, one of which changes enough behavior to fix the problem. It’s the medical equivalent of curing the common cold by transplanting every single organ.
Of course, every now an then the preferences file is the problem. It turns out that X-Plane 9.6x and 9.31 use the same version ID for the binary preferences file, and yet they use different binary preferences file formats. This was a screw-up on our part, and the result was a hang on start for 9.6x users if they had their old 9.31 preferences laying around. In this case, nuking preferences really was the problem. (Thanks to Andy for getting me the files to see this “in the lab”!)
Send Me Your Preferences!
What is frustrating about the 9.31/9.61 preferences bug is that people have known about it forever, and yet I only received a real bug report about it this week. Part of the problem is how people cope with these bugs: by deleting their preferences.
Don’t delete your preferences! Once you delete the preferences files, they are gone! If there was something wrong with them, we’ll never know.
Deleting your preferences is like burning all of your belongings when you get a cold: the Doctor won’t know what bacterium or virus you have because you destroyed every instance of it!
You don’t have to delete your preferences. You can simply move them out of the preferences folder onto your desktop. This has the same effect as deleting them: X-Plane will not find them in the new location, and will start up with its default settings.
What if moving preferences fixes the problem? Well, now you have two sets of preferences: the old “sick” preferences on the desktop that cause a problem and the new “rebuilt” ones that X-Plane made when you re-ran.
At this point you can now send me a bug report. It might go something like this:
You: Hey Ben, I was getting a crash on start. I moved my preferences files out to the desktop and it fixed it. Here are the bad preferences and here are the good ones, in two zip files.
Me: Thank you! That’s great! I will look at the two sets of preferences to see what changed. If there’ a sim bug I can identify it and find it.
You: So…do I get any kind of reward?
Me: Only the good feeling of knowing you’ve helped your fellow X-Plane users.
You: Screw you! I’m going to go use Google Earth Flight Simulator.
Okay, that didn’t go quite as well as I had hoped. But my take-away point is this: if there is a real problem with the preferences system, such that deleting preferences is necessary, we want to look at the files and fix it in the sim! In the long term this will lead to less need to delete preferences, which will save a lot of users a lot of re-doing settings.
Why Do Preferences Files Matter?
Why does moving your preferences files out of the way (so X-Plane can’t find them) sometimes help? There are a few causes that I’ve seen:
Incompatible or corrupt binary files. Sometimes the preferences files actually contain junk, although this is rare. In some cases, we don’t handle the preferences file version change and we need to fix the version check code (this is what we’re fixing in 970).
Sometimes a single particular setting is problematic for a particular machine. For example, your machine might have a driver problem that causes it to crash when “per pixel lighting” is on. If you remove the preferences and the default is to have that setting be off, then you “fix” the problem. In this case, a comparison of the “bad” preferences with the default ones can reveal settings that might be the problem.
Sometimes settings can be set while you fly, but cause problems when the sim starts with the settings turned on. This is rare, but with tweaky graphics drivers we sometimes do see this behavior. Sometimes a setting takes effect on restart, so it’s not until you restart with the new setting that the preferences file causes problems.
Either way, having the “bad” preferences to inspect is critical to catching any one of these cases!
In testing some ATC stuff today, I was curious why I saw smoke coming from the water on the left side of the runway. As I got closer, I found a KC-10 rising from the ground like a mighty Phoenix….struggling to get out of the controlling grips of Air Traffic Control. Ben said it reminds him of Free Willy.
And of course the usual disclaimer…the purpose of the video is to have a giggle at the AI plane’s bug…ignore the ugliness of the rest of the situation. The colored lines are used for debugging aircraft routing issued by ATC.
An author recently asked me what the relationship was between the size of PNG files on disk and the amount of VRAM X-Plane uses to store the PNG file as a texture. I actually get that question fairly frequently, and the answer isn’t totally obvious.
The first and most important point: the size of a texture on disk in an image file is not the same as its size in VRAM, so changing the file’s size on disk doesn’t save you any VRAM. (It does save you hard drive space, which might be interesting if you are building a huge orthophoto scenery, or maybe download size, which can be handy.)
The size of an image in VRAM is a function of its dimensions, the amount of color information, an whether texture compression is on. The formula is basically this:
size = 4/3 * width * height * bytes_per_pixel
The 4/3 term is due to mipmapping – we save progressively smaller versions of the texture to improve the speed and quality of texturing when your image is far away.
Bytes per pixel varies depending on whether texture compression is on:
RGB, or RGBA, no compression: 4 bytes per pixel.
RGB, compression on (or DXT1 compressed DDS files): 0.5 bytes per pixel.
RGBA, compression on (or DXT3/DXT5 compressed DDS files: 1 byte per pixel.
Alpha-only: 1 byte per pixel.
That first point might be a little bit surprising to authors: when compression is disabled, you don’t save 25% for getting rid of your alpha channel. This is because modern video cards round up the texture from 3 to 4 bytes to keep the image size a power of 2. It is still typically a win to drop the alpha channel if you’re not using it: X-Plane can fill the screen faster if it knows that there is no alpha (and thus no blending) and the RGB-only texture will be half sized when compression is on.
DDS File Size
DDS (Direct-Draw Surface) is a file format designed to mirror exactly what video card want in VRAM. As it turns out, a DDS file is a good representation of how much VRAM your image uses, because it is already in the format that the card wants – including the 4/3 term. So if you need a quick and dirty estimate of your texture’s VRAM use at extreme res, the DDS size isn’t a bad proxy. (Note that when LOAD_CENTER is in use, your texture is reduced in size most of the time.)
PNG File Size
PNG file size is unrelated to VRAM used; PNG files are internally compressed. In fact, PNG files have multiple possible internal encoding schemes; utilities like PngCrush try all of the encoding schemes to see which one reduces file size the most. The key is: none of this translates into VRAM savings. PngCrush may make your downloads faster, but it won’t make your add-ons use less VRAM.
When I tore my shoulder playing frisbee and went to the Doctor, he took an X-Ray. Why? Well, that’s obvious: my skin makes it hard for him to directly inspect what was wrong inside my shoulder; the X-Ray shows right through the skin – in that case, it revealed a torn ligament.
It’s the same way with X-Plane. If I want to see what’s really going on in the sim, using real test cases makes life hard. Our artists try hard to hide the underlying mechanism of the sim and instead create a seamless plausible experience. In a good scenery pack you can’t see where one object ends and another begins, or what is a beach and what is an orthophoto.
So my life as a developer is a lot like a radiologist – I spend a lot of time looking at the inside of the sim, and I do this by using test art assets. A good test case is easy to build and clearly identifies the underlying mechanism in the sim. What follows are a few examples of the kinds of test cases we use.
This was one of the original global lighting tests. I needed to see if the lights performed on a large scenery, so I used George Grimshaw’s KBOS, and installed two custom lights. The lights are intentionally primary colors (green and red) so make the blending and mixing completely clear.
This is a test airplane panel to look at the effect of odd-sized instruments and resized panels on pixel clarity. The test case goes through every alignment possibility using art-work that has 1-pixel lines to show blurring. This is a test pack of roads – the pack contains one of every kind of road. The sim is put into a debug mode that labels the road with its code. Using this, we can quickly inspect the entire road art asset pack to see if any road types are buggy. This is a very early version of the roads – one of the original mockups. (That’s why a lot of the roads are missing whole sections.)
Here’s another road test – in this case, there is a custom scenery pack that replaces the cars with colored axes, making it easy to debug the alignment of the cars relative to hills and highways.
These two are test autogen packs; the markings allow me to debug the code that places autogen elements in the sim. The real autogen elements blend together, making it impossible to spot bugs; these ones show alignment errors clearly.
This is an older shot from the weather system with the shadowing vectors on – the lines all over the clouds show the measurement of light going into each part of the cloud. Since clouds can be soft and nebulous, the lines show clearly what would otherwise be hard to visualize.
This test object was built by propsman and exercises approximately 100 different features of the version 10 pixel shaders. In this case, each shader ‘trick’ has a piece of artwork with a clear label to show which part of the shader is where. The real art assets use the shaders in subtle ways that make them hard to debug, but in this case, if the albedo is missing (for example), it’s really obvious.
This is a piece of DSF with the beaches replaced with a beach calibration texture. The beaches are solid and have numeric markings to show which part of the texture is being applied at any given time.
Why show these? My point is this: I spend most of my day looking at the sim with test artwork to see what’s really going on. Many of the art assets I work with are specifically built to test – others have been hacked to find bugs. When I take a screenshot, it is usually to illustrate a particular point about the sim; the rest of the picture probably contains random junk that built up on my hard drive.
So: the marketing people will get you nice pretty screenshots that you can get excited about, but for the time being, most of the pictures here are designed to illustrate only one specific point, and are not at all meant to illustrate what the final X-Plane 10 will look like. You’re welcome to speculate all you want on the rest of the picture, but you’ll be using an X-Ray to look for a skin rash.
In my previous post I suggested a few ways that we will try to ease the introduction of new rendering technology in v10: complete compatibility when the feature is off, close-as-possible compatibility when it’s on, and a simple path to migrate to fully using the new tech.
But: if you’re doing something that should be illegal, we can’t help you. And this can be tricky because X-Plane 9 doesn’t always complain about illegal content.
Propsman just hit a case of this: he ran one of his old plugins on an X-Plane 10 development build* and discovered that (unlike X-Plane 9) it accidentally disabled most of X-Plane 10’s drawing.
It turns out that his plugin wasn’t quite adhering to the OpenGL guidelines for X-Plane plugins. But the particular state he was leaving altered had no visible effect on X-Plane 9. That’s not the case for X-Plane 10. So he’ll have to fix his plugin; the fixed plugin will work correctly with both X-Plane 9 and 10.
Unfortunately, it can be hard to find these kinds of problems without a build of X-Plane that actually has problems when you violate a rule. So if you make a third party add-on, when we finally do have beta builds available, I encourage you to check your add-ons for possible plugin or authoring mistakes that might actually cause problems in X-Plane 10.
*Let me nip this one in the bud before it gets out of control: Propsman is one of five authors who are working on the new art content for X-Plane 10. These five authors have development builds of X-Plane 10. No other authors have X-Plane 10. We are not giving out copies of X-Plane 10 right now. We are not loking for testers. Do not ask me for a copy of X-Plane 10. Do not ask me to test X-Plane 10 early. If you send me an email angling for early access, I am going to mark your email as spam in Thunderbird.
A third party developer recently asked me what the purpose of panel regions was, and how they were meant to be used.
Now I blogged about this before, and if you’ve read the post you are probably thinking: “Ben, were you high when you came up with panel regions?”
Sadly, the answer is no, but that doesn’t make panel regions any less weird. This post explains what happens. (Sound nerds: see the end for the moral of the story.)
One thing to note: the panel texture is expensive. Because the sim re-draws it every frame, it has to sit in VRAM all of the time – thus it puts more pressure on VRAM than regular OBJ textures (which can be paged out when not used). No matter how you do your panels (the old way, the byzantine way, or the new way), make your panel texture as small as you can – it’s worth it!
The Short Version
The short version is this:
Always use the 3-d panel.
Keep your 3-d panel as small as possible – pack it tight!
Make your 3-d panel a power of 2 in each dimension, e.g. 2048 x 1024.
Use one cockpit region in your object, that matches the size of the 3-d panel.
The long version explains how we got here.
You Might Ask Yourself, “How Did I Get Here?”
In order to understand how we ended up in this situation, remember that panel regions were invented before the 3-d panel existed, and they were invented for X-Plane 9, which runs on some fairly old video cards.
Going into X-Plane 9, we wanted to simultaneously fix a bunch of problems with using panels as textures for 3-d cockpits.
The 2-d panel supports alpha; the technology to do this was at the time expensive and hurt performance of the 3-d cockpit as a whole.
The 2-d panel contains a lot of wasted space. The texture space dedicated to the cockpit windows is not useful in a 3-d model.
The 2-d cockpit might be huge; version 9 introduced 2048 x 2048 panels. Authors have a motivation to make the 2-d panel huge to cover a large monitor.
The 2-d cockpit includes 2-d spot light and shadow effects, which are expensive to prepare for the 3-d panel texture.
The 2-d cockpit might not be a power of 2 – at the time, this meant wasted VRAM.
The panel texture was drawn in 3-d by simply drawing the 2-d panel. Thus the panel textured area of 3-d cockpits didn’t respond to real-world directional lighting.
The panel region system fixed all of these things:
A panel region is a sub-area of the panel, and it must be a power of 2. This fixes items 2, 3, and 5.
A panel region ignores alpha and ignore spot lights. This fixes items 1 and 4.
A panel region is drawn with correct additive lighting, fixing item 6.
In other words, ATTR_cockpit_region fixed everything wrong with ATTR_cockpit.
3-D Panels Make For Chaos
It’s about at this point in our story that things start to go south design-wise. At the time (near version 9 beta) the panel system was in sort of a never-never land, not really my code, not really Austin’s code, being driven by one-off requests. There was no road map.
Shortly after we got the panel region system working, we added the “3-d panel”. The 3-d panel is an entirely separate parallel panel whose only role was to be “the panel texture” for 3-d cockpits. We realized we needed this because often the 2-d cockpit is useless for 3-d texturing.
Consider the overhead panel. If you have an overhead panel in 2-d, it’s going to be drawn at a crazy perspective, to make it look sane in a forward view. But to texture a real 3-d overhead panel, you need an orthographic texture, taken straight on.
Once we had a 3-d panel, a lot of the reasons for panel regions were gone. The 3-d panel could be tightly packed, with no windows, no alpha, and it didn’t need to be enormous. In other words, the 3-d panel solved most of these problems better than panel regions did.
Had I used my brain, at this point I would have simply merged the two features together and gone home and life would have been really simple. But instead, I let the two features co-exist. This makes for four combinations – we can have the 3-d panel without panel regions (generally a poor idea) and we can have the 2-d panel with panel regions (which is arguably unnecessary). And I let the 3-d panel be as unrestricted on the 2-d panel, which was just silly.
It should also be noted that having no alpha in the panel and a power of 2 panel size aren’t as important as they used to be, and can be overcome with modern hardware. They were important for version nine though.
So the bottom line is: you still have to use a panel region to tell X-Plane to use real additive lighting, and to guarantee that you’re going to use a power of 2, no-alpha texture. But the “region”-ness (the ability to pick out parts of the panel) isn’t useful since you can now create a separate packed panel for 3-d cockpits.
Thus the recommended practice: use the 3-d panel, pack it tight, and use one region to get additive lighting.
Clean-Up?
Can anything be done to simplify and clean up the panel creation process? I’m not sure.
For the authoring environment, there are three things we can do to make life easier for panel authors:
Introduce new commands that are simpler. For example, we could set up a single OBJ command that uses the entire 3-d panel with additive lighting – this command would provide the interface that always should have been there. Of course, this would be yet another command, making the “full” system (including backward compatibility) even more confusing.
Simply change the behavior of existing commands. I am always hesitant to do this, because it has the potential to break older airplanes. But the current system does provide a number of feature combinations that make very little sense. There might be some things that we could do that could simplify the system for authors. (For example, we could use new modern correct additive lighting even for ATTR_cockpit if the 3-d panel is used, since it’s almost always the right thing to do.)
We can provide Plane-Maker “hints” to encourage people not to use the strange un-recommended paths. This would hopefully start to shape the set of airplanes to use the features we want without breaking any airplanes.
None of these options is great, but my gut feeling is that if we start with 3, we might be able to go to 2 – that is, we can start encouraging people to use the 3-d panel with one full-size cockpit region, the preferred way to use the system.
Epilogue: What Have We Learned?
If there’s a moral of the story it’s this: when it comes to features that affect how third party content is created, think twice, cut once. Once you design a bizarre interface and everyone starts using it, it’s nearly impossible to fix, and the “cost” of maintaining backward compatibility in the sim goes up for years.
This is why I have said no to any and all incremental proposals for sound features in X-Plane. There is no question we need better sound support. But I believe that the future format of sound is (1) not particularly obvious in terms of today’s sound system and (2) not likely to look anything like what we have now.
If we put in incremental sound features one at a time, every one of them will have to be heavily modified to work with future sound features. There’s no easy way to add just a little bit here, just a little bit there. In other words, it’s the panel system all over again.
When there is a mature, modern design, it is possible to add incremental features quickly and cheaply. This is the case with OBJs – we’ve been able to add new features (mostly in the form of new attributes) one at a time and it more or less just works, because the basic OBJ design is solid, and adding new attributes doesn’t radically change the system. The same thing is true for adding new properties to generic instruments.
But when a feature fundamentally changes the structure of authoring, there’s a big penalty for not getting it right the first time. The panel system continues to be a pain in the ass to maintain and confusing to use because some major features were thrown together ad hoc without a good plan. I don’t want the sound system to be like that.
In a previous post I discussed some of the hang-on-start problems we were seeing on NVidia cards. Since these started relatively recently, but the sim code hadn’t changed much, we thought maybe there was a driver issue.
Since then we’ve collected a few more data points:
Some users with older cards are seeing problems starting X-Plane – there are shader-compile errors in the log file when this happens. Our shaders haven’t changed.
Some users with newer cards see rendering artifacts: typically the cloud shadows will darken some terrain triangles, but not others, which make the terrain look very weird with broken cloud cover.
Some users still see a hang on startup which is “fixed” with –no_fbos.
Now here’s the interesting thing: the shader compile errors and rendering artifacts appear to be happening with the 270.61 drivers; going to the 260.99 drivers seem to help.
So if you have NVidia hardware, there are a few data points I am looking for. Please post a comment if you meet any of the following:
If you have the 270.61 drivers and see either visual corruption, crashes or hangs, please try going back to 260.99 and post your results, including what card you have.
If you have the 270.61 drivers and a new card (GeForce 400 or 500) and you do not see corrupt cloud shadows, please post so – we need to know if the bug only affects some users.
If you have the 270.61 drivers and an old card (GeForce 7xxx or older) and you can run without crashing, please post so – we need to know if the crash bugs only affect some users.
As always, we don’t know if this is a driver bug or an X-Plane bug, and who “fixes” the bug may not be an indication of whose fault it is. We will work around the bug even if it is in the drivers, and sometimes this kind of problem is due to X-Plane doing something naughty that some but not all drivers tolerate. We won’t know if this is really a driver bug until we have a full diagnosis.
Edit: These bugs are Windows/Linux specific; Macintosh users will not see them, nor do they have driver version numbers that match the normal NVidia scheme.
A while back I described some of the new lighting features in version 10, including lighting calculations done in linear space. The very short version of this is: X-Plane 10’s lighting will be more physically realistic, and this is important when you have a lot of lights shining all over the place.
The change doesn’t affect how you author content (you still texture your models and we shine the sun on them) but it has been a source of bugs, as we find all of the different parts of the sim that use lighting equations. In the picture on the right, the landing light hasn’t been updated, and as a result, it lights up the city 2 miles away. The picture on the left has the landing light turned off.
I kind of like the right hand version, but that’s why I’m not in charge of the overall “look” of the sim.
(Will the runway lights look that “splattery”? Probably not, but I don’t know; Alex will be in charge of the final decision. That picture is zoomed in, which makes the lights look a lot bigger, but also the tuning of the runway lights for size is only partially done right now.)
