There was a lot of confusion in the comments from my last post regarding X-Plane and free updates. Let me try to clarify.
First, here’s how updates have worked historically:
Users get free patches for the life of a major version run.
These patches include bug fixes, performance improvements, and sometimes substantial new features.
No new patches once a major new version is released.
The old patches are available and hosted indefinitely while the new version is out (at least for versions that use our updater).
There has never been any kind of guarantee as to what features will be released as patches, or how long a version run will last. Users who buy X-Plane buy the new version because historically the sum of a major version release plus its patches has been substantial.
The global scenery doesn’t change for a major version run, and therefore there are no automatic updates to the global scenery.
That’s how version 9 has worked, that’s how version 8 worked more or less, and it is how I expect version 10 to work in the future.
To address specific concerns: yes, the version 10 run will have free updates to the core sim, just like before.
The point of my previous post is this: in the past, we have not done any updating of the global scenery, and I am not committing to anything more now. OSM gives us a sane way to collect new data, but it’s too soon to talk about distribution.
So the problem with this blog thing is that if I go radio silent for a few daysweeks you all think I’ve been abducted by aliens. Let me first set the record straight:
I have not been abducted by aliens.
Austin has not been abducted by aliens.
Chris was abducted by aliens, and they may have done some experiments, but he seems about the same as before, so we’re not worrying about it.
So what I have been up to? Well, first, LR closed down the Westborough office and relocated scenery development to Grafton, MA. Here’s the new home office, ready for action:
As you can see, the move was orderly and the new office is spacious and uncluttered.
(This post is going to get really nerdy really fast – a lot of what goes into X-Plane isn’t easily observable – a lot of the work I do is engineering that goes in under the hood. It’s a long post, but coffee will help!)
Besides reassembling my office I have been working on global scenery DSF creation. In particular, I’ve been trying to kill off the remaining “wicked” problems – that is, the parts of DSF generation that require first-of-their-kind algorithms.
(More technically, a design problem is “wicked” if the only way to learn enough about the problem to solve it is to go through the act of solving it. This chicken-and-egg problem is usually broken by trying to solve the problem multiple times – the first few times are “learning experiences” – that is to say, the first version of the code often has to be thrown out.)
Stuffing Your Luggage
Most of the problems with global scenery generation turn out to be “knapsack” problems. The knapsack problem is basically this: let’s say you have one suitcase (or knapsack) and you want to fill it as much as possible with your clothes. All of your clothing are of different sizes. What subset of clothing do you take that leaves the minimum wasted space?
Well, it turns out that if you want the best fit, with the minimum wasted space, the only way to answer that question is to try every combination of your clothing. Yep. You have to pack and unpack your suitcase approximately a gajllion times and just try every packing to see which one is best.
This matters to computer programmers because often we have to “stuff” a space as optimally as possible (or find some optimal combination within a set of combinations); what we learn from the knapsack problem is that the only way to find the truly optimal solution is to try them all. The problem is that when you have a lot of things to try packing, it takes too long (even for a fast computer) to try every combination.
Just Shove The Biggest Things In First
I’ve put “best” and “minimum” in bold above because they are key to defining knapsack problems. If you have to have the truly best fit, you have to try packing your suitcase or knapsack with all combinations of clothing. But if a “pretty good” fit is okay, you can do something a lot faster. You pick a heuristic (that is, a rule of thumb) and hope it works out well.
Let’s take a look at packing in action. When you load an airplane, X-Plane packs all of the tiny instrument overlay textures into a few big textures to optimize frame-rate. (This is called “texture atlasing” – authors have to do this by hand for scenery.) Packing small textures into a big texture is a knapsack problem – to find the truly optimal packing we’d have to try every combination of instruments in each texture. That would be very slow, so X-Plane instead uses a simple rule: we pack the biggest things in first, building rows. As you can see from these pictures, the packing isn’t very good – a human could do a lot better. But the packing isn’t bad, and it’s fast enough (a perfect solution would take days to run for each airplane load).
Pretty Heuristics
When we have a knapsack problem, and our solution is a heuristic (that is, rather than find the most optimal solution, we’ll find a pretty good solution) there is another question: are the particular kinds of solutions that we come up with nice looking? Do we like them? For panel textures, this is a non-issue; the user never sees the “pack” of the panel texture while flying.
But for global scenery, the look of our heuristics matter. Global scenery generation is riddled with problems where the ideal setup of data would be too slow to compute, so we want our approximation to look “good”.
Here’s an example from global scenery of a knapsack problem where the look of the heuristic solution matters. The map on the left is the OSM vector data for San Francisco. The map on the right shows the same region broken into terrain zones. Calculating terrain zones is a knapsack problem with a heuristic solution where the look matters. Let’s break that down and see what that means.
First: what is a terrain zone? A terrain zone is a set of adjacent city blocks in a DSF that all have the same base terrain. If you recall Austin’s rantdescription of the way cities are created in version 10, he said some stuff about starting with grass and building cities on top of them. Well, it turns out that if you build every block up on the same grass, you get something that starts to look like a mini-golf course.
We have a lot of ways to cope with that, but one of them is terrain zones: several city blocks are grouped together with one base terrain – but then, at a road boundary, we change to a different base terrain to “break up” the landscape along road boundaries. The look is a lot less monotone in the far view – at least we think; the picture on the left is San Diego (terrain only) with the urban terrain zones set to primary colors for easy examination. (When the real artwork is in place, the terrain will be similar and color, with roads dividing the various zones.)
The packing problem is: which blocks get grouped together, and which are kept separate? Some groupings produce a more efficient final mesh, which results in better framerates. Like the knapsack problem, it would take too long to find the best grouping for framerate, so instead we try to come up with a pretty good grouping.
If you look at the groupings the above maps and diagram, you’ll see there is sometimes a “tooth” pattern where the boundary beteween blocks jumps up and back. That’s not so good looking; the algorithm that produces the groupings sometimes picks ugly block combinations, and we can’t just tell it “don’t do that” because it doesn’t check every combination to find the best one.
What Have We Learned?
Putting it together, you can see why global scenery generation is tricky.
We want to combine city blocks to make moderate sized areas of similar terrain, preferably in clumps that look nice.
We can’t just try every combination of block groupings; that would be too slow (it’s a knapsack problem).
Instead we have a heuristic – a simple strategy to try that produces “pretty good results”.
That strategy is tricky to develop because it has to come close to optimal results (for fps) and we don’t want the groups of blocks to look silly (which is hard to quantify).
Therefore, calculating the terrain zones is a “wicked” problem – only by implementing a heuristic do we get to see how it “looks” and decide if it is acceptable.
Like previous betas, this build mostly fixes bugs around import and export, which was the area that caused the most trouble. Please file WED bugs via the scenery tools bug base; I don’t often get to work on WED, but when I do, I can kill off a bunch of bugs at once if there are good reports. Thanks to all of the users who have submitted good bug reports with example scenery packs to help reproduce the problems.
Future WED Versions
Simultaneous to bug fixing on the WED beta, we are adding new features to support X-Plane 10 ATC: WED will be the tool you use to edit taxiway routing diagrams and airport layout flow information for an airport. (This means that you’ll have your physical pavement, your routings, and your custom scenery all in one environment.)
These features are disabled in the WED 1.1 betas, but once version 10 is released and WED 1.1 is final, we should be able to rapidly go to beta on the new features; since we use WED to build our test layouts now, it should be in reasonably good shape.
In the long term, I’d like to see WED provide a visual front-end for MeshTool*, but I won’t let this hold up a release of an ATC-ready WED; so far having “future features” floating around in the code (disabled for release) hasn’t caused any bug reports or stability problems for the existing 1.1 beta candidates.
WED also needs a file format change; my original decision to use an sqlite3 database was the wrong one; when the file format change happens, there will be full backward compatibility for reading old projects, so you’ll be able to bring your old projects forward just by opening them; you won’t have to reimport the DSFs and you won’t lose your groups and other WED-specific information.
* There’s no reason why WED has to be the only front-end for MeshTool, and it may not even be the first one released. One option might be to make a MeshTool export script for qGIS.
First, two quick notes on comments: all comments are moderated, so if you post a comment and it doesn’t appear, please don’t repost. Sometimes Chris and I are both out for a few days, and the comments sit there. Without moderation, WordPress-based blogs get the hell spammed out of them. Chris has also turned off commenting on very old posts. We’ve had a few cases of people digging out old threads to try to jump in to a discussion after the fact, but a lot of what goes onto the blog is time-sensitive – by the time a month is up, a newer post covers the topic in more detail.
Now on to tools: I have posted new release candidates for MeshTool, version 2.0 release candidate 6:
MeshTool has been in a “perpetual release candidate” state for a while. I think this is partly because we don’t eat our own dog food when it comes to MeshTool – the tool shares a lot of code with our default DSF generator but they’re not the same thing. Not surprisingly, the MeshTool code that is specific to MeshTool and not used for global scenery is where the bugs crop up.
So my thanks to those MeshTool users who have patiently sent me bug reports and reproduction cases – I’m hoping that this release candidate is final.
What’s Next for MeshTool
I think MeshTool will go in a few different directions over the next year:
The scenery tools code is “branched”, with two forks. The older fork targets X-Plane 9, and the newer one targets X-Plane 10. I suspect we’ll eventually have to have two MeshTool versions, 2.xx for X-Plane 9 scenery, and 3.xx for X-Plane 10 scenery.*
One logical next direction for MeshTool is to add “processing” features. For example, we have code to roughly flatten an airport area – useful when using SRTM data (where concrete runways make false radar returns that turn into tiny hills). This code could be made available from MeshTool.
The other logical direction is to put a real user interface in front of MeshTool. Just like WED and OE provides graphical ways to place objects (where-as DSF2Text is low level), it would be easier to work with a mesh scenery if you could see your work, and have MeshTool run automatically in the background.
I don’t know what the time frame for future MeshTool work will be; future MeshTool work isn’t necessary to ship X-Plane 10 so it may have to wait a while.
* This is really not ideal, and it is not the way that most other compatibility will work. Wherever possible, I’m trying to make the tools simply work for both X-Plane 9 and 10 in a single tool. It’s less code for me, and more useful for you (because when we need two tools for two sim versions, inevitably one of the two tool versions will be less feature-filled than the other).
“We eat our own dog food” is a term in software development for a company that uses the product it sells for its own operations. The idea of “dogfooding” is that developers who use their own software will have much stronger incentives to make the software usable, productive, and reliable.
Laminar Research practices a form of dogfooding with the scenery tools:
All of the scenery tools are open source, so we don’t have any “special secret magic tools” that you don’t have.
Our art team uses the same set of tools that third parties use.
Dogfooding does have an effect on development: when the artists get on my case about the usability of a scenery tool, it gets my attention. We pay those guys to churn out a lot of artwork; if they aren’t productive, we can’t get the sim out.
However, there is an Achilles heal to dogfooding: you can only dogfood software that is useful for your company’s business. We have this problem with MeshTool: since we don’t make our own orthophoto sceneries, MeshTool doesn’t get heavy internal use, and as a result I think it’s been a lot slower to become usable. The MT users who have filed bugs have been very helpful, but I feel for their suffering with a very raw tool.
Dog Food and Version 10
For version 10, we are producing art assets that use new rendering technology for the very first version. This means we’ve had to build the tools to support new version 10 technology early on, and that’s good news for third party developers. When version 10 is released, I’ll package up the various version 10 tools we’ve created for public consumption.
In most cases, the tools need additional polish – our artists often use tools in very early, buggy, raw form. But what we already have is a big head start, which should shorten the time from version 10 release to a full set of tools.
I’ll post more over the next few days regarding some of the specific tools we have and what we will do with them once version 10 ships.
The billboarding scheme for cloud puffs has been rewritten for X-Plane 10; for the most part you should not see puffs spinning or reorienting themselves as you move the camera. However, while poking at the cloud puff shader, I managed to induce this picture on the left. The weather system makes some of the best bug screenshots.
(The post was originally labeled “bug or tornado”…I changed it because it seemed obnoxious after what happened in Joplin, MO – just me being a jerk from the East Coast. A few days later we were hit by a series of Tornadoes fairly close to home.)
Bad preferences have become a superstition in the X-Plane community. If anything goes wrong, the first thing anyone says is “delete your preferences”. This often “fixes” the problem – in that it undoes all of your careful settings and changes about 100 different things, one of which changes enough behavior to fix the problem. It’s the medical equivalent of curing the common cold by transplanting every single organ.
Of course, every now an then the preferences file is the problem. It turns out that X-Plane 9.6x and 9.31 use the same version ID for the binary preferences file, and yet they use different binary preferences file formats. This was a screw-up on our part, and the result was a hang on start for 9.6x users if they had their old 9.31 preferences laying around. In this case, nuking preferences really was the problem. (Thanks to Andy for getting me the files to see this “in the lab”!)
Send Me Your Preferences!
What is frustrating about the 9.31/9.61 preferences bug is that people have known about it forever, and yet I only received a real bug report about it this week. Part of the problem is how people cope with these bugs: by deleting their preferences.
Don’t delete your preferences! Once you delete the preferences files, they are gone! If there was something wrong with them, we’ll never know.
Deleting your preferences is like burning all of your belongings when you get a cold: the Doctor won’t know what bacterium or virus you have because you destroyed every instance of it!
You don’t have to delete your preferences. You can simply move them out of the preferences folder onto your desktop. This has the same effect as deleting them: X-Plane will not find them in the new location, and will start up with its default settings.
What if moving preferences fixes the problem? Well, now you have two sets of preferences: the old “sick” preferences on the desktop that cause a problem and the new “rebuilt” ones that X-Plane made when you re-ran.
At this point you can now send me a bug report. It might go something like this:
You: Hey Ben, I was getting a crash on start. I moved my preferences files out to the desktop and it fixed it. Here are the bad preferences and here are the good ones, in two zip files.
Me: Thank you! That’s great! I will look at the two sets of preferences to see what changed. If there’ a sim bug I can identify it and find it.
You: So…do I get any kind of reward?
Me: Only the good feeling of knowing you’ve helped your fellow X-Plane users.
You: Screw you! I’m going to go use Google Earth Flight Simulator.
Okay, that didn’t go quite as well as I had hoped. But my take-away point is this: if there is a real problem with the preferences system, such that deleting preferences is necessary, we want to look at the files and fix it in the sim! In the long term this will lead to less need to delete preferences, which will save a lot of users a lot of re-doing settings.
Why Do Preferences Files Matter?
Why does moving your preferences files out of the way (so X-Plane can’t find them) sometimes help? There are a few causes that I’ve seen:
Incompatible or corrupt binary files. Sometimes the preferences files actually contain junk, although this is rare. In some cases, we don’t handle the preferences file version change and we need to fix the version check code (this is what we’re fixing in 970).
Sometimes a single particular setting is problematic for a particular machine. For example, your machine might have a driver problem that causes it to crash when “per pixel lighting” is on. If you remove the preferences and the default is to have that setting be off, then you “fix” the problem. In this case, a comparison of the “bad” preferences with the default ones can reveal settings that might be the problem.
Sometimes settings can be set while you fly, but cause problems when the sim starts with the settings turned on. This is rare, but with tweaky graphics drivers we sometimes do see this behavior. Sometimes a setting takes effect on restart, so it’s not until you restart with the new setting that the preferences file causes problems.
Either way, having the “bad” preferences to inspect is critical to catching any one of these cases!
In testing some ATC stuff today, I was curious why I saw smoke coming from the water on the left side of the runway. As I got closer, I found a KC-10 rising from the ground like a mighty Phoenix….struggling to get out of the controlling grips of Air Traffic Control. Ben said it reminds him of Free Willy.
And of course the usual disclaimer…the purpose of the video is to have a giggle at the AI plane’s bug…ignore the ugliness of the rest of the situation. The colored lines are used for debugging aircraft routing issued by ATC.
What is the limit to how precisely you can build scenery in X-Plane using WorldEditor?
In this picture, I created a runway “OBJ” exactly the same reported size of the runway, in the same location and orientation in WED. The windsocks are also placed on the exact corners of the runway in WED. This is a typical “MRI” scenery pack – that is, a test scenery package designed to show how well/poorly the sim is performing.
So what do we see?
The alignment of the windsocks with the runways is pretty damned good. It might not be perfect, but it’s awfully close.
The alignment of the OBJ runway and the real runway below it is not particularly good – the heading isn’t quite perfect enough to stay aligned over a 10,000 foot runway.
This lack of precision is a design limitation – that is, X-Plane does not try to provide unlimited precision in scenery authoring. (At some point, increasing precision increases file size and processing time, which would have a detrimental effect on overall system performance.)
In particular, X-Plane tries to provide high quality point-to-point matching. That is, given two locations that are both specified by latitude and longitude, X-Plane tries to get them to be as close together as possible.
What doesn’t work well is trying to correlate a specific location (specified by latitude and longitude) with another location specified by relative location (e.g. from this location, go 50 meters to the west).
Generally, the longer the “arm” of relative location, the worse the precision. in this example, the runway is specified by the center of the ends, and the windsock locations are specified directly. Thus the runway corner is on a 25 meter “arm” from the runway end center. We may pick up a little bit of imprecision on a 25 meter arm, but the correlation is good.
By comparison, the object end is on a 5 km “arm” because it is specified by its center location and heading, while the runway end center is specified directly. As you can see, over a 5 km arm, a fair amount of error is accumulated.
Thus we can come up with a working strategy for scenery placement:
When placement precision is important, keep objects small.
When two scenery elements must be locked together, be sure to use scenery elements that can be specified by exact location and not by a metric distance from another location.
“Looks close” in WED only means “pretty close.”
The hardest part of this work-flow is using art assets other than objects for large scale features. But (for example), a draped polygon is better both for precise positioning over a large area and for polygon offset/Z-thrash, so you probably need to use draped polygons anyway.
An author recently asked me what the relationship was between the size of PNG files on disk and the amount of VRAM X-Plane uses to store the PNG file as a texture. I actually get that question fairly frequently, and the answer isn’t totally obvious.
The first and most important point: the size of a texture on disk in an image file is not the same as its size in VRAM, so changing the file’s size on disk doesn’t save you any VRAM. (It does save you hard drive space, which might be interesting if you are building a huge orthophoto scenery, or maybe download size, which can be handy.)
The size of an image in VRAM is a function of its dimensions, the amount of color information, an whether texture compression is on. The formula is basically this:
size = 4/3 * width * height * bytes_per_pixel
The 4/3 term is due to mipmapping – we save progressively smaller versions of the texture to improve the speed and quality of texturing when your image is far away.
Bytes per pixel varies depending on whether texture compression is on:
RGB, or RGBA, no compression: 4 bytes per pixel.
RGB, compression on (or DXT1 compressed DDS files): 0.5 bytes per pixel.
RGBA, compression on (or DXT3/DXT5 compressed DDS files: 1 byte per pixel.
Alpha-only: 1 byte per pixel.
That first point might be a little bit surprising to authors: when compression is disabled, you don’t save 25% for getting rid of your alpha channel. This is because modern video cards round up the texture from 3 to 4 bytes to keep the image size a power of 2. It is still typically a win to drop the alpha channel if you’re not using it: X-Plane can fill the screen faster if it knows that there is no alpha (and thus no blending) and the RGB-only texture will be half sized when compression is on.
DDS File Size
DDS (Direct-Draw Surface) is a file format designed to mirror exactly what video card want in VRAM. As it turns out, a DDS file is a good representation of how much VRAM your image uses, because it is already in the format that the card wants – including the 4/3 term. So if you need a quick and dirty estimate of your texture’s VRAM use at extreme res, the DDS size isn’t a bad proxy. (Note that when LOAD_CENTER is in use, your texture is reduced in size most of the time.)
PNG File Size
PNG file size is unrelated to VRAM used; PNG files are internally compressed. In fact, PNG files have multiple possible internal encoding schemes; utilities like PngCrush try all of the encoding schemes to see which one reduces file size the most. The key is: none of this translates into VRAM savings. PngCrush may make your downloads faster, but it won’t make your add-ons use less VRAM.
