Since before I’ve been involved in X-Plane, scenery has been broken into tiles. You know the naming scheme…+42-072.XXX is some kind of file for Boston. Perhaps it’s a .env or a .dsf, but what is unchanged is that the world is broken up into bite-sized pieces.
Tiling is very necessary for X-Plane…it allows the sim to rapidly load only the information we are interested in. The world is too big to go fishing for the relatively small amount of data that is loaded at one time.
The source data that is used to build global scenery is also tiled – if you look at the SRTM files, they often have names like N42W072. It’s no surprise that global data requires tiling to make it manageable.
When we produce global scenery, we work on a per-tile basis. We load the raw tiled data* into source XES files (one per tile), then process and convert that XES file into a DSF. By working one tile at a time, we limit how much RAM we need.
But if you used WorldMaker on an airport or city that spanned a tile boundary, you know how annoying tiling can be. With WorldMaker you couldn’t see both halves of an airport at once.
WED will not use tiles. For custom scenery packages, the total data is not so large that we have to tile. When you work on a scenery package in WED, you work on the entire package at once as a single seamless workspace. When you export your work, WED will split the data into tiles as needed. This will mean:
- You do not have to decide in advance where you are going to work. All scenery packages cover the entire planet, and DSFs are created only as needed.
- You will be able to work on more than one “tile” of area at a time, because there are no tiles.
- You will be able to work on airports that span tile boundaries seamlessly.
I just finished fixing some bugs in WED regarding the hierarchy view. (To get a sense of what a hierarchy view is like, try AC3D…)
In WED, the contents of your airport layout are viewed in a hierarchy view. This is where you reorder your runways and rename entities. You can also group entities (as many times as you want) to help organize your layouts. (The groups will not be visible in the final apt.dat.)
An airport is, in a way, sort of a super-group…it’s a group of “stuff” (runways, taxiways, etc.) plus it has its own information.
But this leads to some tricky situations:
- You can’t have an airport inside another airport.
- You can’t have a runway outside an airport.
The result is that there are a number of grouping and drag & drop combinations in the hierarchy that are simply illegal. Cris and I debated this a bit and what we have now is: commands that would produce illegal results simply cannot be executed. (That is, the group command is grayed out if the new group would violate these rules, and the drag & drop won’t show a legal drop if the new order would violate these rules.)
Once we get into beta we’ll see how well this works. The alternatives include:
- Putting up a specific error message when an illegal configuration is made (annoying).
- Allowing illegal configurations and putting up an error message on export (by this time the whole file layout could be pretty badly messed up).
- Attempting to silently export the illegal configuration (unpredictable – there’s really no good way to export an airport inside an airport).
Also, I am sure there are some illegal configurations I have missed…we’ll have to catch those in beta too.
Posted in Tools
by
Ben Supnik |
First, I just want to be clear: I am not announcing any future features for the scenery tools. I am not saying when they will be released, and I am not saying what they will do, because honestly I do not know. There have been too many cases when users have emailed me and said either “I thought you guys were going to do X” or “you guys promised you were going to do X”, so now I am officially paranoid.
So this blog post is not about what the future scenery tools will do – it is simply a discussion of the difference between editing source data and compiled DSFs.
When we make the global scenery, we start with a bunch of source data that roughly consists of: road maps, coastlines, elevation, landuse, climate data, airport locations, etc. When we build a DSF out of it, we “bake” these items together into a single file. During this baking process, our tools apply some “integration effects”. Here are a few of the more obvious examples:
- Terrain under airports is forced to be an airport grass texture, appropriate for local climate data.
- Roads are removed from airport areas.
- Intersections are computed for highways – that is, a highway and city street form an overpass, but two city streets become a real intersection.
- Generated Buildings are put around the roads, not under them.
- Buildings are oriented to “face” the slope of the ground they are under, based on their shape.
That’s not a complete list, but it gives you an idea of some of what goes into making a DSF. All of this information is precomputed and represented in the final DSF. But consider this last point: the DSF contains the actual orientation (north/south/east/west) of each building. It does not contain the slope underneath the building and it contains no information about which buildings need reorientation. In other words, the results of the process, not the inputs to the process, are present.
So consider what would happen if you could simply edit the data in the DSF:
- If you moved an airport, the airport grass would stay in its own location.
- If you moved an airport over a road, the road would still be there.
- If you changed a city street to a highway, it would not form an overpass.
- If you moved a road on top of a generated building, the building would remain in place.
- If you changed the ground elevation, buildings would not change their orientation to face the new slope.
If you changed the source data and re-ran the DSF building process, these effects would occur. But remember, we need elevation, land use, etc. to build a DSF from scratch, and the DSF itself doesn’t contain its source data.
So we have two possible strategies for editing DSFs:
- We could build a DSF “retouching” tool that let us make very small changes to the existing DSF without having any source data. None of these effects would “work” so authors would have to make very small changes and then hand-fix any problems that appeared.
- We could build a DSF “rebuilding” tool that let authors make new DSFs from source data. All of the effects would look good, but authors would have to get some of the source data. (We could post our source data, or provide links to places where it can be downloaded.)
Note that we can’t have our cake and eat it…we cannot get the “integration effects” listed above unless we go back to the source data. If I can stretch my cake metaphor to the breaking point, once we make our cake, we can’t easily remove the flour and add another egg – we need to start over with the raw ingredients.
Which strategy will the scenery tools use? I don’t know yet. I am focusing on airport and overlay editing, which sidestep this issue a bit (we can easily edit an apt.dat 850 or overlay from the final product). We may do a bit of both strategies – it depends on what users want and what we can code efficiently.
Why don’t the finished DSFs contain everything we need to edit them? The answer is size. The finished global scenery was about 56 GB of DSF. When I last checked, the raw data that forms the DSFs was at least 100 GB or more. So for each DVD we shipped of scenery, we’d have to ship two more DVDs of source data, for a 21 DVD set, most of which wouldn’t be useful to most users.
That take the longest to program. This may surprise non-programmers, but won’t surprise anyone who has done UI development before.
Example: in WED, there is a spreadsheet-like view (the “property” view) that shows text-based information, usually with a list of runways or taxiways in one direction and different aspects of them (name, displaced threshhold length, surface, etc.) on the other.
In WED, if you press the tab key while editing text, WED will move to the next editable field, first across, then down. Furthermore, it will scroll the view as needed so you don’t need to adjust the scrollbars while working. The result is that you can, for example, use the mouse to select all of your taxiways, and then rename every single one of them from the keyboard with the tab key without ever going back to the mouse.
It’s a small UI feature that can make a big difference in how long it takes to work on a project, and takes surprisingly long to code.
Posted in Tools
by
Ben Supnik |
A user reported a bug: the 3-d buildings in all parts of the world outside the United States* are European-style, even in Australia, Canada, China, etc.
It takes two things to see 3-d objects in X-Plane:
- Location data in the DSF you are flying over.
- 3-d models (stored in OBJ files) that match the definitions called for by the location data.
Now we have location data for our “generic” objects (objects that are placed pseudo-randomly based on local topography and roads, to synthesize cities) for all urban areas that are covered by the 7-DVD global scenery set. But we only have two sets of artwork: a set of US-style buildings for use in the US, and a set of European-style buildings for the rest of the world.
So we had an unfortunate dilemma. Either we could have European style buildings everywhere, which looks funny, but at least it’s 3-d. Or we could use the European-style buildings only in Europe and have no 3-d at all everywhere else. Both options make some users unhappy and neither is very good. We chose to have the European buildings everywhere – my guess is that it generated slightly less complaints.
Well, the silver lining is: it’s actually really easy to remove the European-style builldings from the rest of the world (Canada, Australia, China, Russia, Africa, etc.). All you have to do is make a custom scenery pack that applies to the regions where you don’t want the buildings and maps them to a blank object. I have built such a custom scenery pack and posted it here. If you install this scenery pack, then Canada will be devoid of 3-d, but at least there won’t be mismatching architecture.
This begs the question: why not use the US buildings to populate some of those regions? The answer is that the US buildings are built to a different spec than the European ones, because the road grid data for the US is more detailed than the rest of the world. So amongst our current artwork, only the European buildings will work globally.
* When I refer to the United States, I really mean the continental United States. Sorry to everyone in the beautiful states of Hawaii and Alaska. This is just an artifact of how the data was imported – I could not get the Census data for Hawaii, Alaska, or the territories to import, so I reverted to the generic “global” data.
With X-Plane 860 there are two ways to draw stuff on the ground:
- Use an OBJ and ATTR_poly_os. This is the way most people have built custom taxiways, markings, etc.
- Use a DSF overlay with a bezier polygon (which is controlled by a .pol or .lin file).
Which should you do? Hrm…
- For any geometry that goes over a very large area, the DSF overlay with a bezier polygon is basically a requirement, because the polygon will “drape” over the ground. The OBJ might just stick out in the air.
- If your element repeats a lot, the OBJ may save memory. (OBJs are stored in memory once and used many times – bezier polygons use RAM each time they’re used.)
- If the overlay is very simple, the DSF overlay may perform better.
- It may be too expensive to build complex lines with DSF overlay lines. They’re meant for taxi-lines, not full vector graphics!
So here are a few decisions I’d make:
- Taxi lines – certainly DSF overlays if we’re not using apt.dat 850.
- Parking spot at an airport – use one large polygon with a texture, not a bunch of bezier lines.
- Use a DSF overlay for that parking spot, even if we use it 60 or 100 times. The simplicity of the overlay (vs. the compexity of starting and stopping an OBJ from drawing) outweighs the cost of using it many times.
- A very structurally complex overlay on the ground with thousands of triangles (that for some reason cannot be created with a texture) that is used a lot — well, this is an “artificial” case I’ve made up, but in this case use an OBJ.
I’ve blogged in the past about ATTR_poly_os…it’s a tricky topic. ATTR_poly_os is
a feature of the OBJ file format designed to let authors fix z-buffer thrash problems. Unfortunately, the cause of z-buffer thrash is pretty complex. To make things worse, it turns out I never finished my intended documentation on the subject. (I’m an idiot!)
The fundamental problem I think is that what we have now (ATTR_poly_os and ATTR_layer_group) provide a mechanism to correctly fix z-buffer thrash, but they don’t in any way enforce good behavior over bad behavior. The two attributes are very flexible, and if used together, can do all sorts of bad things. The problem is that ATTR_poly_os was thought up years before the layer-group mechanism, and thus they don’t really reinforce each other.
So…here are a few simple rules to help with z-buffer thrash in X-Plane 860:
- Never use the names of objects or their order in the DSF to accomplish anything. X-Plane ignores both names and orders when processing your scenery.
- Do not move your polygons above the terrain to fix z-thrash. This won’t work.
- When possible, divide your objects into ones that are 100% on-the-ground (and thus may z-thrash) and ones that are 100% 3-d above the ground (and will not thrash). I realize that more objects means slower fps…so this applies best when you have many objects and can pick how you divide them up.
- Always use ATTR_poly_os for any polygons that lie along the ground. Use the smallest number you can to fix the thrash.
- If you have an object with ATTR_poly_os geometry and non-poly_os geometry, make sure the ATTR_poly_os geometry is first!
Those five rules should keep you out of trouble.
What about ATTR_layer_group? Well, secretly X-Plane 860 will change the layer group of an object that is ATTR_poly_os for you. So as long as your object contains only offset geometry (this is what I recommend in rule 3) it wll always be drawn before the rest of the objects, preventing artifacts.
You’ll need ATTR_layer_group if you want to put objects underneath runways, or underneath taxiways, for example.
I am working on more comprehensive documentation on the topic, and appreciate any feedback on stuff that I’ve written that’s unclear…the rules are complicated!
The internal joke about WED (the new scenery editor) is that it stands for Will it Ever be Done. So I should say that, given my total inability to predict when it will be done, and how many delays and setbacks there have been, I don’t expect you to believe anything I say until I actually ship something you can run (or at least post some screenshots*). I realize that I have destroyed my credibility about ship-dates by having no ability to predict ship dates for WED.
With that in mind, I am pleased to report that today I was able to run WED with a few features working in concert:
- Multiple selection (map and hierachy view).
- Multiple undo everywhere.
- Marquee multi-selection tool on the map view.
- Tree-based hierarchy view with editable property fields.
WED still can’t do anything remotely useful, but these four features represent a huge amount of infrastructure investment. Basically my goal is to provide some of these user-interface features (multiple undo, multiple select, multiple tools, etc.) for every single type of scenery component that you can edit.
Since apt.dat contains a lot of different components (just look at all the record types in the file format) and WED will also have to edit DSFs with a wide variety of data, it made sense to me to write a generic mechanism for these features that could be used over and over again without writing new code.
So my hope is that I’m just getting over the “hump” of writing infrastructure, and soon will be able to add two dozen more types of editable things to WED (other than my simple test objects) very quickly. We’ll see if it pays off.
The plan is still for WED releases to be separate from X-Plane and to be open source (MIT/X11 license). I’ve been meaning to post in Chris’s programming blog about some of the design ideas WED employs – it makes a good laboratory. Hopefully those posts can also provide some guidance if anyone decides to modify/work on/steal the WED source code. (It is my expectation that most programmers will not want to get into the guts of how WED works, as it is a fairly complex application. I think I can make simpler interfaces to things like import/export to make extending the program simpler.)
Anyway, WED is not the only thing I am working on right now, and you don’t have to believe me, but the WED codebase is growing, and some of it even works!
* I am not posting screenshots because the program is running right now with “scab art” – that is, ugly green, red and purple boxes that will be replaced with nice PNGs once the layout settles down. I do not want to make our artists draw the UI components more than once, and I don’t want to answer 1000 emails about “Why is it so ugly” by posting the scabs, which are perfectly adequate for my own coding purposes and not intended to ship.
I can’t say enough good things about Jonathan Harris (Marginal) — his work on X-Plane has been fantastic, he is one of the most advanced third party scenery authors I know of, and when he sends us a bug report, it is usually so perfectly patched up that I’m looking at the bad line of code in minutes! (One of these days I’ll post one of his bug reports — he always isolates the bug in a simple package that makes it very clear, with no extra “stuff”.)
He emailed me a while ago with some questions about layer groups, and I saved them to rewrite into a blog entry. A little bit of background:
X-Plane 850 and 860 introduce the concept of “layer groups”, which provide a way to control the draw order of scenery to some extent. Objects naturally fall into a layer group based on their type (e.g. objects go into the “object” layer group by default, and runways always go into the runway layer group). However, some scenery elements let you customize their layer-group placement in two ways:
- By changing which layer group the element goes to entirely, or
- By providing a “bias number”, which indicates that within the catagory of scenery elements, this one must be drawn early or late.
Layer groups let you do a number of useful things:
- Make sure that polygons and objects are drawn under runways or over taxiways when needed.
- Make sure that runway markings with polygon offset are drawn before 3-d objects.
Simply changing the order of objects in the DSF is not a reliable way to control draw order! Layer groups are. You can read about layer groups in the OBJ8 spec.
With that in mind, some Q and A. (I will elaborate on my answers from what I originally sent Jonathan.)
J: I assume that objects and polygons within a single layer can be drawn in any order – ie there’s no defined drawing order between different scenery types.
B: You assume correctly! Within a layer group, X-plane is free to reorder to improve fps. So you cannot rely on the draw order of any two scenery elements without assuring that they are in different layer groups, either by using different group names, or different relative offsets.
J: But is there are defined drawing order between objects/polygons and apt.dat-generated scenery?
B: Yes because airport scenery goes into specific layer groups! (In fact, all scenery has a “default” layer it goes into, and they usually vary by type of scenery element.)
J: eg if I have an object or polygon with ATTR_layer_group runways 0 will it be guaranteed to be drawn after the runway?
B: You’re close. The runways go into the runways group so
ATTR_layer_group runways -1
will always be before runways and
ATTR_layer_group runways 1
will always be after.
ATTR_layer_group runways 0
is the default layer group for runways, so your object or polygon would be in the layer group with all of the other runways, and X-Plane would be free to change the order amongst your object and runways in any way that would optimize framerate.
J: If not, is there any other way to insert objects and/or polygons between the taxiways and the runways?
B: The numeric offset is provided for this. The “spacing” of the layer group numbers is such that you can have up to 5 groups before and after each “named” group. So anything from runways -5 to runways +5 is fair game. (In other words, you can separately control up to 5 different “layers” of elements with a well-defined order for any layer group name.)
This series of posts will discuss how the LOD system works for objects. Understanding LOD is very important for tuning the performance of custom scenery that uses either very complex objects, or a very large number of objects.
Authors View vs. X-Plane’s View
A “level of detail” for an object is a complete representation of the model with complexity changed to fit a viewing distance. In other words, LODs are copies of your objects, some of which have been simplified so they will draw faster when far away.
Each LOD for an object contains a viewing range, indicating for which range it is appropriate to use this LOD.
An X-Plane object can have LOD information, via the ATTR_LOD attribute. If it does, it may have one or more “LODs” – that is, separate representations of the same object.
If an X-Plane object has no ATTR_LOD command, X-Plane turns the entire object into one LOD by automatically calculating the viewing range based on the size of the object. (I’ll discuss this in more detail in later posts.)
This last point is important because it means that, to X-Plane, every object has at least one LOD. If an LOD is not provided by the author, one is created!
Picking an LOD
When X-Plane draws an object, it measures roughly how far away it is and picks the LOD within the object to draw. This can save some frame-rate but isn’t a cure-all.
– The cost of animation and attributes is based on the LOD drawn. So if you have an object with a lot of animation that is not visible a long way away, using a second, simpler LOD with no animation will help framerate because X-Plane won’t evaluate the animation when the complex LOD is not drawn.
– The cost of moving the texture to the graphics card is incurred no matter what LOD is picked, so you don’t save on “working set” (that is, the effective amount of VRAM X-Plane needs) with LOD. If your object has a huge texture, you pay that cost any time it is drawn.
– The cost of the number of triangles in the model is partly saved by using a simpler LOD, but it is partly global to the entire object. The problem is that all of the geometry coordinate data for objects is kept together in one giant glob. So when the tiny simple version of your model is being drawn, X-Plane may be shoveling the complex version to the card anyway.
This last point is important: making more LODs may actually increase the total amount of geometry data and make performance worse. Therefore while it may help a lot to have a second LOD that is much simpler, having 10 LODs, each slightly simpler than the last will probably hurt performance. Be sure to try your objects on a computer that doesn’t have a lot of free VRAM to see the effect of more LOD!
Don’t Draw That Object
While you can save some frame-rate by having X-Plane draw a simpler LOD, the true power of the LOD system happens when an object is so far away that none of the LODs apply. For example, if my object has 3 LODs from range 0-1000 meters, 1000-3000 meters, and 3000-6000 meters, then once the object is over 6 km away, X-Plane simply doesn’t draw it at all!
Not drawing an object is great for framerate. Not only do we save the time on the animations and attributes, but X-Plane never has to process the texture or geometry data, and that’s where the real framerate win comes from. Therefore the most important question for object LODs is “what is the maximum LOD” – that is, what is the farthest distance before X-Plane can stop drawing your object. X-Plane is pretty efficient at recognizing cases where objects, or even whole classes of objects, are simply too far away to draw.
(Consider the runway lights with the new “high detail world” setting…X-Plane places an object for every single runway/taxiway light, of which there are almost 10,000 at Chicago O’hare. But since the maximum LOD for these tiny objects is only 500m, X-Plane then eliminates the vast majority of the objects very early on and doesn’t bother to draw them, keeping framerate up.)
To draw a stupid analogy: the shorter the grocery list, the quicker your shopping. But if your grocery list has NO items and is totally EMPTY, you don’t even have to drive to the store, which really saves you time.
In the next part I will discuss world level of detail and maximum LOD didstances.
