This post is going to be a bit of a brain dump: I will explain where the various new types of data go for the new airports and ATC system.
First: everything lives in scenery packs. Since X-Plane 9, every bit of scenery X-Plane has (including the default airports, global scenery, and art assets for taxiways) all live in scenery packs. Those scenery packs live in Custom Scenery (if installed by user) or Resources/default scenery (if installed with the sim). The global scenery packs have their own folder in the main X-Plane folder so you can find them easily – at 78 GB it’s likely you might need to delete some, then reinstall later.
A scenery pack in version 9 can contain the following “stuff”:
- DSFs – that is, scenery tiles for 1×1 degree areas of the world. Those DSFs can contain a base mesh or just overlay 3-d on top of another base mesh.
- Art assets (.obj, .fac, .png, etc.) in any sub folders desired.
- apt.dat – an apt.dat file can provide replacement airport layouts on a per-airport basis. You can replace one or all of the airports.
- library.txt – a scenery pack can publish its art assets to the library for use by other scenery packs.
In X-Plane 9 we ship a scenery pack with most of our airport art assets (published to the library) and we ship another scenery pack containing Robin’s latest data – we try to update this scenery pack in each patch to match Robin’s latest.
In X-Plane 10, scenery packs are still the way you get your data into the sim. In fact, all ATC data sits inside a scenery pack too. We figure that you might want to make a custom airport with custom ATC, custom airport layout, and custom buildings. So rather than reinvent the wheel, ATC data is part of scenery data. The ATC data shows up in a few places:
- Airport ground operations data actually live in the apt.dat file. This includes “flow” information (e.g. which runways are in use, what’s the pattern runway) for various operations as well as the ground taxiway layout. The ground taxiway layout specifies a connected line graph of where airplanes can taxi. (Note that there is only one taxi layout even if there are multiple flows. This is because even if the runway in use changes, the concrete doesn’t move.)
- Airport controller and airspace specifications (including tower controllers) live in a new atc.dat file – you can put one in each scenery pack, not unlike the apt.dat file.
- Voice packs are created using a .voc file and a pile of sound files. Like OBJs, they can be put into the library – voice packs are the “art assets” of ATC.
Why is it that the ground layout is in the apt.dat file but the controllers are not? Robin collects the apt.dat file, and we wanted to make sure that if someone moves the pavement, that person moves the taxi layout too. Having them in separate files would be a recipe for a loss of synchronization.
Note that there is no mention of airport buildings in any of this. Airport building locations will ship as DSF overlays – they’re a scenery pack just like we have in version 9. The default art assets made by Tom will also live in a scenery pack and be exported to the library for anyone to use.
Since everything is implemented via scenery packs, you can create anything we can create, and you can publish your work directly via a custom scenery pack. You don’t have to share your data with any of the central collection-and-databasing efforts to use this new technology.
There will be two types of scenery packs that ship with X-Plane (and are updated via free patches during the lifetime of the sim).
First, we have default art assets that we provide to make the sim work and to help you create scenery.
- Default airport elements, including the airport building lego brick library.
- Default ATC voice packs.
Then we have scenery packs that come from collected data. We ship the latest updates.
- Default airport file from Robin, containing airport pavement layouts, gate positions, and (in v10) taxiway diagrams and runway flow information.
- Default airport building placements (in DSF form). We will start this file off but then it will become something that everyone can contribute to and use.
- Default ATC controller file, containing airspace information and controllers/frequencies. We will seed this file with center controllers to cover a big chunk of the world, but I am hoping that we can accept user-submitted corrections and contributions.
(If you squint at this list, you’ll see the divide: we are trying to involve the entire community in data-gathering activities, but we are having our internal art team make the default art assets.)
Finally, there is one more piece to the puzzle: autogen. Autogen is a horribly ambiguous world in flight simulation because it means 85 different things. So let’s be more specific: automatic in-sim generation of data. When the sim is missing data, what data will X-Plane “make up” on the fly? There are two cases of automatic data generation in the sim while you fly in version 10:
- If there is no controller specified in any scenery pack for a given airport, but there is a tower/ground/delivery frequency for the airport, Chris’s ATC engine will automatically create a controller based on the frequency. This means you get tower/ground/delivery ATC on the right frequencies for all of the towered airports, even if we don’t put those controllers in the default controller file.
- If there is no airport layout for a towered airport, we will generate a taxi layout and default flows by picking runways and analyzing the taxiway structure. This allows you to have AI ATC at any towered airport. (The layouts that X-Plane generates are not going to be as good as what you can build in WED.)
We almost always prefer pre-generation of data to in-sim generation – it lets us use a higher quality algorithm without hurting fps. (One of the limits of the ATC layout generation algoirthm is that it can’t use too much memory since it is running inside X-Plane.) But in the case of airport controllers and layouts, a custom scenery pack can contain an airport that we’ve never seen before – so we have to be able to generate the matched ATC data on the fly.
My previous post describes some of the new tech behind the airport system. It’s important to remember that we can provide a single “lego brick” in the airport system that places multiple OBJs (e.g. an .agp scene or a .fac facade with attached objects), and it’s also important to remember that simple OBJs are reasonably fast in X-Plane 10, so using a bunch of small, simple objects can be a win.
With that in mind, our current plan for gates and jetways is:
- Provide a lego brick to create a parking spot. This lego brick would be located at the same place and heading as the apt.dat start location – it would be an .agp mini-scene that would bring in the ground markings (via an OBJ with draped geometry) as well as any other nearby equipment that we want to populate and align relative to the airplane.
- Provide a lego brick jetway that has an anchor point on the terminal facade wall. The jetway will be static in 10.0 – we’ll address animation in a patch.
- Provide wall type rules that control where there are (and are not) windows, so that you can ensure that there aren’t windows underneath the jetway.
Thus, placing each gate in WED should be three clicks: jetway, ramp start, parking scene. The ramp start and parking scene will be colocated. (We could probably create a function in WED to make ramp starts for every parking scene to make this even easier.)
The jetways will align with the facades by locating the point location of the OBJ (that is, the single point where you tell X-Plane to put the OBJ) with a point on the facade wall. First, you’ll split the facade wall at the point where the jetway belongs, then you’ll place the OBJ.
Why split the wall? Splitting the wall guarantees there won’t be windows underneath the jetway. The facade wall definitions are set up to have non-glass at the end of each side. so by dividing a side in half (for example), you ensure that there will be a few meters of non-glass right where you need them. The art definitions will be set up to ensure at least enough “jetway-friendly” wall for half a jetway on either side of a segment.
We considered attaching the jetways and parking scenes directly to the facade wall, so placing a facade terminal would “auto-gen” the parking spots, but ultimately we rejected this for a few reasons:
- The facades can’t spawn ramp starts – you’d have to place those by hand anyway.
- The ramp starts might not align very well with the facade, which would look bad.
- Users who fly online or care about realism care about having real-world gate locations. Autogen gates and jetways wouldn’t provide enough precision.
- We looked at a number of proposals to autogen gates with exact precision but they were all insanely complicated.
Before I can describe the way we are planning on handling Jetways in X-Plane 10, I’ll have to describe some of the rendering technology that goes into the lego bricks; the airports are built on a bunch of new version 10 features.
- Global lighting. The airport objects come with lights that cast halos in all of the expected places. Sometimes the light is attached to a building, but there are also tand-alone “light pole” lights. This simplifies placement and arrangement because you don’t need to worry about matching LIT textures on the ground to your objects. You simply place lights and the resulting lights do what they should. If things are too dark, just place more lights.
- Draped geometry in OBJs. In version 10, an OBJ can have draped geometry that sits on sloped ground perfectly. Tom uses this for parking spot markings, etc. To place a parking spot, it’s one click in WED to place the object and you’re done.
- Improved OBJ performance. On machines with new video cards, we use instancing to draw OBJs if they are simple enough (e.g. no animation, no material attributes). This means that the cost of very small, simple objects is much lower than in version 9. Thus you can place a lot of clutter on the apron and it shouldn’t hurt performance too much.
- Attached objects on facades. A .fac facade definition in version 10 can have attached objects, the same way the roads do in version 9. In version 9 we used attached objects to add pillars to road bridges; in version 10 Tom can attach a light pole object to a facade and it will automatically be placed in alignment with the facade section that it matches.
- Custom facade wall selection. in X-Plane 10, you can pick which art asset wall definitions are associated with a given facade. For example, when you place the fence facade in WED, you can pick which section has the gate. When you place a terminal, you can pick which sections have windows and which do not. (The wall selection is made via a popup in WED – WED reads the wall names out of the .fac file.)
- Autogen Point scenes (.agp files). An autogen point scene is a draped footprint texture which is annotated with a mix of vegetation (defined by a .for file), objects, and even facades. Tom builds the .agp in Blender, forming a “mini-scene”. You place an .agp just like an OBJ in WED – you specify its location and heading – point and click. (The file is called autogen point because it is located at a single point in the DSF.)
- The ground tile from an AGP can use a special shader that adds various amounts of grit and other high-res textures; a control texture with primary colors painted into it specifies where the various high-res texture effects take effect.
Autogen point scenes allow Tom to build a building that comes with extra objects (parked cars in the parking lot, a mailbox), facades (a fence around the object) and a draped polygon all in one click. Putting it all together, you can place an autogen scene in one click and get a facade with exact wall types, lots of objects (that run quickly) with instancing, and global lighting shining on the entire situation.
All of this rendering tech is also completely available to third parties – you can make your own .agp art assets or your own facade types with custom walls, etc. Everything that the airport lego brick library uses will be available via text files in your custom scenery pack.
On my todo list for X-Plane 10.0: do something to get rid of the goofy reflections in the planet-wide water. The problem is that the reflection tricks that we pull to draw water look wrong when we zoom out so much that water surfaces are curved.
So what if we just got rid of planetary water entirely?
Apparently the ocean is “full of stars”…
Here’s another picture, lest you think I spend 100% of my time destroying X-Plane’s shaders.
That’s still in the shader development stage – I’ll toss it to Javier to work out the tuning once it’s a little bit further along.
There were a bunch of questions in the comments about the airport lego brick system. We’ve talked about it a bit, but we’ve never really described the project in one place. Tom sent out some pictures of the system as he was testing it, so I’ll repost them and explain what’s going on.
First, the basic idea: for the last few versions, X-Plane 9 has shipped with default airport layouts (built off of the apt.dat file) but no buildings. The apt.dat file is an open source data file managed by Robin Peel; Robin integrates multiple public data sources and users submit their own improvements and changes. The result is high quality airport layouts despite the lack of a good free global data source.
When we were first looking at new features we wanted to put into X-Plane 10, airport buildings were a high priority; we didn’t want the airports to be empty. Thus airport lego bricks were born. The idea is:
- Our art team builds a series of useful art components for airports (“lego bricks”), e.g. terminal buildings, hangers, light fixtures, trucks, etc.
- LR seeds part of the world with some initial buildings. I don’t yet know how many buildings we will put down. When an airport layout is automatically generated by Robin (just a runway and taxiway) we can easily put down buildings, but for a more complex layout a human may have to look at the layout and say “that’s where the hangers go.”
- WorldEditor 1.2 will have features to edit these layouts. (Tom is using WED 1.2 now – in fact, the source is posted. It’s my hope that we can get WED 1.1 out ASAP once X-Plane 10 is available.)
- LR or Robin or someone collects the placement data for these structures and integrates user submissions, as well as layouts that LR builds internally.
- The new data from all sources goes into X-Plane updates so that users get new layouts quickly.
The lego bricks come in forms you already know of: OBJs and facades, as well as a new “autogen point” file which I will explain in another post. Here is a picture of WED 1.1 with a simple airport layout that Tom built.
The green polygons are forests; the gray polygons are facades. In X-Plane 10 and WED 1.2 you can pick the wall types for each wall of a facade, so the walls are now color coded. (When you select the facade, a popup menu has names like “wall with jetway” or “glass wall”.) Placements of OBJs and autogen have real previews from the top down.
It is my hope that building these kinds of scenes will be easier than building the underlying pavement, because most of the airport elements are drag & drop objects, e.g. you just point and click; the facades are simply traced as outlines. To build a layout using these parts you won’t need to know how to use a 3-d editor, hack an OBJ file, texture-map a 3-d mesh, and you don’t have to do your own texturing work.
But what do the results look like?
ff
Not bad for point and click. The airport lego bricks leverage the rendering engine enhancements in version 10 to provide a lot of detail without having to build a custom scenery pack:
- Since X-Plane 10 has global illumination, light fixtures (either attached to a premade object or in one of the dedicated light posts) simply shine down on whatever is nearby. Put an airplane under a light post and the results look correct – no render-baking required.
- The version 10 facade engine is greatly enhanced; that terminal with jetways sticking out is a v10 facade, and it is in full 3-d.
- X-Plane 10 features new shaders that add extra detail to surfaces – that’s where the grit is coming from in the driveway in the third picture.*
- If the user enables shadows, 3-d casts shadows on everything else. Put a car next to a building, and it will be in shadow, but only at certain times of day.
We have not yet built the submission editing and review process, but at this point the plan is to collect and redistribute overlay DSFs – they are the logical container choice for a bunch of OBJ placements. We are not planning a special .dat format for these buildings.
A few other questions that have come up:
How will ATC data be handled?
Traffic flow information (e.g. where the planes taxi) sits inside the apt.dat file and will be collected as part of the regular apt.dat process. Non-airport ATC data will be collected separately; more on that some other time.
Can this be used for custom cities?
No. The scope is strictly airports. We can collect placements for airports because:
- Airports start out truly empty – the DSF generator clears them out. So there are no conflicts with autogen. For cities, we’d have to “coordinate” with the default buildings and roads, which gets complicated fast.
- The number of “things” in an airport is small – maybe hundreds if you really get into detail, compared to hundreds of thousands for a city. So it is practical to completely build an airport.
- Because one person can build an airport, we only have to worry about conflicts between submissions – we don’t have to worry about merging several parts of a city.
- The set of art assets we need for an airport is limited, so we can provide a reasonably complete library.
- Airports are really important to a flight simulator, so it’s worth all of the effort on everyone’s part to do this.
Generally speaking, I would like to increase user involvement in all parts of the scenery process. I get a lot of emails where people ask “how can I change X” or “can I fix this and send it to you”. The scenery system needs to address both highly skilled authors making complex custom payware packages and casual authors who want to do a freeware pack or just contribute to the overall quality of scenery. So we’ll be looking more at how we handle cities and OSM data in the future. For now we have our hands full just getting OSM water and roads into the sim.
Can we use other objects, custom or from OpenSceneryX?
No. The default layouts need to be built entirely from art assets that ship with the sim. We think OpenSceneryX is great, but the goal of the airport lego bricks is to solve a small, specific problem in a self-contained way, so that people can download the sim and immediately see buildings.
How does this relate to custom scenery?
The lego bricks definitely do not make custom scenery unnecessary – they provide a general set of elements for airports, but I think users will continue to want partially or fully customized airports.
A good analogy is the default instruments: because X-Plane ships with over 900 premade instruments, a user can rapidly build a decent looking 2-d panel with no photoshop or dataref work. But to make a panel that truly looks like the real plane (or to work in 3-d) custom work is needed – it’s more effort but the results are better. Airports will be similar: generic layouts will look good, but custom work will look better.
(Could a payware custom scenery pack include generic elements? I don’t know. In the airplane world, if a payware airplane contains default instruments it tends to get poor reviews. But I don’t see why a custom airport couldn’t use our lego bricks for some of the smaller elements like light poles and baggage trucks, if the main buildings were fully customized.)
* It is possible to make grit effects in X-Plane 9 in the same manner that it is done in MSFS: place a translucent overlay polygon with a high-frequency repeating texture on top of the surface you want to make “dirty. Having this capability in a shader in X-Plane 10 improves performance – we can draw one layer of “stuff” and have all of the effects go down at once.
If you want to submit an airport layout to Robin in time for the X-Plane 10.0 scenery render, please submit it to him by October 1st, 2011. See http://data.x-plane.com/ for more info on the apt.dat file.
(And yes…my previous estimate for when we’d be cutting the scenery was off by a lot. I apologize to everyone who crunched for that deadline. For what it’s worth, I told Austin that if we didn’t get X-Plane 10 out this year I’d cut off one of my fingers.)
We will not be accepting ground traffic flow layouts for the new ATC system or user-submitted building placements before the sim is released. Once the sim is out there we’ll start working on that process. Our intention is to collect and redistribute airport building layouts and ATC data (using X-plane 10’s new airport element art assets) the same way we do for apt.dat files.
This post is going to be geeky, and probably only of interest to X-Plane authors who have worked with OpenStreetMap data.
OpenStreetMap represents roads as a series of points…a lot of points. One of the problems with importing OSM into X-Plane is that the data density goes well beyond what X-Plane was designed to handle. I have seen OSM roads with a point every 8 meters!
Even if X-Plane could handle that data density, how do we fit all of OSM onto a few DVDs? The answer: convert the list of points to bezier curves.
This is the raw OSM line data for the interchange of I-8 and I-5 in San Diego. Actually, that’s a bit of a lie; the data has been reduced significantly in density. We reduce the data up front to avoid killing our DSF generation software, but also because the curve-generating process doesn’t need all of that data to do a good job.
This is the result of curve generation. The DSF generator tries to fit the minimum number of curves possible to the points by first converting line segments to approximate curves, then replacing pairs of curves with larger single curves up to a certain error point.
The error tolerance varies by road type: exit ramps are given a lot of leeway because they look good as long as they are loops; city streets are kept precise so that the curves don’t crash into buildings. Most precise: railroad tracks, where pairs of tracks are often very close together.
In this diagram, the blue and purple lines are the original curves and the white ones are the simplified replacement, showing error; the points are the actual data written to the DSF. Look at how few points we need for those on-ramps! At current settings, the algorithm reduces the size of road data in San Diego by about 25%.
WED 1.1 beta 4 is now posted – see the Scenery Tools wiki for download links. Thanks to Matthias for the patches to remember the hierarchy’s open/close state – I suspect a lot of users will appreciate this detailed touch. Thanks also to the handful of users who tested some early versions of this beta – there are a bunch of big changes in beta 4 that needed a few tries to get right. (If you find bugs in WED or any of the scenery tools, please report them here.)
This will hopefully be the last beta for WED 1.1, setting us up to move on to WED 1.2 with X-Plane 10 editing features; beta 4 contains three areas of improvement that had needed attention.
Correct Preview Order
WED used to preview scenery based on the order of your work in the hierarchy. This isn’t so good because X-Plane has its own draw order based on layer groups. The next beta will do a much better job of showing the final draw order that X-Plane will use, so you aren’t surprised when you view your exported scenery.
Saving Work When the Undo System Blows Up
There has been one data loss bug in WED that has persisted since version 1.0: if the undo system blows up, you get a nasty error (“WorldEditor has hit an error due to a bug. Please report the following to Ben…”) and no chance to save.
The next beta will automatically save your work in a special file (earth.crash.xml) that you can use to get your work back in the event that the undo system blows up.
Why does WED crash like this at all? The problem is that the undo system saves incremental deltas as you edit your project, to save memory (so we can have a gajillion levels of undo). If the book keeping gets screwed up, we are hopelessly borked because with missing deltas we can’t ever get the project back to a state that makes sense. Once we lose deltas, we’re lost. The error dialog box pops up when it does so that we can know that the problem is with the undo system – otherwise we’d just have an inexplicable crash with no idea what went wrong.
When last I checked, there were only one or two users still seeing undo problems – I’ve added additional logging, so if you do see problems please file bugs!
Changing the File Format
WED 1.0 uses a database as its file format. While this seemed like a good idea at the time, it has since proven to be a performance and maintenance problem. The next beta of WED will use an XML file format.
WED 1.1 will still read the old database format (including the format used in previous betas), so you can open your old projects. Projects saved in the new beta will not open in the older betas, but this is no surprise.
For the nerds: when I first started working on WED, I thought that it might be useful for WED files to be databases so that bulk operations could be performed on large quantities of data. Since then I’ve realized that WED isn’t a GIS or spatial database app – rather it’s a modeling tool. The new XML file format is basically a direct output of its internal editing-centric format.
My previous post announced that a previously pre-computed operation (setting up the elevation of a DSF) will move into X-Plane in version 10. We have multicore and we can save some space and improve quality.
This led one commenter to speculate as to whether the entire meshing process can move into X-Plane. Unfortunately it cannot; a lot of other scenery creation processes are tied to pre-building the mesh, and some of those are still very, very slow. But it does beg the question of how more we might move into X-Plane.
The Original Precomputation Decision
The original DSF strategy (designed for X-Plane 8.0 a long time ago) was to pre-compute all of the difficult problems for scenery tiles so that X-Plane could simply load the data and draw it. This got us four things:
- We could use expensive algorithms to build scenery without hurting sim performance.
- We could pre-optimize the scenery for fast frame-rate, even if those optimizations were slow to compute. (No one cares if building the scenery takes longer, that’s done before your DVD is burned.)
- In some cases, we could reduce data size if the finished computed output was smaller than the input files.
- Scenery is deterministic – because the work is done ahead of time, changes in the sim don’t affect how scenery looks. New scenery technology mostly goes into the off-line scenery generator, which means less backward compatibility work.
The down-side of this decision was that we didn’t have a good way to integrate or modify scenery data on the fly. Changes in apt.dat files, for example, are not reflected in the scenery, since the scenery is already “fully baked” when the sim boots.
From Precomputation to Multicore and GPU
With X-Plane 10 we’re doing a few things to move rendering work from “ahead of time” to “while you fly”, using multiple cores and the graphics card for the work, since those are areas where computers have gotten a lot more powerful.
- In X-Plane 9, terrain is selected by slope (among other things) – this is why you see steep cliffs and moderate scree above grass and forests on a mountain side. In X-Plane 10, some of this happens on the GPU – we build a single shader with three terrains, and the GPU picks amongst the terrain on the fly.
- In X-Plane 9, we pass the raster elevation mesh to the DSF loader, rather than bake it into the mesh. Eventually we may be able to pass this data directly to the GPU.
These two techniques are meant to someday play together: if the GPU has the elevation and knows how terrain changes with slope, then we can have detailed terrain on the GPU without bogging down the rest of X-Plane.
Going Further
How might we further push this technology? I can imagine that at some point, if the sim can arbitrarily increase the triangle density of a mesh using the GPU and the raw elevation data, then we could reduce the triangle density in the DSF to only the triangles needed to represent land class changes, saving DSF space.
Using a triangulation to represent polygonal vector data isn’t as crazy as it might sound; there are a number of computer algorithms that subdivide polygons into triangles to lower the computational cost of processing them. (We even use one such algorithm to simplify complex polygonal shapes – see MapHelpers.h in the scenery tools code for the scary details.)
So in the long term the triangle mesh that drives X-Plane scenery can change from an approximation of elevation (with terrain along for the ride) to an optimized way to store terrain polygons (with elevation to be applied later).
Here’s a clearing house of past links on the topic of precomputation and global scenery:
For something like six or seven years, DSF has fundamentally been a vector scenery format, meaning it contains points, lines, and connections between lines that define how scenery looks. With X-Plane 10 we’ll be adding raster data to DSF.
One way we’ll use raster data inside DSFs is to store the raw elevation data for a DSF tile. Originally we saved only the final triangulation of the mesh in 3-d; we will now save the triangulation in 2-d* and the raw elevation, which X-Plane will put back together again.
We get a few wins from storing elevation separately:
- After compression the files are actually smaller. This is because the data is more “regular” when stored in raster format than as part of the triangulation, and also because we don’t need to store normal vectors.
- Since we’ll have the elevation data in its original form, we can use it to someday enhance the mesh for graphics cards that support hardware tessellation.
If raster data is a win in both quality and file-size, why didn’t we do this originally? Two reasons:
- Originally DSFs shipped in zip files; the big win in compression with regular data comes from the more advanced 7-zip compression we started using to ship X-Plane 9.
- Raster encoding means increased load time in the sim as it “puts the mesh back together”. Today in a multi-core world this is totally moot – DSF loading happens on another core, but originally DSFs had to be loaded on single-core machines, so load time was a key performance point.**
We will also be able to put other data into the DSFs, although I’m not sure what the final file set will be. Good candidates include bathymetry data and urban-density data to affect autogen.
Finally, we get a lot of requests from plugins to access X-Plane’s elevation data; with an irregular triangulation, access via plugin isn’t practical. But with raster data, the code to locate and view the raster block inside a DSF is actually pretty easy and the data comes in a simple, easy-to-use format. This might be useful for moving maps and other such technologies.
* Technically we store the triangulation as a flat 3-d mesh; DSFs RLE encoding means that the all-zero elevation and normal-offset fields crunch down to nothing.
** The decision to make roads 2-d and set up their height at runtime is a similar decision; the original 3-d roads took up more DSF disk space to save load time.
