If you read the original X-Plane scenery web pages, you’ll see references to two file formats:
- DSFs – the files we distribute scenery in.
- XES – the “X-Plane Editable Scenery” file format, which you won’t see very much of.
Here’s the story:
When I was first working on the scenery system design, we decided on a pre-processed approach, which implied two types of file formats: pre-baked (editable source data) and post-baked (distributable finished scenery). XES is a GIS container format for the source data.
When we create the global scenery, the process is something like this:
- Import lots of data from multiple sources in multiple formats, so that it is all in one giant tile in our format.
- Process the data, deriving new information (like terrain type) from existing data (like slope) and fixing problems (like bumps on runways).
- Export the data as a DSF, which involves additional conversions (such as converting generic road types to x-plane roads) and DSF encoding.
We keep our raw data partly in XES format, and partly in the original raw format, depending on how slow the importer is – some vector formats are very slow to import (or are not already tiled), so we preconvert to XES. Other formats, like SRTM, are so easy to import quickly that we just use the data as is.
If you have ever tried to use MeshTool, you may have used XES files yourself – the landuse and climate data that MeshTool needs are saved as XES files – it’s an easy way to encode a few variable sized raster maps with portable enumeration encoding.
WED does not use XES files – when I started work on WED, I realized that the XES container format was too GIS oriented and not application-oriented, so I created a file format particular to WED. WED will continue to use .wed files, which can contain anything that WED can edit.
In the long term, I don’t see XES as being used by anyone except for LR internally; WED will continue to have a WED native format, and we will try to use common simple GIS formats for import/export – most likely SRTM hgt files for elevation and .shp (shape) files for vectors.
This thread on X-Plane.org sparked off quite the discussion. Now a lot of this is a discussion of when LR will have an overlay editor – there are a few overlay editing functions that Jonathan Harris’ excellent OverlayEditor apparently does not yet support, sparking this discussion.
(I am not saying that LR should rely on Jonathan to do an overlay editor. But I am saying that the complaints I hear about a lack of overlay editing go down when Jonathan’s overlay editor does everything that the file formats can do.)
But another part of the discussion focused on the problem of mesh editing. In particular, the basic terrain in a DSF is a fully baked output of a complex process that starts with higher level GIS input data. In other words, we start with a raster DEM, polygon coastline, apt.dat file, vector roads, and a bunch of config files and hit “bake” and a DSF comes out the other side, with a lot of processing.
This is very different than FS X, which integrates its data sources on the fly. Why did we choose a precomputed route for scenery? It has some pros and cons. (In understanding how we made these decisions, think back to what scenery was like with X-Plane 7 and ENVs and single-core machines.)
Performance
The main benefits of preprocessing scenery are performance related. When you process scenery data into the final scenery while flying, that computer power takes away from the rendering engine, thus cutting down fps. At some point you have a zero-sum game between how much cost there is to loading scenery and how complex the scenery integration can be; you have to pick very simple scenery integration algorithms to keep fps up.
(This is less of an issue as more cores become available, but is still a factor.)
When pre-processing, we can use algorithms that take minutes per DSF without affecting framerate.
Similarly, there might be scenery processing algorithms that improve fps by optimizing the output triangles – but do we have time to run these algorithms during load? With preprocessing we have all the time in the world because it happens once before the DVDs are burned.
Preprocessing also breaks a similar zero sum game between scenery data size and quality; the source data we use to make the scenery is a lot bigger than the 78 GB of DSFs we cut; if we had to ship the source data, we’d have to cut down the source data quality to hit our DVD limitations. With be-baking we could use 500 GB of source data without penalty.
Format Flexibility and Stability
The second set of benefits to preprocessing are flexibility benefits. (Consider the file format churn of the ENV days.)
– With a preprocessed scenery file, what the author creates is what the user sees – X-Plane does not go in and perform subjective integrations on the scenery later that might change how it looks in a negative way.
- There is no need to revise the scenery file formats to introduce new data sets, because new data sets and old are all processed to the same final DSF container format.
- A wide variety of mesh generation techniques can be employed because the mesh generation is not built into X-Plane. This is a flexibility that I don’t think anyone has really utilized.
- Changes of behavior in the scenery generation toolset can never affect existing scenery because that scenery is already preprocessed; this help compatibility of old file formats.
Integration Issues
There are some real limitations to a pre-processed format, and they are virtually all in the bucket of “integration issues” – that is, combining separate third party add-ons to improve scenery. In particular, in any case where we preprocess two data sources, we lose the opportunity for third parties to provide new scenery to replace one of those data sources and not the other.
Airport is the achilles heal where this hurts us most; while airport layouts are overlays and can be added separately to the scenery system, the elevation of the base mesh below the airport needs to be preprocessed. This is something I am still investigating – a tolerable fix that other shave proposed is to allow an overlay scenery pack to flatten a specific region regardless of the user setting (so an author can be assured of a flat base to work from).
Preprocessing does fundamentally limit the types of third party add-ons that can be done; with version 9 and overlay roads, we are getting closer to letting road add-ons be overlays (see this post).
It appears to me that integration isn’t the primary complaint about the scenery system (the primary complaint is lack of tools) but we’ll have to see once we have mesh editing tools (mesh recreation tools really) whether preprocessing still limits certain kinds of scenery.
Note that a lack of tools or a lack of tool capability is not an inherent limitation of pre-processed scenery. We have an incomplete tool set because I have not written the code for a complete tool set, not because it cannot be done.
(The complexity of writing base mesh editing tools is a function of the complexity of a vector-based base mesh – this is also not related to pre-processing per se.)
Tools
In the end, I think the question of tools is not directly tied to the question of pre-processing. Whether we have scenery that is processed by X-Plane or a preprocessing tool, we have the same issues:
- Good tools require an investment in coding user interface.
- The code to convert source data which users might want to edit (like a polygon that defines a lake) to data the simulator might want to use (like a list of 78,231 triangles) has to be written.
I don’t think either option (pre-processing or in-simulator processing) reduces the amount of work to be done to create a good toolset.
As a final thought, using scenery file formats that are “easier to edit” (e.g. a file format that contains a polygon for water rather than triangles) doesn’t make the total code for scenery tools + simulator any easier; it just moves the task of “processing” the scenery from the tools to the simulator itself.
An author asked me some questions that I think are so important that I’ll blog the answers:
- The new texture paging system (LOAD_CENTER) works for both terrain textures (.ter files) and draped polygons (.pol files). You do not have to use draped polygons to get texture paging – you can use paging in a base mesh!
- Orthophoto terrain via a (.ter) file is by far the preferred method for orthophoto sceneries – it is a vastly better option than draped polygons. Draped polygons are horribly wasteful of hardware resources, and should really only be used for tiny areas, e.g. airport surface areas. If you are using even a moderate amount of orthophotos, make a base mesh!
- MeshTool is the future of photo scenery, and will continue to be the way to make high performance orthophoto meshes for X-Plane.
The future of MeshTool is bug fixes, a richer syntax, and some day maybe a UI front-end.
As 920 beta finishes up, I am working on some WED features. Future WED updates will be small and more frequent; I think it’s more useful to get at least some features into WED quickly, rather than holding them until I have a huge update.
So this list is short – that doesn’t mean that other features won’t happen – they just won’t happen as soon. In particular, I am very aware that WED needs better taxiway sign editing and better polygon editing. Those will have to wait though.
WED 1.1 will provide overlay editing. WED will not be a replacement for overlay editor – Marginal has done a really great job with his tools. However, having overlay editing in WED will allow people making airports to see complete apt.dat 850 data and overlay data at the same time. This is particularly useful for authors making custom scenery where the apt.dat format is augmented with custom DSF overlay pavement and lines.*
WED 1.1 will have 3 features:
- Ability to edit these DSF overlay types: object placements, facades, forests, object strings, draped lines, and draped polygons (both tiled and textured with ST coordinates).
- Very limited preview of those types. I know you will be able to see the texture for a draped polygon for orthophoto placement. I do not know if I will even have OBJ preview in 1.1. Editing will not be WYSIWYG. It will be more CAD-like.
- Import and export of DSF overlay types to DSF overlay files.
Note that while roads and beaches are technically allowed in overlays, they are not on the list. The reason is that right now both of them require elevation data in the road and beach itself. Since WED doesn’t have a base mesh, it can’t sanely provide this information.
Eventually we will have some way to “drape” roads – at that point it will make sense to provide WED road editing too. But let’s not delay at least some overlay editing that long!
I don’t know exactly how long this work will take – my rough guesstimate is about 2 weeks. But…that depends on my working on WED for two weeks straight without interruption!
* Every type of element in apt.dat can also be created with custom-provided artwork and an overlay DSF — draped polygons for taxiways, draped lines for taxiway lines, object strings for taxiway lights, etc. The idea is to provide a way for authors who want to extend realism beyond the scope of apt.dat 850 to insert custom artwork. Apt.dat 850 is not a modeling format, so you cannot provide PNG files with it.
If you look at the LOWI demo area, you’ll see that some of the pavement in that layout is in the apt.dat file, but some pavement is in the DSF overlay. Creating this demo area required a bunch of DSF2Text hacking by Sergio and myself. With WED 1.1 it will be possible to do this completely using WED.
My previous post on scenery tools stirred up some discussion; y-man brought up a point fundamental enough that I think it warrants explanation.
The questions is whether to fix broken DSFs by editing the source data or the DSFs themselves.
Let me be clear: both are viable options, both have limitations, and neither are possible today. So in choosing one over the other, I am picking a strategy that I think is superior, and discounting the other not because I think it is useless, but because it is less useful and development time is very limited (doing both would take away from other good features).
Basically the two ways to fix a DSF are:
- Edit the DSF itself until it looks correct.
- Edit the source data and then rebuild the DSF from scratch.
I strongly believe we must pursue the second strategy for this simple reason: if we correct a DSF but don’t fix the source data, the same mistakes will be made in the future.
We have to keep recutting the global scenery to keep up with:
- Higher capacities for detail in newer computers.
- New global data that becomes available.
- Improved generation algorithms that makes better results from existing data.
To lose any of these would be a big set-back in scenery quality, so not recutting DSFs isn’t a great option. (Furthermore, improvements in scenery often come from new global data, so picking user changes over new data would be a tough choice.)
By letting users change the source data, we can have the best of both worlds: problems are fixed while new technology is adopted.
To answer y-man’s direct questions:
But that data is not available to us users is it? If it is, where is it, and where is the spec to work with it?
It is not available yet! I am proposing to focus on making the data available and creating the infrastructure to share data and receive improvements (choice 2) rather than providing a DSF mesh editor (choice 1).
Alternatively, a user who goes thru the trouble of correcting base DSF could send in the modified DSF, or may be even a diff between before and after states of DSF text files, and attach an explanation of purpose.
Here’s the problem: we can’t preserve the diff to the DSF and apply it to future renderings.
Imagine, or example, that you relocate 500 mesh vertices from the existing DSFs to correct a coastline. (I am being generous here and assuming a clear, single, unifying edit. But some authors would more likely move the vertices to make a number of improvements.)
In the meantime, another author creates an airport nearby, and someone else improves the SRTMs (there is at least one person attributed in our about box who has been collecting void-filled SRTM tiles). The effect of the nearby airport’s pavement changing size and the raw elevation changing height is to change greatly how the mesh is generated, such that 400 of the 500 vertices that were moved simply no longer exist, and 450 new vertices are in the nearby area now that were not in the original DSF.
This case is known as a “merge conflict” in computer programming terms (and happens when two changes to a program are “merged”). The problem is that we can’t sanely know what to do with our 500 edited vertices in this case. Do we take the 100 vertices that still exist and move them without the others? What if that produces very strange results? (Triangles might become inside-out because some vertices are moved a lot but their neighbors are not because they did not match the diff.)
We could try to apply some kind of change to the new vertices similar to what happened to the old, but how do we know if this is making things better or worse? What if that change simply deforms the outline of the airport that was added?
I can go on and on with these kinds of examples, but the point is this: you can’t unbake a cake. A DSF is similar; you can’t necessarily recover the sources that were processed together to form the final product.
This is why it’s important that we create infrastructure to correct source data, rather than focus on editing the final DSFs. I can assure you that my negative attitude toward editing DSFs is not a negative attitude toward user participation!
There are still a lot of details to be worked out about how we can work together. Who will own the data, and on what terms? How will it be distributed? How will it be shared?
Unfortunately I can’t answer those questions yet. I still need to do more research into what is technologically possible – then we can figure out how to proceed.
WED 1.0 has gone RC. The on
ly change from beta 5 is that I have the latest manual changes from Tom (including Cormac’s illustrations of taxiway signs). Both of them did some great work – the WED manual is a lot nicer than the docs I have done myself for the other tools.
A while ago I posted a tools update…has anything changed? Not a whole lot.
- WED is definitely the future of the “heavy” scenery tools (ones with extensive UI) – it has a lot of infrastructure for features like multiple undo, multiple selection, hiearchial editing, etc.
- The next big feature for WED will be some kind of overlay editing. I don’t think this will happen very soon though – my todo list is pretty out of control.
- In the long term, I think WED may provide a visual way to do MeshTool-like operations. In other words, you’ll be able to build a base mesh in WED by specifying polygons for airports, applying orthophotos, and importing one big DEM per tile.
- I don’t think that WED will ever be a DSF editor – that is, you won’t be able to open an existing DSF, move a single node, and re-save it. This just isn’t high on my priority list.
To this last point, how are you supposed to fix scenery if you can’t edit DSFs? Well, I’ll try to post more on that over the next few weeks, but the short answer is that we need structured edits to source data.
A DSF is a compiled result of a very complex set of processes. The vertex that you adjust in the mesh to fix a mountain top may not exist in the DSF if another user submits an updated airport layout, even if they are fifty kilometers apart. (All parts of the DSF affect each other through the adaptive irregular mesh.) So I don’t want to take user submitted corrections to the final DSF because those changes would be lost at the next render.
We’ve been down this road before – when the V6 global scenery was done, the plan was: now we’ll take user-edited ENVs. The problem was that this plan assumed that we’d never re-render the ENVs again, which proved totally wrong – we re-rendered them at the end of the v6 run with improved algorithms.
What we need to do is identify what components of the source data are reasonable candidates for user editing, and set up processes (similar to Robin’s collection of apt.dat data) to gather data and share it.
(We are looking at OSM – it is still under investigation!)
I have
blogged about correct airport layouts before, but let me bring the point up again, because it is so important:
- You must create structurally correct layouts (that is, vertices connecting to vertices, not lines) in order to get good rendering in X-Plane. Just because the preview looks okay in WED doesn’t mean your layout is correctly formed!
I wrote some documentation on the scenery site that describes the problems in more detail, with pictures.
(I’ve been trying to create more permanent documentation – it is tempting to simply blog the issues because it’s so easy to throw a blog post up, but after 110 blog posts in 2007, the scenery site is still very thin on the documentation front.)
Do Not Add Vertices To Make Smoother Curves
I really can’t stress this enough: please do not go adding extra vertices in your layout to make bezier curves look smoother in X-Plane. Why is this such a bad idea? Let me count the ways!
- X-Plane will fight you all the way! X-Plane adds vertices to curves (to make them smoother) when it detects large errors. If you have a lot of small curves, the errors are inherently smaller and X-Plane will add fewer points. So the first vertices you add to your curve do almost nothing. You have to add a huge number of vertices to get a marginal improvement in your curve. In the meantime…
- X-Plane will provide variable-quality curve rendering in the future! Curve detail should be a user-controlled setting. X-Plane has to run on a wide range of hardware; any time we can let the user pick rendering quality, this is a win, because it helps bridge the gap between the user who just bought a brand new Core 2 Extreme system with GeForce 9800 and the user trying to keep X-Plane 9 running on his G4 laptop which can’t be upgraded. When you add vertices, you take the decision about rendering quality out of the hands of the user, and force high quality on a user who may not be able to handle it. Adding vertices forces a decision of lower framerate on some users.
- Adding vertices bloats the size of apt.dat. This is not a huge factor for custom scenery, but is a factor for the default apt.dat. Robin received a big pile of new airport layouts, and that’s great. But one risk is that the total size of user submitted data could get out of control. For new layouts made with WED, vertices represent a big chunk of the data. If you are increasing your vertices by a factor of 5x or 6x to improve tessolation, you are bloating the apt.dat file.
- Manually adding vertices to smooth curves lowers the level of abstraction in the apt.dat file. Any time we can have a high level abstract representation of scenery, X-Plane has the freedom to improve rendering in the future. If your layout is made up of a large number of small curves (instead of a small number of large curves), X-Plane cannot tell that those small pieces make up some larger structure; in the future it may not be able to render those layouts as nicely as ones that are made with fewer control points.
In summary, please use the smallest number of vertices to create your layouts. (But always add vertices to ensure that your T junctions are correct!)
I had to do some research into compression algorithms recently, because we had to squeeze the global scenery onto fewer DVDs for retail distribution. We did this mostly by completely filling the DVDs, but we also had to use 7-zip compression to get about a 10% improvement in compression ratios.
DSFs are not the best test of compression efficiency because the format has been organized to help algorithms like zip compress them – the improvement with 7-zip and RAR was a lot less than you’d get with, say, a text file.
Anyway, my point here is: let’s not use RAR – it’s the new GIF. Every now and then a file format comes along with some kind of restriction that keeps everyone from doing everything with it. In GIF’s case, you had to buy the right to create GIFs, and in the case of RAR you have to buy the right to compress RARs.
I think that having these kinds of entanglements in fundamental low level file formats (like how do we compress our data or save our images) is really bad for the software community as a whole; it balkanizes raw materials. And file formats stick around for a long time – even though GIF is made obsolete by PNG you’ll still find them all over the web.
The lure of RAR is of course higher compression ratios than zip. But 7-zip can do the same thing, and unlike RAR, has the potential to be completely free, which means it can be completely ubiquitous.
Macintosh users understand the problem here: for the longest time “StuffIt” archives were the standard way to compress data on the Macintosh. The file format was proprietary, so you couldn’t even make your own program work directly with StuffIt archives. Now that zip has taken over on the Mac, getting data between Mac and Win is easy – you can just zip something using the operating system and send it to all your friends.
Let’s not go back into the “bad old days” of proprietary utilities and a lack of integration with regular apps. I say: if you can stand to use zip or bzip instead of RAR, vote for what’s open and has a future, not what is slightly better now but will just be a pain in the ass in three years.
I have posted the first “combined” scenery tools release. Basically ObjView, ObjConverter, and DSF2Text (now named DSFTool) are packaged together with DDSTool (for making DDS images from PNGs) and the new MeshTool (for making base meshes); a UI application called XGrinder provides drag & drop support for most of the converters.
The tools releases will be by month (e.g. March 2008) and each release will contain the latest for all of the tools in the release. WED and the AC3D plugin will continue to be separate downloads.
I have also posted another snapshot of the source code, and generally I’ll try to do a source code post whenever I do a tools post to keep the two in sync.
The AC3D plugin has been updated; I hope to have a new WED posted in the next few days.
Last night I created the “seed” files for MeshTool. Let me explain what these files are and why we’ll need them.
MeshTool is a wrapper around our irregular-mesh generation code. It will allow third parties to create base mesh scenery without having to create triangulations. Just like DSFTool saves people the work of having to encode DSFs (with point pools, command lists, and all that ugly stuff), MeshTool saves people the work of having to create their own triangulations.
MeshTool is a low level tool – you provide a text input file and some data. It’s designed to be an engine underneath tools like PhotoSceneryX, not an end in itself.
MeshTool will create “default” land-use terrain that approximately matches the global scenery, water, and custom orthophoto-based terrain. You (or a program you use) provide a text file that describes the boundaries between custom photos, land, water, and airports. You must also provide a SRTM-style HGT file for elevation.
How does X-Plane know what land-use should go on what terrain? That’s where the seed files come in. Our global scenery is generated from a set of rules that take into account morphology (land height and slope), approximate climate, and general land use. You provide the terrain shape via the HGT file, and we provide you with a seed file that contains climate and land use for that DSF tile.
Why do we provide the seed file rather than letting you find and create climate data? Well, our rules are tuned for a very specific pair of data sets; by providing the exact climate and land use data that we use, we assure that the rules files work correctly. The purpose of MeshTool is not to customize land use terrain, and we do not provide a mechanism for it. The purpsoe of MeshTool is to let you put orthophotos and new coastlines into the base mesh.
The good news is: seed files are tiny. They are typically 4 kb-8kb each; the entire data set is 322 MB total. That’s because the climate data is only 3×3 per DSF and the land use is only 121×121.
I hope to get MeshTool into some kind of testing within the next few weeks; if you are a programmer and would like to feed MeshTool from your own program, please contact me and perhaps I can arrange an alpha copy. I will also post the seeds as soon as I can.
