In my previous post I drew an analogy between a scenery system with its file formats and a turtle within its shell. We are limited by DSF, so we are making a new file format for base meshes so we and all add-on developers can expand the scope of our data and make better scenery for X-Plane.
The really big change we are making to base meshes is to go from a vector-centric to a raster-centric format. Let’s break that down and define what that means.
Vectors are fancy computer-graphics talk for lines defined by their mathematical end-points. (Pro tip: if you want to be a graphics expert, you just need the right big words. Try putting the word anisotropic in front of everything, people will think you just came from SIGGRAPH!) DSF started as an entirely-vector format:
All 3-d clutter is defined by lat/lon locations, so we have the vector outline of polygons, autogen blocks, etc.
The base mesh is pre-triangulated, so most base-mesh features are defined by the corners of the triangles forming lines (e.g between land and water).
This isn’t the only thing DSF does – we added raster capabilities and there is e.g. raster sound and season data in X-Plane 12, but DSF is fundamentally about vector data – saying where the edges of things go exactly.
This was great for a while, but now that we have more and more vector data (complex coastlines, complex road grids, complex building footprints) the DSFs are getting too big and slow for X-Plane.
Raster data is any data stored in a 2-d grid. This includes images (which in turn includes orthophotos) but it also includes 2-d height maps (DEMs), and the 2-d raster data we include in DSF now (e.g. sound raster data, season raster data etc). Any time we store numbers that mean something in a 2-d array, we have raster data.
Raster data has several advantages over vectors:
Raster data is what the GPU wants to consume.
Raster data has really good LOD characteristics for close detail with long view distances.
We can put more interesting and dense information into a raster tile without it getting bigger.
Twenty years ago, when I first worked on DSF, computers didn’t have the capacity to use lots of raster data – this was back when 8 MB of VRAM was “a lot”. But now we no longer need to depend on vectors for space savings.
Raster tiles are raster data broken into smaller tiles that get pieced together. Raster tiles have become the standard way to view GIS data – if you’ve used Apple Maps or Google Earth or OpenStreetMap or any of the map layers in WED, you’ve used raster tiles.
Raster tiles have a bunch of advantages too:
They have really great LOD/VRAM usage properties.
They can be loaded incrementally.
They provide an easy way to vary resolution and let authors skip providing data that they don’t need to provide. (E.g. “forest” raster data over the ocean? Just don’t provide any tiles!)
So our plan for the next-generation base mesh is “all raster tiles, all the time” – we’d like to have elevation data, land/water data, vegetation location data, as well as material colors all in raster tile form. This would get us much better LOD/streaming characteristics but also provide a very simple way for custom scenery packs to override specific parts of the mesh at variable resolution with full control.
Raster tiles are not the same thing as orthophotos. A raster tile is any data contained in a 2-d array, not just image data cut into squares (e.g. orthophotos). So while a raster-tile system may make it easier to build orthophoto scenery, it does not mean that the scenery can only be orthophotos.
At the X-Plane Developer Conference in Montreal this year I gave a presentation sharing my thinking on our next-gen scenery system. This has created a lot of interest but also a lot of confusion. So in these next two blog posts I want to start by clarifying two fundamental ideas about scenery.
Here’s the key point for the first one:
A new scenery format is not the same as new scenery.
This can be confusing because we haven’t changed either our scenery file formats or our scenery in quite a while, and often the two change together. Let’s break this down.
A scenery file format is the way we represent scenery in our simulator. It consists of several things:
A file format specification, describing how scenery data must be encoded.
A file parser inside X-Plane that can read those files.
A renderer inside X-Plane that can render those files.
Tools that help people encode those files.
My first work for Laminar Research (two decades ago) was building all of those things: I invented DSF files, wrote the DSF reader inside X-Plane (DSFLib), worked on the rendering engine, and created the tools to write the files (DSFTool).
When we talk about just the scenery, this is the final rendered files that people fly with. Remember when we shipped 60GB worth of content in 12.0.9? That was new scenery rendered out with all the latest and greatest data.
X-Plane ships with the “global scenery” – a set of about 18,000 DSFs that ensure land everywhere from 60S to 75N. But this is not the only scenery out there – there’s TrueEarth, HD base meshes, SimHeaven, and scenery made with Ortho4XP.
Lots of people can make scenery, often in many different ways (using land class, orthophotos, autogen, etc.) but all of that scenery must be in X-Plane’s scenery file format, e.g DSF.
The scenery is the turtle and the scenery file format is the shell. The scenery can only be as complex as there is capacity in that file format (shell).
So the first part of my talk was a tour of how we have outgrown DSF, and pointed out that there are some things that DSF can’t do. For example, several add-on makers want to stream custom scenery, but DSF makes that basically impossible. DSF also isn’t meant for really high detail vector data, so we’ve been having trouble using all of the latest OSM imports.
The second talk discussed our plans for a replacement to the base mesh file format, which is based on raster tiles. This part of the talk said nothing about what kind of scenery we (Laminar Research) would make, which raised a lot of questions.
But now that you understand the the turtle and the shell, you have a lens to understand what we’re saying. This wasn’t an announcement of next-scenery, only an announcement of a bigger shell that will make that next-gen scenery possible.
So the next-gen scenery format is all about potential. The scenery file format limits what is possible for all scenery (both what is built into the sim and add-ons), so we want to raise those limits quite a bit in the next-gen base-mesh format.
The way we are doing that is by moving the base mesh from a vector-centric approach to a raster-centric post; I’ll break that down in another post.
This alpha contains changes to the updater. Make backups before using!
Global Material Settings -> OBJ Settings
This new version contains one of the most requested fixes (#357, #599) for XPlane2Blender: Normal Metalness, Blend Glass, and Global Tint have been moved out of the Material Properties Tab and into OBJ Settings! The annoying “All Materials in an obj must have the same Normal Metalness/Blend Glass Value” error is gone!
The updater tries its best to guess if you wanted Normal Metalness, Blend Glass, or Global Tint, however it isn’t perfect. If you have to manually correct more than 3 of your OBJ settings, please tell me.
These new settings can be found under the Textures section of the OBJ Settings.
Emissive Panel Texture Only and Panel Mode Selector
#595 Also known as ATTR_cockpit_lit_only, this directive has actually been in X-Plane since 11.10, but now is accessible in XPlane2Blender! It makes a panel only use the emissive “Lit” texture – a great speed boost if that is all you need. This is the perfect feature for computer displays.
For Aircraft or Cockpit export types, look in the Cockpit section for “Panel Mode”. This changes the meaning of “Part of Cockpit Panel” for the whole OBJ. Setting it to “Emissive Panel Texture Only” mode activates the feature. You cannot mix panel modes.
Cockpit Regions
People who use Cockpit Regions will have to manually change “Panel Mode” from Default to Regions to see the UI and have regions export again. A one time fix. A future updater will do it for you. Until then I hope it isn’t too many OBJs to change.
Updater Version History Synchronization
#471 The updater now synchronizes its last version across every scene and new updater functions will not use data cleaning to protect against accidental or purposeful updater re-runs. Simply put this means a safer to use XPlane2Blender (but still make backups). Since this is new updater code, however, it had to be put in an alpha. People with multiple scenes should be especially on the look out for problems. I feel very confident about it however.
With testers and users like the ones XPlane2Blender has we’ll be getting to v4.1.0-rc.1 much much much much faster this time! For the users who requested moving Normal Metalness and Blend Glass, thank you for your patience. I know now just how annoying that error message was! I’ll certainly keep this experience in mind for future decisions and don’t worry, “make it the same across every material” is not going to be chosen again without an extremely important reason!
(This post was edited to reflect a discussion Ben and I had)
A constant request for XPlane2Blender and other exporters maintained by Laminar Research is an import OBJ feature. So, one last time, here is the news: by now an importer is basically never going to be a part of XPlane2Blender, but it could be its own project! It could take years of development to make it very well polished, however.
There are a few projects started on the forums you can checkout if you’d like!
Why Is This Such A Hard Problem?
It is simple to read the vertex table, attributes, animation keyframe table, and turn that into Blender data, some projects can already do this, in fact. The trouble is getting the exact or nearly the exact .blend file back.
OBJs Don’t Save Everything You Want And Need As An Artist
Some of the things things you could never import from an OBJ
Window Settings
Text Blocks with scripts, notes, or annotations
Any non-XPlane2Blender Blender settings we don’t care about
Object, layer, material, and texture names
A bunch of settings that can’t easily be inferred (more on that later)
Blender’s parent child relationships
If the OBJ was produced with comments and correct indentation, you might be able to get some of these things back (likely just a few names.) A large complex Blend file without this stuff is a nightmare of an unorganized mess and would make the rest of the manual reverse engineering process even harder.
Multiple .blend files Can Produce The Same OBJ Content
If A.blend, B.blend, and C.blend can produce the same OBJ, which one should the OBJ be reversed engineered back into? The relationship between XPlane2Blender settings and what appears in the OBJ can be very esoteric and there is not always a 1:1 relationship. Some ATTR_s only appear when certain combinations of settings are used. You may find there are absolutely no good defaults.
A Moderately Smart Importer Would Need To Know Massive Amounts of History Of All Exporters
In order to perfectly solve these ambiguities and produce .blend files that are similar to what originally exporter the OBJ file, it would be incredibly useful to know about the behavior of the exporter that exported the OBJ in question. This means an importer would need to know all the bugs and features of every exporter, and we don’t even know that after developing the exporter. Bugs are waiting to be discovered, or used by artists until they have to be turned into a feature. Our exporter currently struggles to take into account past bugs, and that’s the exporter!
This turns into it’s own ambiguities to solve: “Is this OBJ’s mention of deprecated ATTR_s a choice the artist made. despite the deprecation warning, or was it a bug that this was still getting written, or was this OBJ written before it was deprecated?” Now you’re messing not only with valid or not, but what the OBJ is supposed to look like.
Optimizations Create Further Challenges
Exporters often take optimizations to improve an OBJ’s performance and quality. For instance:
Ignoring or appending ATTRs to handle deprecations or obsolete OBJ features
Now you will have even less information or, now, seemingly incorrect information! OBJs are meant for X-Plane, not humans. As such exporters can take many liberties with the content of OBJs as long as they match what the artist meant. This can result in very complex optimizations that might even break our own guidelines, all to deliver the best (and deliver on time) to the consumer and our artists. This makes developing an importer that reproduces the importer OBJ, either exactly or simply visually matching (let alone animated or textured properly) even harder.
In a raw .blend file, objects are like Lego bricks which get baked into one solid piece. Going in reverse will likely not get the same neat separation. Blender is not smart enough to tell what is a wing shape, what is a wheel shape, what is a throttle level shape. It may be impossible to separate the vertices back into these distinct meshes (especially after optimizations)! Making a “really smart” importer that is shape aware is a brutally hard algorithm that the world’s best computer scientists are still attempting to solve. It may not be challenge you want to take on.
We May Build A Reasonable Importer One Day!
With all these challenges, an importer would have to be willing to not handle all edge cases and not attempt to reverse engineer an OBJ back to the exact .blend file that made it. For instance, a 3 year old OBJ would not be reversed engineered into the .blend file that produced it 3 years ago, especially since an artist would want to then update their assets anyway! Having art assets “marooned” on other exporters or “dead” is a pretty terrible waste of time, likely be more painful than hand fixing all lot of little things. As Ben pointed out “If we can’t make a direct import [.skp->.blend, .ac->.blend] path, OBJ import is the least bad option.
2.49 to 2.7x Converter
A 2.49 converter is a much more manageable project (far on the back-burner.) The complexity of this tool is much more manageable, because the 2.49 is not in active development and you are converting .blend to .blend, not .obj to .blend. Blender to Blender is something which Blender is already very good at.
While everyone looks at the new UI in awe, X-Plane 11 also had a few important changes under the hood. With Aerosoft Navdata Pro now also supporting X-Plane 11 beta, let’s talk about one of the most boring technical features of X-Plane 11: The completely redesigned database for navigational data, which makes it much easier for data providers like Aerosoft and Navigraph to supply data updates for the X-Plane navigational facilities, while preserving scenery compatibility.
The most important goal when designing the new database was to eliminate the duplication between data in X-Plane’s world and X-Plane’s navigation systems to leave less room for subtle inconsistencies. I also wanted to address compatibility of navdata updates and global scenery (mostly concerning localizers at airports). Other improvements were the integration of SBAS path points (needed for LPV approaches) and RNP service volumes. Last but not least I wanted the ability to work with ARINC424 data directly, and eliminate most of the subtle encoding differences that result from different providers generating files with slightly different converters.
The specification of the database was finalized in September, and both Navigraph and Aerosoft were provided the tools they needed to create navdata for X-Plane 11 in the new format. Actually, we are not limited to those “big two” – as the tool is available for everyone, open-data purists can actually generate their own navdata for the US and Canada using the FAA’s file.
With the great power of the unified database comes great responsibility: The navigational data can only be as good as the world scenery it is placed in, especially the airports. Some of X-Plane’s airports in the default scenery have not kept up with the pace the real world is evolving at: runways are renamed (due to magnetic shift), extended, built or closed and X-Plane’s airport scenery is only as good as the community who cares for it. To make their life easier, we are currently working on a big automated scenery update on the server side. We will rename several thousand runways all over the world on the scenery gateway soon, and this will solve the most annoying issue people are currently facing with the new database: runways not being found because they have been renumbered.
This automatic scenery update is however only part of the solution – because we can only rename runways we have! If an airport is extended in the real world because new runways are built, we rely on the scenery gateway and its incrediblecommunity for updated airports.
I took the time to write an even more boring technical article on how everything works together in X-Plane 11: Navdata in X-Plane 11. If you are an end-user, you don’t need to bother, because here’s the TL;DR: It’s awesome. It gives you RNP approaches for airliners, and LPV approaches for GA aircraft.
Writing this article though, when I compare it to the new UI, I can’t help but feel like the poor guy in this webcomic because this is exactly how the end-user will experience the change: Most won’t even notice.
A user asked me whether we would ever have a conditional directive in the OBJ format to let an author vary the look of the object based on weather. My answer:
Definitely not – this is definitely not the intention of conditionals!
The goal of conditionals is to allow authors to cope with multiple “rendering modes” of the engine, where:
The look of the sim is radically different between the rendering modes and
The performance cost of working around the lower rendering mode is expensive.
So for example, when shadows are off, you have to use some ATTR state change to drop a fake shadow on an object. This kills instancing, so when shadows are on (and the sim needs more performance), leaving that double shadow is not only ugly, but slow.
The conditional works by fully “stripping” those commands for shadowing from the object, then re-analyzing it.
So the win of conditionals is that you do not pay for what you don’t use, because everything is done on load – it’s like shipping two highly optimized objects.
But the down-side is: conditionals require scenery to reload, which is slow, disruptive and annyoing. This is why we would never use them for something that is dynamically changing in the sim, like weather or time of day.
We do want to have weather effects, but our approach is going to be very different: to have a “weather map” texture on the object (same UV as the object) indicating what sections of the texture will receive changes to their material due to weather effects. This will allow the object to look progressively more wet over time without a reload as the rain falls.
X-Plane 10.45 fixed one of the big “ecosystem” problems with the global airports by excluding airport objects by ICAO code. This makes the global airports much less likely to conflict with custom scenery, even if that custom scenery lacks exclusion zones.
For 10.50, I am looking at another change to how we manage airports that should help gateway authors: per airport control of flattening.
As of X-Plane 10.45, airport flattening is a user setting; a user can pick to have all airports flat, or all airports follow the terrain contours.
This is a rather silly setup. One size does not fit all for airport flattening, and the author of the airport is more likely to know how the airport will look with/without flattening than a user who is flying to that airport. It certainly isn’t efficient to have everyone in the X-Plane community set flattening individually without authors being able to set up their airport the way they want.
From what I’ve seen, there are two legitimate uses of airport flattening.
To fix bugs in the underlying terrain, e.g. if the DSF is screwed up, then the airport may need to flatten it to make the airport usable at all. We want this to be the exception, not the rule. (X-Plane 10 is more buggy than past X-Plane releases WRT bumpy runways, so this may sound funny right now, but overall we aim to have users be able to fly with non-flat runways.)
To accommodate highly customized airports where the 3-d models depend on a flat surface.
In X-Plane 10.50, it should be possible (using a new version of WED) to mark an airport as “always flatten”. The expected usage is:
Users leave “runways follow terrain contours” on, all of the time.
Authors mark individual airports as needing flattening, e.g. to fix bugs or accommodate custom scenery.
There is one use case that isn’t handled well by this: if an airport needs different flattening based on different base meshes, there is no way to tag that in the airport. But I view this as a fairly difficult problem to solve with existing technology – we would need a 2-d grid matching every custom version of an airport against every version of the mesh ever to exist.
Fixing the Mesh
Please note that this feature is not a replacement for being able to customize mesh elevation at a local airport from an overlay. “Mesh patching” is what we want to be able to do in the long term, but for X-Plane’s engine, it also means a lot of complicated internal changes to how the rendering engine works; customizing flattening is something I can ship now in 10.50 that at least helps.
Flattening is also not a replacement for fixing bugs in the base meshes themselves; one trend I have observed over my decade+ working on X-Plane (!) is that the accuracy of source data keeps getting better. Ten years ago it would be silly to say “how about if everyone just uses the real world values for their scenery and it will just work”. At this point that’s not actually a pipe dream, it is often completely manageable. So my hope is that someday we can reach a point where the terrain is just accurate and everyone uses it.
As of right now I have this code working in X-Plane but I do not have a build of WED that supports it. We are still in the middle of working on 10.50 apt.dat features, so I’m hoping to post something on that soon.
Ondrej has posted a public thread about the new version of the Blender 2.7 exporter here.
The 3.3 release is the first release where we (Laminar Research) have worked closely with Ondrej to build what we hope will be one of the future cornerstones of modeling for X-Plane.
If you are currently using Blender 2.49 or AC3D, I expect that 2.73 will provide the best way forward for modeling in X-Plane.
New Stuff
The new release has a few major features, all aiming to create a new modern work-flow:
All animation bugs are fixed. (At least,we think – if you find one, please report it ASAP!) This means animation is WYSIWYG. Armatures are supported for animation as well as animated data block containers.
The exporter understands all modern OBJ features, including ones scheduled for release in X-Plane 10.50.
Instancing is handled via a single option with exporter validation, so you don’t have to know how instancing works to get instancing.
The material rules are validated, and materials are found automatically; you can simply model as you want and Blender will do its best to export it or tell you if there is a problem.
Specular and Normal maps are ‘merged’ together from two separate sources. This lets you set specularity and normal maps in separate material channels in Blender for a more WYSIWYG approach. It also means no more messing with Photoshop to combine these layers.
The goal of many of these is to hide the idiosyncrasies of X-Plane’s modeling format and provide a cleaner, more artist-friendly view of modeling.
Regarding instancing: the model we have adopted is the one we used internally on the 2.49 exporter: you (the artist) tag an export as either “instanced” or not.
If instancing is on, the exporter writes only data to the OBJ that will be hardware-instanced by X-Plane. If you do something that cannot be instanced, like animation, you get an export error telling you what you have to remove.
If instancing is off, the exporter writes the object normally.
The win here is that you don’t have to know the (complicated) instancing rules; set instancing for the scenery objects you need to make fast in bulk (e.g. a luggage cart, a house, something that will be used many times in a small area) and you’ll get optimal output.
Optimization – Coming Soon
The goal of the 3.3 release was to set up an environment where authors could work using the new work flow. We have not finished optimizing the OBJ output.
If you are using the existing 2.7 exporter for airplane work, the output should be similar in performance. If you are using the 2.49 exporter, the new output is (like the current 2.7 exporter) not as well optimized.
In a future update, we will tighten up the generated OBJ code; this should not affect anyone other than producing better OBJs.
Compatibility
The new exporter should be fully work-flow compatible with the previous Blender 2.7 exporter; if you find your existing project does not work, please file a bug!
Our plan is to create a migration tool for Blender 2.49 projects; this will forward-convert the datarefs on annotations, manipulators, and object properties from 2.49 to 2.73 format. This lets authors using 2.49 move forward to 2.73 and have a migration path for existing content. (This work is not started yet, but is planned – we have our own aircraft we need to keep working on.)
MH1212, developer of the Prefab Airports for X-Plane, requested this feature, and it looks like we are going to be able to sneak it into X-Plane 10.45. (If we hit bugs, it might get pushed out to 10.50, but so far things look okay.) The feature is: the ability to exclude objects by airport ID without using exclusion zones.
Right now when a custom scenery pack replaces an airport (via apt.dat), the old apt.dat is completely ignored. But the DSF-based overlay objects, facades, etc. are included; the custom scenery pack has to use exclusion zones to kill them off.
With this extension, the DSF-based overlay objects in a scenery pack can act as if they are in the apt.dat file, disappearing when the apt.dat airport is replaced. This means that when you replace an airport (via apt.dat file) not only do the runways go away, but so do the overlay elements.
Here’s the win: we can export our global airports from the X-Plane Scenery Gateway this way, and custom scenery will remove the overlay elements automatically just by replacing the apt.dat, even if no exclusion zones are present.
Here are some details:
The scheme works if the underlying airport is correctly marked as having its objects and facades “inside” the airport. So unlike exclusion zones, this scheme works if the underlying airport is modified, not the overlaying airport.
This is a new scheme – no existing scenery already does this; scenery must be re-exported to gain access to this functionality.
The functionality requires the latest version of X-Plane, but is harmless to old X-Planes – the DSFs will still load.
Exclusion zones still work and are still recommended; if you are making custom scenery and you are on top of autogen or an old scenery pack that is not modified using this new scheme, only an exclusion zone will save you.
There are two big advantages of this scheme:
We (Laminar Research) re-export our collection of thousands of airports on a regular basis, so we can tag the entire set of global airports using this new scheme and have them ready for by-airport-ID exclusion very quickly. So this scheme will start to help conflicts immediately.
The scheme doesn’t require modifying the overlying scenery at all. There are freeware airports that are effectively orphaned – the author cannot be found and the license doesn’t allow the community to modify them*. Since these airports cannot be legally redistributed with exclusion zones, this technique will help.
Once X-Plane has this extension and the global airports are re-exported using it, global airports will fully disappear when any custom scenery pack replaces the airport by apt.dat, even if the custom scenery pack doesn’t have good exclusion zones.
This functionality will be available to third parties in WED 1.5 when it goes beta. In WED 1.5, if an overlay element is in the folder for an airport, it will be excluded when that airport is excluded. If an overlay element is ‘loose’ in the outer level hierarchy, it will not be excluded by airport (but will be affected by other pack’s exclusion zones).
Since gateway airports already have the objects “in” the airport folder, they are already authored to make this scheme work.
If you create your own DSFs using a low level tool like DSF2Text, the DSFLib source code, or something else crazy, I have posted the proposed schema here. That’s a technical link for people tinkering with the DSF format itself, but if you’re in that category, please do ping me to get early test materials. (The new code is also on GitHub.)
Two warnings to custom scenery authors: if you are creating a custom airport scenery pack, especially payware, please read these very carefully:
This is not an invitation to stop using exclusion zones. There are plenty of scenery packs in the wild that do not have their objects tagged by airport, as well as autogen and all sorts of junk that can be under your payware. If your airport doesn’t have an exclusion zone and it conflicts with another pack, it is your fault. Go add exclusion zones, like I told you to do yearsago.
If a scenery pack from X-Plane Airport Gateway is removed by your pack (by airport ID) then our exclusion zones are removed too. This means that if trees on the runway have been removed by an X-Plane Airport Gateway airport, you will no longer get those exclusion zones for free. You must exclude those trees yourself! (If you put exclusion zones in from point 1 this is a non-issue.) Test your airport with global airports enabled and disabled to make sure your pack is good.
* As a side note I consider it a real problem that airports get uploaded and shared with the community under licenses that don’t allow for abandon-ware to be maintained. It’s clearly the right of the author to use any license you want, but as a community I hope we can encourage freeware authors to use a permissive open license.
