I’ve commented in a past post on the cost of vertices in an OBJ. Here’s a general thought on your 3-d modeling: consider using more vertices in your OBJs.
When I started using X-Plane as a user/third party developer (back in X-Plane 6) every quad was hand-coded and removing the floors from building cubes was a key optimization.
Fast forward a decade and things have changed. X-Plane can draw a lot of vertices. Go look at one of the big oil rigs or the aircraft carrier. Does your frame-rate blink? probably not.
A few reasons to at least consider using more vertices:
- If X-Plane is limited by CPU or fill-rate while showing your content, the vertices are basically “free” – the GPU can probably draw more vertices without hurting fps.
- Since X-Plane 10 will feature dynamic shadows, it’s going to be a lot more obvious what’s really drawn in 3-d and what’s just a nicely painted flat surface.
- Similarly, with global lighting, lighting on your scenery may come from many directions, including from multiple landing lights on the airplane. The multi-directional light will emphasize more correct 3-d.
Here are a few pictures of LOWI. Now we shipped LOWI as a demo area a long time ago. But this is LOWI with global shadows and lighting from X-Plane 10.
Looking back at LOWI, it could have even more 3-d; Sergio got a lot of leverage out of his textures. But the 3-d that is there helps make the shadowing work correctly.
Authors are already getting dinged in payware aircraft reviews for not going full 3-d in the panel (that is, for building parts of the panel via paint instead of real surfaces); I think we’ll reach a point where scenery is evaluated the same way.
Every time I talk to a 3-d modeler, I hear the same thing: the 3-d modeling is the quickest part; UV unwrapping and texture painting takes a lot more time, and wiring up the model to systems and animation is worse than 3-d too. So maybe it makes sense to model the 3-d in a little more detail.
If you do want to push the 3-d, please be aware of the following performance issues:
- Object 3-d costs VRAM at a charge of 32 bytes per vertex. You can get a lot of mesh out of VRAM – 32k vertices per MB of VRAM – and you might use 8 MB for a 1024 x 1024 day/lit/normal texture set. (In other words, you can have 250k vertices for the cost of your textures.)
- You pay for all of that VRAM even if the low LOD is drawn, but you pay nothing if the OBJ is skipped. So for seldom-used or one-time-used objects with huge vertex count (like an airplane fuselage or airport terminal) it might be better to have a “details” object that is fully separate from a main building. The details object can have a super-low LOD and you don’t pay for VRAM from 5000 feet in the air.
- If an object is repeated a lot, the vertex count can be an issue. A 10k vertex object is not a big problem until it is drawn 5000 times on screen. So if your object is used a lot (like an autogen house or a sign attached to a road) make sure that the LOD with a lot of triangles has a low distance! (This is how we can have such complex airport runway light fixtures and still have 8,000 per airport – the LO is really low.)
Finally one other tip for future-proofing with version 10: version 10 will have particular enhancements if the LODs of your OBJ are “additive” – that is, if the high detailed LOD is the low detailed LOD plus extra pieces. I’ll explain that in more detail when v10 is in beta, but basically if you do a basic building and then “decorate” it you’ll be able to use a fast path in v10.
Edit: the original post listed the version as 2.0 – the new WED beta is 1.1, MeshTool is 2.0, and my brain doesn’t work in this heat.
More scenery tools: I have posted a new beta for WorldEditor 1.1:
Like previous betas, this build mostly fixes bugs around import and export, which was the area that caused the most trouble. Please file WED bugs via the scenery tools bug base; I don’t often get to work on WED, but when I do, I can kill off a bunch of bugs at once if there are good reports. Thanks to all of the users who have submitted good bug reports with example scenery packs to help reproduce the problems.
Future WED Versions
Simultaneous to bug fixing on the WED beta, we are adding new features to support X-Plane 10 ATC: WED will be the tool you use to edit taxiway routing diagrams and airport layout flow information for an airport. (This means that you’ll have your physical pavement, your routings, and your custom scenery all in one environment.)
These features are disabled in the WED 1.1 betas, but once version 10 is released and WED 1.1 is final, we should be able to rapidly go to beta on the new features; since we use WED to build our test layouts now, it should be in reasonably good shape.
In the long term, I’d like to see WED provide a visual front-end for MeshTool*, but I won’t let this hold up a release of an ATC-ready WED; so far having “future features” floating around in the code (disabled for release) hasn’t caused any bug reports or stability problems for the existing 1.1 beta candidates.
WED also needs a file format change; my original decision to use an sqlite3 database was the wrong one; when the file format change happens, there will be full backward compatibility for reading old projects, so you’ll be able to bring your old projects forward just by opening them; you won’t have to reimport the DSFs and you won’t lose your groups and other WED-specific information.
* There’s no reason why WED has to be the only front-end for MeshTool, and it may not even be the first one released. One option might be to make a MeshTool export script for qGIS.
First, two quick notes on comments: all comments are moderated, so if you post a comment and it doesn’t appear, please don’t repost. Sometimes Chris and I are both out for a few days, and the comments sit there. Without moderation, WordPress-based blogs get the hell spammed out of them. Chris has also turned off commenting on very old posts. We’ve had a few cases of people digging out old threads to try to jump in to a discussion after the fact, but a lot of what goes onto the blog is time-sensitive – by the time a month is up, a newer post covers the topic in more detail.
Now on to tools: I have posted new release candidates for MeshTool, version 2.0 release candidate 6:
MeshTool has been in a “perpetual release candidate” state for a while. I think this is partly because we don’t eat our own dog food when it comes to MeshTool – the tool shares a lot of code with our default DSF generator but they’re not the same thing. Not surprisingly, the MeshTool code that is specific to MeshTool and not used for global scenery is where the bugs crop up.
So my thanks to those MeshTool users who have patiently sent me bug reports and reproduction cases – I’m hoping that this release candidate is final.
What’s Next for MeshTool
I think MeshTool will go in a few different directions over the next year:
- The scenery tools code is “branched”, with two forks. The older fork targets X-Plane 9, and the newer one targets X-Plane 10. I suspect we’ll eventually have to have two MeshTool versions, 2.xx for X-Plane 9 scenery, and 3.xx for X-Plane 10 scenery.*
- One logical next direction for MeshTool is to add “processing” features. For example, we have code to roughly flatten an airport area – useful when using SRTM data (where concrete runways make false radar returns that turn into tiny hills). This code could be made available from MeshTool.
- The other logical direction is to put a real user interface in front of MeshTool. Just like WED and OE provides graphical ways to place objects (where-as DSF2Text is low level), it would be easier to work with a mesh scenery if you could see your work, and have MeshTool run automatically in the background.
I don’t know what the time frame for future MeshTool work will be; future MeshTool work isn’t necessary to ship X-Plane 10 so it may have to wait a while.
* This is really not ideal, and it is not the way that most other compatibility will work. Wherever possible, I’m trying to make the tools simply work for both X-Plane 9 and 10 in a single tool. It’s less code for me, and more useful for you (because when we need two tools for two sim versions, inevitably one of the two tool versions will be less feature-filled than the other).
“We eat our own dog food” is a term in software development for a company that uses the product it sells for its own operations. The idea of “dogfooding” is that developers who use their own software will have much stronger incentives to make the software usable, productive, and reliable.
Laminar Research practices a form of dogfooding with the scenery tools:
- All of the scenery tools are open source, so we don’t have any “special secret magic tools” that you don’t have.
- Our art team uses the same set of tools that third parties use.
Dogfooding does have an effect on development: when the artists get on my case about the usability of a scenery tool, it gets my attention. We pay those guys to churn out a lot of artwork; if they aren’t productive, we can’t get the sim out.
However, there is an Achilles heal to dogfooding: you can only dogfood software that is useful for your company’s business. We have this problem with MeshTool: since we don’t make our own orthophoto sceneries, MeshTool doesn’t get heavy internal use, and as a result I think it’s been a lot slower to become usable. The MT users who have filed bugs have been very helpful, but I feel for their suffering with a very raw tool.
Dog Food and Version 10
For version 10, we are producing art assets that use new rendering technology for the very first version. This means we’ve had to build the tools to support new version 10 technology early on, and that’s good news for third party developers. When version 10 is released, I’ll package up the various version 10 tools we’ve created for public consumption.
In most cases, the tools need additional polish – our artists often use tools in very early, buggy, raw form. But what we already have is a big head start, which should shorten the time from version 10 release to a full set of tools.
I’ll post more over the next few days regarding some of the specific tools we have and what we will do with them once version 10 ships.
What is the limit to how precisely you can build scenery in X-Plane using WorldEditor?
In this picture, I created a runway “OBJ” exactly the same reported size of the runway, in the same location and orientation in WED. The windsocks are also placed on the exact corners of the runway in WED. This is a typical “MRI” scenery pack – that is, a test scenery package designed to show how well/poorly the sim is performing.
So what do we see?
- The alignment of the windsocks with the runways is pretty damned good. It might not be perfect, but it’s awfully close.
- The alignment of the OBJ runway and the real runway below it is not particularly good – the heading isn’t quite perfect enough to stay aligned over a 10,000 foot runway.
This lack of precision is a design limitation – that is, X-Plane does not try to provide unlimited precision in scenery authoring. (At some point, increasing precision increases file size and processing time, which would have a detrimental effect on overall system performance.)
In particular, X-Plane tries to provide high quality point-to-point matching. That is, given two locations that are both specified by latitude and longitude, X-Plane tries to get them to be as close together as possible.
What doesn’t work well is trying to correlate a specific location (specified by latitude and longitude) with another location specified by relative location (e.g. from this location, go 50 meters to the west).
Generally, the longer the “arm” of relative location, the worse the precision. in this example, the runway is specified by the center of the ends, and the windsock locations are specified directly. Thus the runway corner is on a 25 meter “arm” from the runway end center. We may pick up a little bit of imprecision on a 25 meter arm, but the correlation is good.
By comparison, the object end is on a 5 km “arm” because it is specified by its center location and heading, while the runway end center is specified directly. As you can see, over a 5 km arm, a fair amount of error is accumulated.
Thus we can come up with a working strategy for scenery placement:
- When placement precision is important, keep objects small.
- When two scenery elements must be locked together, be sure to use scenery elements that can be specified by exact location and not by a metric distance from another location.
- “Looks close” in WED only means “pretty close.”
The hardest part of this work-flow is using art assets other than objects for large scale features. But (for example), a draped polygon is better both for precise positioning over a large area and for polygon offset/Z-thrash, so you probably need to use draped polygons anyway.
In my previous post I suggested a few ways that we will try to ease the introduction of new rendering technology in v10: complete compatibility when the feature is off, close-as-possible compatibility when it’s on, and a simple path to migrate to fully using the new tech.
But: if you’re doing something that should be illegal, we can’t help you. And this can be tricky because X-Plane 9 doesn’t always complain about illegal content.
Propsman just hit a case of this: he ran one of his old plugins on an X-Plane 10 development build* and discovered that (unlike X-Plane 9) it accidentally disabled most of X-Plane 10’s drawing.
It turns out that his plugin wasn’t quite adhering to the OpenGL guidelines for X-Plane plugins. But the particular state he was leaving altered had no visible effect on X-Plane 9. That’s not the case for X-Plane 10. So he’ll have to fix his plugin; the fixed plugin will work correctly with both X-Plane 9 and 10.
Unfortunately, it can be hard to find these kinds of problems without a build of X-Plane that actually has problems when you violate a rule. So if you make a third party add-on, when we finally do have beta builds available, I encourage you to check your add-ons for possible plugin or authoring mistakes that might actually cause problems in X-Plane 10.
*Let me nip this one in the bud before it gets out of control: Propsman is one of five authors who are working on the new art content for X-Plane 10. These five authors have development builds of X-Plane 10. No other authors have X-Plane 10. We are not giving out copies of X-Plane 10 right now. We are not loking for testers. Do not ask me for a copy of X-Plane 10. Do not ask me to test X-Plane 10 early. If you send me an email angling for early access, I am going to mark your email as spam in Thunderbird.
Two more pictures from the test package Tom sent me. These illustrate both some cool things that happen in X-Plane 10 with global lighting and the process of adopting the new features. Our strategy for the new rendering features in X-Plane 10 is 3-fold:
- When the new features (shadows and global lighting) are off, scenery that works in version 9 should just work. So users always have the option to turn off the new features, use existing scenery, and get some fps back.
- We try to minimize the artifacts between new features and old scenery.
- We try to minimize the amount of rework necessary to be fully compatible with new features. For example, switching from ATTR_poly_os to ATTR_draped is a simple search-and-replace job.
The hangar during the day, with shadows with visible skylights, required some bug fixing in the engine, and a new attribute. Normally an object is either blended or not blended. The problem is that a blended object doesn’t let light through for the purpose of shadows. (Even with the “bug”, the hangar with shadows still looked pretty good!)
X-Plane 10 introduces a new OBJ attribute, ATTR_shadow_blend, that will make an object translucent for rendering (note the grime and dust on the windows) but fully transparent for shadows (hence the skylights let light through). The attribute works the same as ATTR_no_blend syntax wise, making the update quick.
(You don’t have to use this attribute, but without it, the sim may not be able to produce quite as nice shadows. Note that this attribute is not necessary for objects marked “glass” on an airplane – they are already handled by a separate process.)
There’s a second bug, visible in both pictures, but more visible in this second picture; the static airplane, which is from version 9, contains a ‘fake’ shadow, consisting of a single quad on the ground via ATTR_poly_os. During the day, we have a double-shadow (both the one generated by the sim and the fake one) and at night the fake shadow does not go away.
With X-Plane 10, this kind of problem (a technique that is useful in X-Plane 9 clashes with new rendering settings) can be addressed via the conditional OBJ commands. The conditional commands let you specify that certain parts of the OBJ are only to be used if the user has shadowing off (or global lighting off), for example. Thus the old shadow works for users who turn off shadows, but goes away when the sim takes care of it. The same technique can be used to have two versions of LIT textures (or even remove LIT textures) when global lighting is turned on and off.
Poor Tom…he sends me his work in progress (at my demand) so I can more easily investigate sim bugs and I turn around and post it to the developer blog.
Tom is working on the new airport scenery library. This is something that Sergio had started to work toward with version 9 (you’ll note that in version 9, a lot of the custom elements from LOWI are actually in the library), but with version 10, the goal is to have a complete set of library art assets. This will facilitate:
- Casual authors creating detail at local airports without having to create their own 3-d models.
- The X-Plane community to share default airport structures in an open source database, the way we already do for taxiway layouts.
The spill lighting in these pictures comes from version 10’s global illumination – there is a light source at the top of each light pole, and it casts light down onto whatever happens to be below. There are a few reasons why we think this is going to be a great feature for scenery creation:
- It’s really easy to use. A light source is part of an OBJ (like light billboards now – in fact, the light source is added to an OBJ using LIGHT_PARAM or LIGHT_NAME) so adding lights is as simple as placing objects in OE or WED.
- Since the lights simply throw spill, you don’t have to worry about whether the light is aimed at an object that you can then “draw” lighting onto.
- The lights are fully dynamic – if yo animate the object, the light moves. If you drive your airplane under the light, the airplane is lit up.
With X-Plane 9 (or any rendering engine that can’t do global lighting) you have to “bake” lighting in order to create night effects. In other words, you have to draw the light effect onto the textures the light shines onto. With real lights, you don’t have to set up any baking or complex texturing layout, you just place the light and you’re done.
Not having to bake is good for plausibility. In real life, light shines all over the place – in the case of the fuel depot picture, the light illuminates not only the asphalt road, but also the grass nearby. When creating LIT textures, the author is forced to constrain light effects to parts of the scenery that actually have LIT textures. (We had to do this for X-Plane 9 – the lights on the highway don’t ever illuminate the ground nearby, because we can only light the LIT texture.) Global lights give you spill onto all sorts of nearby surfaces for free, without setting up a ton of complex baked textures.
(The global lights do not cast shadows – only the sun does that. The shadow below the airplane at night is a bug – more on that tomorrow.)
Hint: it might not be what you think! Vertex count isn’t usually the limiting factor on frame-rate (usually the problem is fill-rate, that is, how many pixels on screen get fiddled with, or CPU time spent talking to the GPU about changing attributes and shaders). But because vertex count isn’t usually the problem, it’s an area where an author might be tended to “go a little nuts”. It’s fairly easy to add more vertices in a high-powered 3-d modeling program, and they seem free at first. But eventually, they do have a cost.
Vertex costs are divided into two broad categories based on where your mesh lives. Your mesh might live in VRAM (in which case the GPU draws the mesh by reading it from VRAM), or it might live in main memory (in which case the GPU draws the mesh by fetching it from main memory over the PCIe bus). Fortunately it’s easy to know which case you have in X-Plane:
- For OBJs, meshes live in VRAM! (Who knew?)
- For everything else, they live in main memory. This includes the terrain, forests, roads, facades, you name it.
Meshes In VRAM
If a mesh is in VRAM, the cost of drawing it is relatively unimportant. My 4870 can draw just under 400 million triangles per second – and it’s probably limited by communication to the GPU. And ATI has created two new generations of cards since the 4870.
Furthermore, mesh draw costs are only paid when they are drawn, so with some careful LOD you can get away with the occasional “huge mesh” – the GPU has the capacity if not everyone tries to push a million vertices at once. (Obviously a million vertices in an autogen house that is repeated 500 times is going to cause problems.)
But there is a cost here, and it is – the VRAM itself! A mesh costs 32 bytes per vertex (plus 4 bytes per index), so our mesh is going to eat at least 32 MB of VRAM. That’s not inconsequential; for a user with a 256 MB card we just used up 1/8th of all VRAM on a single mesh.
One note about LOD here: the vertex cost of drawing is a function of what is actually drawn, so if we have a million-vertex high LOD mesh and a thousand-vertex low LOD mesh, we only burn a (small) chunk of our vertex budget when the high LOD is drawn.
But the entire mesh must be in VRAM to draw either LOD! Only things drawn on screen have to be in VRAM, but textures and meshes go into VRAM as a whole, all LODs. So we only save our 32 MB of VRAM by not drawing the object at all (e.g. it being farther away than the farthest LOD).
Meshes in Main Memory
For anything that isn’t an object, the mesh lives in main system memory, and is transferred over the PCIe bus when it needs to be drawn. (This is sometimes called “AGP memory” because this could first be done when the AGP slot was invented.) Here we have a new limitation: we can run out of capacity to transfer data on the PCIe slot.
Let’s go back to our mesh: our million vertex mesh probably takes around 32 MB. It will have to be transferred over the bus each time we draw. At 60 fps that’s over 1.8 GB of data per second. A 16x PCIe 2.0 slot only has 8 GB/second of total bandwidth from the computer to the graphics card. So we just ate 25% of the bus with our one mesh! (In fact, the real situation is quite a bit worse; on my Mac Pro, even with simple performance test apps, I can’t push much more than 2.5 GB/second to the card, so we’ve really used 75% of our budget.)
On the bright side, storage in main memory is relatively plentiful, so if we don’t draw our mesh, there’s not a huge penalty. Careful LOD can keep the total number of vertices emitted low.
Take-Away Points
- Non-OBJ vertex count is significantly more expensive than OBJ vertex count.
- OBJ meshes take up VRAM; the high LOD takes up VRAM even when the low LOD is in use.
- To reduce the cost of OBJ meshes, limit the total LOD of the object.
No discussion of OSM would be complete without a discussion of licensing and copyright. OSM makes this particularly complex because the project is in the process of changing their license from CC-SA-BY to ODbL. For a full discussion, I recommend the OSM Wiki, but the short version is:
- CC-SA-BY (Creative Commons, Share-Alike, with Attribution) is a license that says that anyone can use a work and modify it, as long as attribution to the original author is maintained and the terms don’t change for the derived work. This isn’t particularly surprising, it’s a typical “open source” license, similar to the GPL.
- ODbL (Open Database License) is a slight twist: with the ODbL, anyone can use the original data, but if you modify the data, you have to give back your changes. You don’t have to make your derived works open source though. (So under the ODbL if you build custom scenery out of OSM, your scenery doesn’t have to be open-source licensed, but if you ‘fix’ any roads, you need to submit the fixes back to OSM.)
OSM is CC-SA-BY now and is working to switch to ODbL. Basically their lawyers realized that CC-SA-BY is great for images and text, but isn’t actually legal for databases (which is what OSM is). The ODbL will protect OSM as it is – that is, as a “database”. Since OSM is a huge open project, the license change is going to take a long time and lots of people will post lots of rants on lots of mailing lists in the process.
Here’s our plan: we are going to make the v10 global scenery abide by both the spirit and the legal requirements of both licenses. (At least, I hope we are going to try to do this. I am not a lawyer and wouldn’t mind if this was all a lot simpler.)
- The version 10 DSFs will be CC-SA-BY. This means that you can modify and redistribute the global scenery DSFs. In the past, we haven’t officially supported this, but we’ve allowed it to go on in the community. With version 10, modifying DSFs will be officially okay. (Please note that copying the sim DVDs will not be okay. You can modify our DSFs, but we are not inviting anyone to sell pirated DVDs! X-Plane is still a commercial product!)
- We are encouraging all users to improve OSM directly in the OSM database – we will not accept modified DSFs as a way to fix roads. (This helps us comply with the ODbL.)
- All of the scenery tools used to create the global scenery will be open source.
This last case is important because we do a lot of processing to the raw OSM data before we create the DSFs. Technically we are required by the ODbL to “give back” thsoe changes, but the truth is that I don’t think anyone really wants the hundreds of GB of temporary files we create as we process. So instead we will give back the tools that do that processing, so people can recreate our processed database as desired. From my discussions with OSM community members, this is apparently an acceptable way to ‘give back’ our changes.
I should say that nothing is guaranteed here. Heck, it’s even possible that OSM will change its license in a way that screws up the whole global scenery project before we ship. (This is highly unlikely – I’m just saying that there is legal uncertainty with OSM that we haven’t had to deal with when using other data sources.) But I think we’ll be okay; X-Plane’s use of OSM (to create a mash-up of OSM plus other data sources like SRTM elevation to create a derived copyrightable work like a DSF) is definitely one of the use cases that OSM wants to make possible.
