On my way back from South Carolina a few weeks ago I had some time in the airport to fix the WorldEditor 1.1 export bugs. All of the reports pointed to a single cluster of bugs that are hopefully now fixed in beta 2, and today I finally had time to get the betas posted.
I also cut a new build of MeshTool (2.0RC2) while I was doing builds; this fixes a bug when using orthophotos with varying physics.
So first: you can get the new WED 1.1 beta 2 and MeshTool 2.0 RC 2 on the tools page on the wiki.
The wiki? Yeah. Tyler has been helping me migrate the scenery tools to here. At this point I believe that all of the content on scenery.x-plane.com is now duplicated on the wiki (which also has additional articles).
WED Bugs
If you should find additional WED bugs, after you get over your initial surprise, please file bugs in the scenery tools bugbase. Please do not email me bug reports. WED has to take second priority to version 10 work, so I don’t have time to process bug reports now, and I will lose them. The bug base keeps your bug safe, keeps a record of what happened to it, and can take attachments.
Please do provide the minimal materials to reproduce the bug; simple packages with an earth.wed file are great. Thanks to all of the WED users who filed bugs with repro cases – this made it very easy to retest the export code.
My Polygon is Poly-Gone
It is illegal to have DSF polygons (or airport polygons) cross themselves; finally with beta 2, WED actually makes this case fairly easy to detect.
- If a polygon cannot be exported (because it is self-intersecting), an error message will indicate that some polygons were skipped, and those polygons are selected.
- If you pick “Error-Check Polygons” from the Edit menu, then for every polygon that has a self-intersection, an OBJ is added at the precise point of the self-intersection. Simply select and zoom in on those OBJs – they act as marker pins to show the problem. Delete the OBJs once you have untangled your polygon.
(I did experiment with error checking on the fly, e.g. simply showing red dots in the intersection points as you drag and resize the polygon, but the math is too slow. I am using CGAL to check bezier polygon intersections, and their algorithm is absolutely rock solid, but it can take up to a quarter of a second to recheck the polygon, which is too slow for interactive editing.)
UPDATE: beta 2 hangs when processing beziers. What is very odd is that this happens on the clean release code but not the working copy of WED I fixed the bugs in. Hence, when I checked all of the bug reports, they all passed. I have reopened all bezier-related bugs. I have not yet located the build environment differences causing the problem.
UPDATE 2: the hang on bezier processing was due to a bad compile configuration for a library WED uses, and was Mac specific. Having fixed this, I have recompiled and reposted WED beta 2 for Mac. If you already grabbed WED, re-download it, and make sure you get the version dated September 24th. You can tell you have the right one because the “compiled on” date in the about box will say “Sep 24 2010 19:34:09”.
There’s a real difference between a useless beta (e.g. what we have with WED now) and a useful one. While the purpose of a beta program is to fix bugs, a beta can be useful if the features users must have work. I don’t think I will have time to finalize WED 2.0 this year, but I do think I will be able to fix the really nasty bugs, the ones that make it useless.
To that end I have fixed a number of crash-on-export bugs. To the users who filed complete crash-on-export bug reports: thank you! It was quite an education to see what was in some of these projects polygon-structure-wise.
If you filed a WED export bug, I may have requested feedback if you didn’t include the WED file. Basically I would like the minimal materials to fully reproduce the bug by opening the package and picking “Export to DSF”. For many packs this is just the Earth.wed file, although sometimes textures or text files may be necessary.
A number of the crash bugs are caused by degenerate polygons. For example, if your polygon crosses itself in an X shape, not only is this illegal in X-Plane, but it also drives the exporter crazy. With the next beta, WED will select polygons that failed to export to help you locate such problems.
To put it mildly, I am buried. If you have emailed me in the last few weeks, I apologize, but basically whatever is going on, I can’t look at it for at least a few more weeks. The four posts for June is an indication of work-load. In particular, since a lot of what I am working on is v10, it is under the radar until we do some kind of more formal announcements.
I did find a little bit of time last night to fix a MeshTool bug (thanks to the users who found this – it was a tricky one!). There will be a MeshTool RC2 to fix the bug: if you use only “wet” orthophotos (that is, orthophotos that have water-like physics) but not “dry” ones, the orthophotos are not exported at all.
I realize that the entire schema for creating mixed wet/dry orthophotos in MeshTool is byzantine at best. Basically you have to manually build the set of GIS files to create this effect, and even with examples in the README it is still pretty hard to do. I hope to automate this a bit in a future version of MeshTool but for now I need to finish version 2.0 as is. I’ll try to cut a new RC within the week.
Also a slight side note: MeshTool contains some hidden commands to let you build road grids inside MeshTool. This was never documented or supported; the code came from a merge of Andrew McGreggor’s work on New Zealand. Starting with RC2, exporting roads will simply not do anything.
The problem is that I did not separate MeshTool from the rest of the scenery code, and the rest of the scenery code is transitioning toward version 10 roads (which do you no good now as v10 isn’t released). If you are successfully using the hidden road code in MeshTool, email me and I can advise you on how to cut your own build. If you are trying to use the hidden road code but not succeeding, please use another tool like Jonathan Harris’ XPOSM.
In the long term we will end up with “draped” roads in overlays – that is, roads that do not depend on the shape of the mesh. Thus you will be able to simply write road data into an overlay file (or someday draw it in WorldEditor). But we’re not there yet.
Tyler has made a lot of progress moving the scenery documentation to the wiki. Once I find some time to give him more feedback we will be able to complete this process. Hopefully this will make it easier to keep the documentation updated.
Austin attended the French X-Plane Congress last weekend; in response to some questions I have received, I want to clarify what is planned for DSF and X-Plane 10.
X-Plane 10 will extend DSF scenery capabilities by providing a number of new art asset types, as well as extensions to existing art asset file formats. We will not be changing the DSF file format or breaking compatibility with existing DSF scenery. If your DSF scenery works with version 9, I expect that it will work with version 10 as is.
(In fact, it looks like we do not even need to add new DSF extensions; DSF was designed to be a generic container for geometry data and properties, so we can usually extend X-Plane’s scenery system by simply defining new art asset classes and properties.)
An example will illustrate what I mean by extending the art assets, not DSF.
In X-Plane 9, you create a forest (whether in a base mesh or overlay DSF) by creating a DSF polygon referencing a .for (forest) art asset. X-Plane will render this by filling the area inside the polygon with lots of trees.
In X-Plane 10, you will be able to optionally tell X-Plane to put the trees only on the edge of the polygon, rather than filling the entire inside. (This feature will be controlled by using values on the polygon parameter that are larger than 255, at least I think.) This will mean that you can use .for files and DSF polygon not only to create forest areas, but also to create rows of trees along the edges of roads or fields. A row of trees made by a .for file and DSF polygon will render much faster than a large number of individual OBJ-based trees.
This kind of art asset extension is similar to what we have already seen; X-Plane 850 introduced three new art asset types (.str object strings, .lin line paint, and .pol draped polygons) that all produced new rendering tricks using DSF polygons. These art assets were added to provide high performance rendering of airport surface areas. The new art assets didn’t require any change to DSF because the representation of the position of these art assets is something DSF has always done: simple polygons.
DSF won’t last forever, but at least for X-Plane 10 it looks like we can do a third full cycle of the sim using DSF as our base container format for scenery position data.
In past posts I have tried to describe the implications of DSF base meshes, which are “fully baked”. The basic idea is: the base mesh is fully formed ahead of time as a single unit. This is a trade-off:
- The advantage is performance. The sim has no work to do except draw that base mesh as fast as possible.
- The disadvantage is flexibility. The sim has no easy way to modify that base mesh.
By comparison, DSF overlays are not fully baked – you can add 8 overlays to an area and they will all run on top of each other. There is a real performance cost to this. Compare the performance of a huge number of draped orthophotos (via .pol files, an overlay technique) with a real orthophoto base mesh cut with MeshTool. You’ll easily get 100 fps on the DSF base mesh, but you won’t come close with the overlay.
If you want to compare X-Plane to a first person shooter, consider the “cost” of overlays as one of the reasons why FPS games appear to be higher performance than general purpose flight simulators like X-Plane and MSFS. In a FPS, each level is likely to be fully baked, with only one level loaded at a time. This is equivalent to X-Plane’s DSF base mesh. The FPS game doesn’t need to manage overlays that are put together at runtime in unpredictable combinations, and this lets the FPS engine optimize for performance.
(In fact, the FPS engine might be able to optimize a second way, if third party level packs are not available. Not only can a level be ‘fully baked’, but it can be fully baked specifically for that particular rendering engine. By comparison, a DSF base mesh will run with X-Plane 8 or 9 – clearly it isn’t specifically optimized for just one version of X-Plane.)
If you look at the scenery system “overview” I wrote around the time of X-Plane 8’s release (this overview is now pretty out of date; I really need to update it) you’ll see this:
There are now two scenery formats – one for editing and one for distributing scenery. Both are new.
DSF stands for “distribution scenery file” – the idea is that DSF was meant to be a container for fully baked finished scenery, optimized for small size on DSF and fast loading, but not editing. Our internal tools use another file formatm “.xes” to contain imported global scenery data before it is baked. Originally I thought that we would provide an editor to .xes files, but that has not happened. With MeshTool, you provide input data in more common public formats like SRTM HGT or GeoTiff, and .shp (shapefiles). You can think of .shp and .tif as the editing formats for MeshTool and base DSFs.
So how do we make it easy for users to edit scenery? I believe OpenStreetMap is the answer. The common request we get from users is for a way to edit the vector source data for global scenery (or sometimes, the request is to edit the features created by vector data). In other words: how does a user edit the coastlines, water bodies, and roads? With OpenStreetMap, OSM itself becomes the “editing” format for X-Plane scenery with DSF as the final result of baking.
Last week I finally was able to post some scenery tools builds. This begins the beta period for WED 1.1 (WED with DSF overlay editing) and a release candidate period for the AC3D plugin and MeshTool. I expect the WED beta to take a while; as long as the program is reasonably functional in beta form (e.g. not losing data) there isn’t a huge rush to package it up relative to other priorities.
WED will continue to evolve as the primary visual editor for scenery. This will include editing air traffic control data for the new ATC system and editing roads once X-Plane is ready for road overlays. (See this post for why road overlays aren’t quite useful in the sim yet.)
I’m not sure what the next features for MeshTool will be – this may depend on user feedback. One thing is clear: MeshTool is not easy to use. Building a base mesh is a complex and low level process, with lots of possible pitfalls. So whatever features MeshTool develops, usability has to be a goal.
There is one more scenery tool I would like to create: a remote render farm. Right now you can make custom base meshes with MeshTool. In the future it will be possible to make edits to the source data used or global scenery (by editing OpenStreetMap itself). You can also edit the apt.dat file and submit the results to Robin.
But once you edit OSM, how do you get a new scenery pack? Using MeshTool is complex, and MeshTool is really aimed at the orthophoto crowd. Waiting 2 or 3 years for the next X-Plane release isn’t a good solution.
My idea is to set up a computer to do “remote global scenery” requests. I would set the computer up to periodically pull down new data updates and recut tiles as requested, then post them publicly. This would allow users to edit the source data and then put in a tile request to get the tiles back, without ever having to know how to make scenery. The tiles would reflect all changes from all users.
Such a service wouldn’t be of interest to the most advanced authors who want to create a truly original scenery pack, but for authors who want to fix specific problems, this process would be much simpler. I hear the question “how do I fix the lake behind my house” all the time; a remote render farm could be part of the answer.
A bug-fix release of AC3D was posted over the weekend, and now it is gone. Andy pointed out to me that I had posted a build for Windows, but not Mac and Linux, a build error on my part.
I should be able to get all of the new tools (the AC3D patch, WED 1.1 beta 1, and a MeshTool release candidate) posted this week.
It looks like the next generation of scenery tools (MeshTool 2.0, WED 1.1, and the latest distribution of “the tools”) may have higher system requirements than their predecessors for Mac users. Those requirements would be: an Intel CPU and OS X 10.5.0 or higher.
The problem at the heart of all of these changes is that the tools use CGAL (a geometry library) and the compilers Apple distributes that are compatible with 10.4 and PPC don’t work with CGAL. So I quite literally cannot compile the latest tools because of the features they offer.
I don’t know how much this affects actual authors, and I don’t know if it is possible (given an infinite amount of self-torture) to work around some of the compiler issues. At this point my plan is:
- Distribute the next-gen tools in binary form for 10.5 and x86.
- Leave links to the old tools for users who need binary PPC tools.
- Continue to make all source code available via the GIT repo.
If someone finds a way to compile these tools for older targets using the source code, I am perfectly happy to provide distribution of those binary tools or incorporate the fixes if they are manageable.
I’m hoping to have some tools posted “real soon”…
After about a week of on and off hacking, I have finally knocked down one of the major stumbling blocks to getting WED 1.1 up to beta quality: exporting UV-mapped (texture mapped) bezier polygons that cross DSF borders. It works! Well, sort of.
If you have tried to program polygon cutting algorithms, you can appreciate the difficulty of an algorithm that:
- Clips polygons robustly (including holes and other weird topology) and
- Maintains a UV mapping while doing this and
- Works with bezier curves and not just line segments.
WED now does all three! This was the ugliest and hardest part of the DSF exporter, and a big missing piece from going beta.
Of course, there is one problem: X-Plane can’t read the bezier curves.
The problem is a simple defect in how X-Plane manages DSFs.
- A valid bezier polygon, fully inside the DSF tile, may have control handles that go outside the tile.
- X-Plane can’t handle any DSF coordinates outside the tile.
Doh!
I am not sure what I will do about this, but in the short term, I fear X-Plane will remain limited. Probably the best short term option is to have WED at least flag such problematic bezier polygons; it is possible to approximate them or edit them to make the export work.
There is still a little bit more exporter code to write, including the line segment exporter (which is separate from the polygon exporter), but with luck the whole DSF export path should be cleaned up in the next few days.
Meshes in X-Plane, whether modeled in an OBJ, or generated as the results of other “3-d clutter” (road .net files, .for forest files, etc.) can be either one or two sided. So first: two-sided geometry is bad in most cases.
In order to understand why two sided geometry is bad, we must consider the alternative. The alternative to two-sided geometry is to simply create each triangle twice, with one facing in each direction. We can do this in an OBJ without making new vertices – because vertices are referenced by indices, we only need more indices, and indices are cheap.
Thus we have an alternative to two sided geometry, namely “doubled” one-sided geometry.
The first problem with two sided geometry is performance: for a small number of triangles, it is must faster to simply emit additional indices than to change the drawing mode to two-sided drawing on the CPU.
Thus in an object it is virtually never a good idea to use two-sided geometry. That ATTRibute will always be worse.
What about for the other clutter? Forests are currently always two-sided, but that’s okay; X-Plane enables two sided drawing just once, then draws a huge number of trees. Same with roads. For facades, there is a cost to using two sided geometry, so only use it for facades that must be two sided, like fences; do not use it for buildings.
Now the second problem with two sided geometry is lighting: X-Plane does not calculate lighting values separately for the two sides of the two-sided geometry. So if you have directionally lit models with two sided geometry, the lighting will look wrong. This is the second reason to use doubled geometry instead.
Things Are Starting To Look Up
There is a work-around to this problem of incorrect lighting on two-sided geometry: “up normals”. With up normals, the normal vector for the triangle (which is used to determine how light “bounces” off the triangle) is set to face straight up. The result is a triangle with brightest lighting at high noon, regardless of which way the triangle actually faces.
The good: the triangle looks the same on both sides and has sort of a “flat” lighting – it doesn’t look wrong when the sun is setting. The bad: the triangle has “flat” lighting – it looks non-3d.
We use up normals for forests because the forests are made of two-quad trees…the trees look less fake if directional lighting hints don’t make the two quads as obvious. You can simulate this in ac3d using the “make up normal” command for vegetation quads you put in your own models.
For roads, the geometry is two-sided, so we use up normals to avoid having the back of a road element look funny. Some day we may do something more sophisticated.
Fixing Facade Lighting
Facade lighting behavior will be changed in the next 950 release candidate. Before 950, facades would receive up normals, always. Starting in 950, facades will get correct normals if they are one sided and up normals if they are two-sided. This avoids artifacts with two-sided facades, but will make one-sided closed buildings look much better.
