In my previous post I announced that we have the ability to recut DSF tiles in an update and that we have already started doing this. I pointed out that we’re not signing up to recut the whole planet because we don’t how we will manage distribution yet.
This sets us up to talk about OpenStreetMap (OSM). First the basics:
- The X-Plane 10 global scenery gets its vector data from OSM – that would be roads and water.
- X-Plane 10 does not get its raster data from OSM – we get land class and elevation data from a composite of sources , both integrated by Alpilotx using a mix of tools I wrote and some GDAL/GRASS voodoo. (I think he may have had to sacrifice a live goat. I try to not ask questions about what goes on with the raster data!)
- OSM is the only source of water body data in X-Plane 10.
The result is a lateral move for water body data quality and an ironic one: for the first time the US isn’t the area of the world with the best data sources. OSM coverage in Western Europe is often near complete. By comparison, the US is still quite sparse.* The result is that water bodies that were present in the US in version 9 are now missing in version 10.
(To make matters a bit worse, the OSM extract for the global scenery is from July. I had to take an extract and stop updating to stabilize the scenery development process. We’ll take new extracts in the future but what you see now is months older than the current OSM data, not weeks.)
Why didn’t we use multiple water data sources? There were two problems:
- We ran out of time. It’s really tricky to integrate multiple data sources that don’t line up spatially, and while we had some promising tech, it made the DSFs worse as often as it made them better.
- Because all other sources of water were not in OSM, we had the potential for road/water conflicts, which makes the integration very tricky. In the old days when we used terrible VMAP0 data for roads, we could just nuke the road if the water didn’t match. But OSM road data is quite accurate and we didn’t want to manually edit it.
So my thinking is that what we really need to do is make OSM as good as possible and then recut tiles. Improving OSM improves the global scenery in the long term, everyone can participate, it helps everyone using OSM, and once OSM has the correct water data, getting it into a DSF is just a matter of recutting the tile. If the water data is all in OSM, then we don’t have to worry about water/road conflicts. This is the true power of using OSM: for the first time we can all focus on the source data, because the source data is public and community owned.
I did some work a few months ago to prepare NHD (the US national hydrography dataset) for OSM use. I am hoping to find a little bit of time to continue with this effort – my goal was to make it easier for people to import high quality NHD water into OSM, but there were some unsolved workflow problems when I left off to focus 110% on getting v10 out the door.
What Other OSM Data?
One question that comes up over and over is: what other OSM data will you use? The short answer is that I don’t know, but I think individual building shapes is not on the short list. The problem is that I don’t think we can adopt our autogen at the quality level we like to such arbitrary shapes.
That doesn’t mean that other third party scenery can’t use the data now, or that we won’t use it someday. But it’s not next on my list.
One type of data that I do think we could make good use of is “point of interest” data – e.g. we there is a school tagged in the OSM data, we could drop in a school autogen art asset for a given location – this kind of “shaping” of the autogen could, I think, provide another level of richness and accuracy while leveraging the existing autogen and DSF generation technology.
OSM vs. Airport
One last note on OSM and recuts: in the past we have cut out roads near airports. Our older road data was often very inaccurate and would run right over a runway. The cutting process tried to minimize the insanity.
OSM data is often very good and there are many cases where the OSM data could be left alone for perfect perimeter roads. I do not yet know how we will modify the cutting code to ‘preserve’ such road data, but I would like to get this right in future renders.
One problem that makes this difficult: often the on-airport roadways are modeled in OSM, but without any tagging. The result (when road cutting is off) is a city street down the middle of the tarmac, often where the vehicle paths should be. So we need a way to know whether a given OSM road is okay or not.
* The US road grid was first seeded by integrating TIGER road data into OSM. This gives the illusion of total mapping when in fact some areas have never been hand edited. The TIGER water was not brought in during the initial import, hence areas with roads but no water.
With autogen buildings, is it currently possible for scenery developers to add arbitrary attributes to “placements” of objects to better inform object selection. Along with this, how scriptable are the autogen rulesets by scenery developers? To use your example re: building footprints, it’d be fantastic if a scenery creator with access to footprint GIS data could parametize attributes like ‘number of sides’ and extend autogen scripts to take these into account when selecting an art asset.
On a related note, what are the system performance trade-offs, if any, of using X9-style object placement versus autogen (assuming equivalent object geometry and texture detail)?
If you are referring to the process of _creating_ scenery that _uses_ autogen, then the system is highly flexible. We don’t have a good third party interface to it right now, we just have a big chunk of source code. But the system is based on having attributes (that sometimes come from OSM) and using matching rules to pick autogen elements.
There isn’t really a performance trade-off for X-Plane 10 vs 9 object placement; the v10 stuff is higher performance because the individual objects are carefully built to hit X-Plane 10’s OBJ fast paths, while the v9 objs definitely don’t. But in theory you could build a v9-style DSF with only “cleaned” OBJs and it would be very fast. We did _not_ clean the v9 obj library that ships with v10 for compatibility, so a v9 DSF in v10 is still unusably slow on “insane” OBJs. The OBJ budget is < 10k unoptimized objs and better than 50k (sometimes much better) optimized OBJs on a DX11 video card. The v10 autogen definitely consumes more resources - but that's by design. Each autogen block will put down the equivalent of several draped pols, several OBJs and a forest. The v9 autogen doesn't bother with the draping and has less vegetation - so it doesn't look as good but is leaner on resources, art asset for art asset.
Fix power line towers stepping on roads (and well actually on everything they shouldn’t step on)…
„X-Plane 10 is the only source of water body data in X-Plane 10.”
Loving it. 🙂
You can only explain recurrency by explaining recurrency.
Sorry. 🙂
Typo fixed!
LOL, no, the goats are all fine 🙂 (blood spilling is not my style). But its true, that the raster data cost me a few months of my life (and burned a “few” CPU hours).
Sure thing. These do happen. Just resulting sentence was funny and reminded me of recurrency. 🙂
Feel free to delete my comment as it has no relation to article at all.
Hi Ben,
first thing first: thanks for xp10!
Now to the OSM issue.
I totally agree with you about your short/long term strategy about OSM vector data usage. I wonder if it could be useful to come up with a proposal of a small global controlled vocabulary of OSM tags to be used during the DSFs cutting. I don’t really know if this would fit in the general OSM usage patterns (but in OSM we already have quite a large body of uncontrolled tags), but in principle this could give a lot of power to your importing algorithms and hence help in raising the bar on our “plausible” world. This could be also a call for action for xp10 users to update the OSM data you need most.
Comments?
Cheers,
__peppo
PS:
Sorry for asking, but I cannot resist: any idea on when the new scenery dev tools will be available on the wiki?
Well, we could definitely _publish_ the “dictionary” of tags we use – it might help people. I try to use the tags that are used by the main map layers (tiles @ home and mapnik) so that if it looks good on OSM it looks good for us.
I think I once did blog that fixing one/way and bridge in the US was useful – since us roads come from TIGER, there are unmodified parts of the us where this important info is missing. We really need one-way and bridge/layer to be right to get the road grid to export properly.
Tools: no – sorry, I don’t know what the time frame is. Moving forward with tools/docs will get into the “doing now” list basically as soon as a little bit more of the crash/card compatibility bugs are put to bed. I am hoping to make a little bit of steady tools progress while working on sim perf. A lot of the hard work is already done but it getting everything into a usable documented package won’t be trivial.
Re tools
I understand. Can you publish just the new revisions of the main file specs? Personally I would love to start working soon with the new facades.
__peppo
Hi Ben,
Will there be any way to add water with an editor? Now I have streets and houses in XP10 where I live (rural area in Germany), but the river which flows through the valley is missing (yes, it is truly a river, not just a creek, and it is shown in OSM). This is the last thing needed to make VFR over my village and valley perfect.
Cheers,
Marco
Well, the expected way to deal with this would be to make sure the OSM data is good and recut the tile. Since the OSM extract is from July, it could be the problem is already fixed.
One of the problems we have with the OSM data is that X-Plane doesn’t cope well with vector river data when the river is small…if the river that is missing is very small (or funky in OSM) our tools might have deleted it to avoid bizarre data problems.
We do not have a solution to edit existing finished DSFs…that was never a scenery tool we wanted to build, and with OSM we’d rather people edit the map itself.
Distribution and a central source is a interesting situation, the problem with the .org is that it is very hard to find anything, yes i can google it or use the find, but this is a different matter in the fact there will be a huge amount of OSM data (default and created) that we will all need to keep up to date with what is being added, Robin Peel’s system is very good but this is on a larger scale, you have no idea on how hard it was just to download the Demo, 3.3gb is not a lot to go through the pipe, but the reality is that it was a major hassle…if an efficient distribution system could be created then it creates an interesting situation,
if we already have the data, then a new X-Plane (11) version including DSF/OSM data would not be needed to be distributed on DVD disks, reducing big problems in releasing an major update, and cost.
Thanks Ben, one probably silly question, instead of you cutting the tiles and making them available, wouldn’t it be possible to provide us the tool for doing so, so every user can update the tiles based on new osm data?
Thanks!
Maybe…but…
– the DSF generator tool requires something like 80 GB of data, some of which are intermediate processes created by alpilotx, some of which are made by processing the planet in a heavily customized manner. It’s all open source and available, but it’s not easy or automatic.
– the tool is lin/mac only – no Windows port.
I was thinking it might be more useful to set up a box as sort of a ’tile server’ or something…
How about re-cutting twice a year, and making it all available for download / upgrade for a fee that reflects your bandwidth costs??? If you wanted to get fancy, you could allow users to select the region they want to update. — Also, is there a way to edit land-class data directly. All the villages in the Kootenays, BC are missing. Just nice, perfectly placed roads, with no houses. ) :
Hi,
it seems, that the land use tags are heavily used outside of big cities – at least in europe. This might be a very good indicator for autogen where to place what kind of objects.
http://wiki.openstreetmap.org/wiki/Key:landuse
Regards,
Heiko
I’ll let Andras take a whack at this one. 🙂
Hi Guys,
Well, its definitely an interesting thing to bring in more information from OSM. And yes, I am aware since a long time about the landuse properties in OSM.
But my problem is still, that OSM is an extremely inconsistent source. Some regions have a super high detail mapping of landuse, and somtimes jsut a few km away from it, you get almost nothing.
As an example, lets look to an area in Austria: http://www.openstreetmap.org/?lat=47.2764&lon=12.832&zoom=12&layers=M
What do you see? Some areas have nice forest landuse, others have nothing.
So, this is my problem. I can’t rely on OSM as a single source. Whereas now look at the data source, where I get my landuse from (at least for Europe, for other regions I have other sources):
http://www.eea.europa.eu/themes/landuse/interactive/clc-viewer
(sorry, I didn’t find a way to zoom in – but let it search for “Zell am See” and it should bring you approximately to the same area as my OSM link above).
So, you see, that the sources I use are maybe not super detailed everywhere, but give a quite good detail with a consistent coverage over a very large area.
Now, of course, we could think about mixing source. But believe, that is something you try as a last resort (or in cases where it brings a very high pay off!). Mixing is a really error prone (and often even technically non trivial) task, which – if done carelessly – can introduce more problems than it brings benefits.
But, at least for cities, it might be an interesting additional source. There it might be of some real advantage to be able – at least where available (where not, we stay with what we have as information) – to better zone the city!
Ben, if one was to update/add a certain piece of water data in OSM, would it then be taken into account on the next/future update of X-Plane?
Cheers
Dominic
It would be taken into account in future renders of the scenery.
I haven’t figured out what you’re doing here yet. I hope after the holidays to delve in a little. Are water towers in your data? That would add a lot of realism as far as usable landmarks.
Howdy,
One thing that probably has been thought of before is the idea of being able to create custom roads in WED. I have been using strings to do roads, dirt paths and the like but they never have lights nor will there be traffic on them. If there could be a new type of art element for WEB in the form of a road it could solve the issues from roads on airport property. These roads would be to override existing road data or exclusion zones.
Right – the element exists in DSFs. WED 1.2 has some partially implemented road editing functionality – I may be able to get it to be usable by the beta.