I’ve been fielding crash-related bug reports for X-Plane 10,30, and a lot of them are turning out to be caused by third party scenery packs.
Now there are two possibilities for why a third party scenery pack might crash X-Plane:
- There is a bug in X-Plane. In this case, I want to fix the bug!
- There is something wrong with the data in the scenery pack (that is, the data doesn’t meet X-Plane’s expectations). In this case, I want to change the result from a crash to some kind of clear error report that an author could use to fix the problem.
The Old Rules
During my work on the scenery system for X-Plane 8 and 9, my view was “garbage in, garbage out” – that is, if you feed junk data into the scenery system, the result would be unpredictable and could include a crash. In X-Plane 8, scenery load was completely synchronous – that is, your flight just stopped while we loaded files, so my idea at the time was that validation of scenery files should be done ahead of time, perhaps by the tools that write those files.
That logic (if it was even a good idea back then) doesn’t apply now. Loading is done entirely on other CPU cores and does not interrupt flight; we can take the CPU time to make sure incoming data is valid. And the cost of not validating the data is clear:
- The instability from badly formed data results in a sea of crash reports – if we have real serious sim bugs, it’s much harder to catch them in the noise.
- Because we don’t trap bad data, the end result crash can be in the OpenGL driver (due to cascading errors), and due to the way 64-bit crash detection works, we don’t get any trace of the contributing X-Plane code path.
- Authors who create scenery packs can’t tell if their pack is the source of crash, or how their data induced a crash.
My goal is to chase down crashes caused by third party scenery and both fix any bugs that emerge and tighten up error checking to detect the cause of crashes early on, in a way that is useful to authors.
The New Rules
Because X-Plane has not run validation over all scenery files in the past, there are probably scenery files in use now that contain data that I would consider illegal (either by the documented format, or just by my own view of how the file format was meant to work; I realize that second definition is quite useless to authors). But for illegal data that doesn’t crash the sim, I can’t just go in and change X-Plane to reject that data in the next patch. (“Surprise! Your scenery packs won’t load!” isn’t cool.)
So my thinking for validation is:
- Detected error conditions that would have caused a crash 100% of the time can be handled by any clear means possible, including quitting the sim with an error message. If the sim can handle the error and continue, that’s better, but even a controlled quit with diagnostics is an improvement over an uncontrolled crash.
- Detected error conditions that would have crashed the sim some of the time need to be logged but can’t quit the sim or cancel loading of the entire scenery pack; while a crash is serious, users were probably still enjoying the scenery pack (and merely being annoyed at how unstable their flying experience was). The user experience shouldn’t get worse.
- Detected error conditions that have only minor effects (e.g. out of range values that might cause rendering artifacts) whose legal values are well-documented can be logged, perhaps with a noisy error. The pack works, so quitting or refusing to load is not acceptable, but if the error is too quiet, the condition will never get fixed.
- Detected error conditions that have only minor effects whose legal values were never documented need to be logged unobtrusively; it’s not fair of me to penalize scenery packs for failing to read my mind.
On those first two points: one of the issues that complicates validation of scenery packs is that some errors are not discovered until mid-flight, because scenery is processed while you fly. This means that a large scenery pack might contain areas that are problematic to fly in; if the original author doesn’t cover every inch of scenery in a test flight, the author might not know about the problem.
The ATC system takes a more aggressive approach: it validates every single .atc file at load time even if it doesn’t need to use the data until later. (Virtual air traffic controllers are instantiated near the aircraft as the user flies, but the data in the .atc files that will be used to set up those controllers is checked at load time.)
This approach wasn’t too expensive performance-wise: the .atc files are small and the ATC system has to read them anyway to index them for later use. I am not sure how to best replicate this for scenery; consider Alpilotx’s HD scenery mesh – reading every single DSF in Europe at load time isn’t acceptable performance wise but without that, how can he know that his scenery doesn’t contain areas that could crash X-Plane?
Please Send Me Bogus Scenery Packs
Here’s where I need you to help: if you find X-Plane is crashing, and you find that removing a scenery pack fixes the crash, please report a bug, including the scenery pack (or where to get it) and where you had to fly/start up to induce the crash. Even if it is the fault of the pack (and it may not be – it may be an X-Plane bug!!!!) I still want to clamp down on the failure condition.
RT @XPlaneOfficial: Dev Blog Post: Please Help Find Crashing Scenery http://t.co/C1gyt5ai6X #xplane
Granted I know nothing about how sceneries are constructed. But I was just wondering, is it possible for the scenery developer to validate the scenery with an off-line validation program that pretend to fly over the scenery in a certain pattern?
Hi Ben, any chance for an x-plane-scenery-validator-tool?
Hi Ben,
couldn’t be possible to create a specific tool that would check the whole Custom Scenery folder, including links to scenery located onto external disks (or giving the user the way to choose the folder to check), automatically sending you all the data you do need in order to understand if and how a certain scenery could crash X-Plane?
Angelo
Hi Ben! Good work trying to find this out. Probably it’s quite a large effort, but what about an offline file validator? I mean, something that you can call on a scenery pack, and simulates an X-Plane loading for all of it, and logs the errors. This kind of tools I normally use on my projects, where loading every single combination as a user is just not possible during deployment.
Keep doing this amazing sim!!
Everyone who posted in the comments had the same idea – validate off line. It is possible! It has its pros and cons:
– Pros: the validator doesn’t have to be as performance sensitive – set it to validate, go to bed, pick up the results in the morning.
– Pros: thus the validator could run tests that are clearly prohibitive for X-plane (e.g. test every single DSF).
– Cons: because the validator is separate from X-Plane, there is a risk of the validator and the sim parsing code getting out of sync.
– Cons: because the validator is separate from X-Plane, invalid scenery will cause a crash that gets reported as a crash – we have to then go and find the user and ask them to run the validator.
“– Cons: because the validator is separate from X-Plane, there is a risk of the validator and the sim parsing code getting out of sync.”
Not sure about this: if the validator will use the same validation routine that, if I recall correctly, you’re planning to insert into WED, it ->should<- be always in sync.
– Cons: because the validator is separate from X-Plane, invalid scenery will cause a crash that gets reported as a crash – we have to then go and find the user and ask them to run the validator."
I believe that would be the same with the validation done with WED: what if a developer for any reason doesn't check his scenery with WED (when available) and his scenery cause a crash?
Just thinking at no longer, poorly maintained and/or little used scenery, for example.
I also think that every single custom scenery that you can find on the .org is used by lot of users, probably there're good chances that at least 1 user out of 100 will use the stand-alone validation tool on its own.
You could even put a new item on, just a thought, the Special menu (or another one) in order to call the validator.
An off-line validator would -not- necessarily be the same as the one in WED – remember, WED is not the only scenery tool, and scenery can get into X-plane without ever going into WED.
So: an offline validation tool has many of the same weaknesses as WED’s validator already has: no guarantee it is run on 1005 of scenery before 100% of launches, no connection to crash reports.
How about this :
X-Plane-x86_64 --validate_all_sceneryWith that option, X-Plane would not run the sim, but would load and validate all scenery packs using the same code as the regular X-Plane.
Marc.
Right – I did consider a mode that would do something otherwise too expensive, like load -every- installed DSF. We actually have this as a test mode – internal code can set X-Plane to simply start the plane over every 6 DSF tiles, to ensure each one is loaded once. It takes a long time to run. 😉
“An off-line validator would -not- necessarily be the same as the one in WED – remember, WED is not the only scenery tool, and scenery can get into X-plane without ever going into WED.”
Sorry but I’m missing your point: why the off-line validator could be different than the one in WED?
I mean, WED will check for specific issues, so I’d expect that the off-line validator would check at least exactly the same, probably even more, but sure not less.
In a more user-friendly format I’d second MarRog’s proposal, but I could even think at an option inside the installer
I think I’m missing your point. 🙁
Are you saying:
– WED’s validator should be -the- validator or
– WED and the offline validator should share the same code or
– WED’s validator should do anything an off-line validator would do or
something else?
Let me try to explain (sorry but english is not my native language).
You said:
“An off-line validator would -not- necessarily be the same as the one in WED”
I believe that the WED validation code will work using the WED logic, is it right?
Then you said:
“WED is not the only scenery tool, and scenery can get into X-plane without ever going into WED”
Every scenery built with a tool different than WED must follow the same rules, but as those other tools can (and usually do) use more libraries than WED and use their own logic in order to generate scenery files I think (of course I can be wrong) that the off-line validation tool should be able to manage more ‘variables’ than the WED validator itself.
So if I’m not wrong the off-line validator should do anything the WED validator would do, and more when checking scenery done using other tools.
Yes! The off-line validator is a super-set of the WED validator.
But for checks in both WED and the offline validator, the check might act differently: the WED validator works on the source earth.wed.xml document, not the finished DSF file.
Ok, understood, thank you!
Posting a ‘blacklist’ of those scenery packs crashing X-Plane (including the payware ones) would be appreciated, thanks. 😀
If we validate errors, there’s no need to black list – if you have a broken scenery pack installed, you’d know.
A blacklist could warn other users to download or buy the broken package.
Again, I don’t think this is necessary.
– If we had solid validation, someone -selling- a scenery pack wouldn’t sell it while it was broken. It would be so obvious there is a problem, they could fix the problem ahead of time.
– For freeware, there isn’t the need to pre-warn.
The issue we have here I think is not one of malice, but rather one of inadequate tools. In the cases I have seen with serious errors in payware scenery, the authors have had no way to easily identify what was wrong because the error reporting was either:
– Too obscure
– Only ran when the error was hit, requiring flying every inch of the pack or
– Not logged at all, causing an unidentifiable crash.
This is why I want the reports – to make x-plane report properly!
Well this sound like the same problem, like the PAHL ATC crashing I sent you months back. There was no way of knowing that X-plane would attempt to load the Seaports as the main ATC and crash ( i.e. the seaport was very close to the airport), and that still was only when the certain load order was in effect.
I think any validator would have a tough time anticipating all the variables that may be needed. We only found what was causing that crash after a long time of picking and poking.
by the way Ben, has that bug been fixed or still on the list?
This fix (seaport mis-handling in the ATC engine) is in progress – Philipp is working on it. It actually shows up in several points in the code. This is an example of a ‘bad’ scenery pack actually being an _X-Plane_ bug. It’s also an example of why detecting these things is hard – you have to get the sim into just the right state that the AI picks the sea port – and the aerodrome ATC picks is not necessarily the one your aircraft is at.