Diagnosing Crashes in X-Plane 10.30

Since X-Plane 10.30 shipped, we have received a lot of feedback about the stability of X-Plane 10.30 – or rather, the lack thereof.  We are getting more bug reports that the sim crashes, more tech support requests, and more discussion of crashing in the forums.

We don’t have a good way to characterize with hard data how much less stable 10.30 may be – unfortunately even the crash reports we gather don’t give us a statistically clear picture.*  But with this much user reporting, stability is our first line of inquiry. 10.31 will come out shortly (and will fix a few issues) and we’ll keep investigating problems and pushing small patches (to fix stability issues without introducing new ones) until we’re back on solid ground.  The rest of this post discusses some of what I’ve seen so far looking into stability issues.

How To Use the Automatic Crash Reporter

Since X-Plane 10.10, we’ve had the ability to send crash reports** directly to Laminar Research.  Here are a few suggestions:

1. If you need help, contact tech support!  This is the most important thing. We do not read every single crash report!  Instead we use statistical queries to locate and identify common crashes.  If you crash and write in the comment field “please help me”, your cry for help will probably be lost.
2. Please do report every time X-Plane crashes, even if the crash fields are blank.  The reporter can handle huge quantities of crashes, and we’re trying to gather a statistical picture of crashes.  If you get a crash every day, we want to know that it wasn’t a one-time thing.
3. If you hit a reproducible crash (e.g. every time you follow a simple set of steps the sim crashes with a crash-auto-report dialog box), please file a bug and include the crash report (from Output/crash_reports).  Reproducible does not mean “fly the sim and it crashes”.  It means “start the sim with no prefs, open the C172, flip the avionics switch 3 times and the sim crashes” – specific steps with a known crash outcome every time.

Reproduction steps are gold to us – sometimes we’ll see a crash and the proximate cause is “hrm – the sim crashed drawing an OBJ”.  Sadly this doesn’t tell us much about how to reproduce the crash – and clearly the sim does not crash every time you draw an OBJ; it wouldn’t start up.

The Problem with OpenGL Drivers and 64-Bit

Our crash reporter can only trace the origins of a crash through X-Plane code.*** Unfortunately, of the top 5 Windows 64-bit crashes in 10.30r2 (and this represents most of the crashing going on), 87% of the crashes originated in an OpenGL driver from AMD, NVidia or Intel.  (This does not mean the driver is bad – it could mean that X-Plane sent the driver junk data.  Or it could be the driver’s fault.  We don’t know.)

Unfortunately that means that for the vast majority of 64-bit crashes, we have no idea what the sim was doing when it crashed – the trace from the driver will contain junk.  All we know is that we were, in fact, in the driver at the time of death.  (We also have the log.txt but often it just indicates the user was flying.)

This is a long term problem that we need to address as developers with our crash report technology; we have some ideas on how to fix it but they will take time to implement. Clearly we’ve hit a point where the low hanging fruit (crashes that affect a lot of users but are entirely in X-Plane code) are gone.

Coping With Add-On Crashes

We do not have good statistics (yet) for what percentage of submitted crashes were on X-Plane installations heavily augmented with add-ons, but it is not uncommon to see a user with a custom aircraft running a custom plugin, an online flight plugin, tons of custom scenery packs, and general utility plugins.

This kind of customization is great!  We wouldn’t have added the SDKs if we didn’t want people to use them.  But when it comes to isolating a crash it does make things quite a bit more difficult.

My goal for X-Plane is to not have X-Plane crash due to bad third party add-on data, no matter how badly it is formed.  To that end, when I find cases where bad data will crash the sim, I am adding code to protect, document, and shut down based on the error.  (Some errors are recoverable, some are not, but the important thing is to give the author some feedback as to what is going on.)

In the long term I think we need to add better diagnostics to our crash reporting to show what add-ons are involved in a crash. If you have 1000 scenery packs, I can’t go download them all in an attempt to reproduce a crash.

In the short term, if you are seeing instability and you have third party add-ons installed:

• Please run without the third party add-ons – the goal here is not to have a more boring flight, and it is not a cure – it is a diagnostic tool.
• By running with different sets of add-ons you can isolate which add-on causes a problem with a given flight.
• If you find a problem with a plugin (whether it’s in an aircraft or the add-on itself is a plugin), you probably need to contact the plugin author – the plugin contains its own code that we don’t have source to.
• If you find a problem with data (e.g. a scenery pack or airplane without plugins), please file a bug, including where to get the scenery pack or aircraft, and how to reproduce the bug.

Reproduction steps are gold to us!  The combination of the add-on and simple steps to reproduce let us find and fix problems, even with third party add-ons.  If you have a flight that crashes, please try to simplify it.  Try going directly to the location that you were at where you crashed, starting with the engines on, etc.

Yosemite

While we’re talking about stability, I have to mention Yosemite: on OS X the vast majority of bug reports we’re getting that involve “weird stuff” and crashes involve Yosemite.  We’re still trying to sort out the issues, but Yosemite is definitely the most disruptive OS X major version release we’ve had in a long time, and because it’s free, a lot of people downloaded it.

• If you haven’t updated to Yosemite on you’re using X-Plane, consider waiting until things settle down.  We’ll work around problems, and Apple will issue patches.
• If you have an older machine, consider never updating.  Perhaps this is heresy but often the maximum OS version your machine can run is limited by hardware requirements, e.g. this is the OS that just about knocks your machine over.  How well is it going to run X-Plane?  How much of the hardware resources are left for us?
• When a new OS comes out, don’t just click “download” even if it’s free; wait a week or two and see how well it works for other people first.

A major OS upgrade is when the OS vendor can make major changes, drop old technology, etc.  On OS X it also means new OpenGL drivers (because the drivers are always packed with the OS), so it’s a potentially disruptive update.

Update: it turns out Yosemite also broke our automatic crash reporter, which would explain why none of the common Yosemite crashes show up in our top crash list. 🙁  Some users have sent me the Apple crash reports that appear (since our crash reporting doesn’t run) so that we can diagnose what’s going on.

Next Steps

10.31 should be out in a day or so (including Steam); as we continue to find and fix stability problems we’ll continue to issue small patches.  We’re looking to do a 10.35-type release with small features and new airports; the current plan is that Julian and Tyler will do a public beta of the new airports (as a single separate scenery pack) so that people can try the new airports and spot problems before we get into beta.

* Crash reports don’t tell us how many times the sim was run overall; users with stable systems don’t send us any data, so an increasing crash report incidence would be meaningless – we don’t have data about increasing or decreasing use of the sim.

** Here we’re talking about true program crashes – the uncontrolled exit of the sim for unknown reasons.  If you have a controlled failure, e.g. you load scenery and it’s missing objects and the sim refuses to run, you don’t get a chance to auto-report that.  We also don’t view this as a stability issue – X-Plane can be very unforgiving about broken third party content, and we can have a separate argument about whether this is a necessary evil or terrible user experience (or both!), but there’s no diagnostic information to gather.

Update 2: Blog comments are now working again.

*** The problem is that for 64-bit there is no standard for back-tracing a stack without the debug symbols to decode the stack linkage; there is no standard link register.  I’m guessing that the ABI designers thought they were helping the compiler guys optimize, but in practice it makes debugging other people’s DLLs miserable.  Microsoft publishes debug symbols for every DLL they make ,which is a big help, but the driver writers do not.

About Ben Supnik

Ben is a software engineer who works on X-Plane; he spends most of his days drinking coffee and swearing at the computer -- sometimes at the same time.

View all posts by Ben Supnik →