How To Tell Whose Plugin Crashed?

In X-Plane 10, we started listing the plugin that crashed in Log.txt when a crash happens. This post is for plugin authors and provides one or two tricks to determine more information.

How Do We Know If a Plugin Crashed?

The mechanism for determining whether X-Plane or a plugin crashed is very simple: every time X-Plane dispatches to plugin code, we make a note that we are jumping into plugin land; when we jump back we clear the note.  This way when we get a crash, if the crash is in plugin-land, rather than collect an automatic crash report (which we won’t be able to interpret) we let the OS crash handling go through.

This mechanism is not perfect.  For example, when the sim reads an animation from a plugin, it does not mark the dataref read as plugin land, so a crashing dataref handler in a plugin will appear as an X-Plane crash.

How Do We Know Which Plugin Crashed?

The log from a plugin crash will say something like “X-Plane crashed because of a plugin: XSquawkBox.”  (This is of course hypothetical; XSquawkBox would never crash. 😉  How does X-Plane know who crashed?

The answer is: it asks the X-Plane Plugin Manager (the XPLM).  The XPLM tracks which plugin is executing at any given time; every time a plugin callback is dispatched, the XPLM notes which plugin owns the callback and marks that plugin as current.  (This is also how the XPLM knows who owns resources; the plugin that is current when a resource is allocated owns the resource).

Note that if your plugin calls into X-Plane and X-Plane crashes, you still get blamed.  For example, if you call us to draw an object (with XPLMDrawObject) but you pass us a bogus object handle, the sim will probably crash and you’ll take the wrap.  This is probably the most useful case from a debugging standpoint; if your plugin calls our API and it crashes, we’re not going to look at it without some info about what your plugin was doing.  You (the plugin author) get the crash report and can thus know in context how the call was happening.

(If these crashes were reported to Laminar Research, we’d have no idea what was happening in the plugin because we have no plugin symbols.)

Note that dataref reads are tracked by plugin, so if, in your flight loop callback, you call XPLMGetDataRef on a dataref provided by another plugin, and that plugin’s read handler crashes, they get logged, not you.

Who Is XPLANE_PLUGIN_XPLANE

X-Plane is actually registered as a plugin, because it provides datarefs.  Therefore if you can crash in callback code registered by X-Plane (that was called from a major callback from a plugin, like a flight loop callback), we’ll log that we crashed due to a plugin, and that X-Plane is the plugin!

An example to illustrate this: your plugin registers a flight loop callback.  From your flight loop callback you call XPLMGetDatavf to read an array dataref.  The array dataref is owned by X-Plane but you pass an illegal pointer to XPLMGetDatavf, such that saving the values crashes.

X-Plane (the app) will crash inside X-Plane’s dataref handler (because writing to illegal memory will cause a crash).  Because we’re in your flight loop callback, a plugin is going to get blamed, and because X-Plane’s “plugin” code (the dataref accessor functions ) were running, XPLANE_PLUGIN_XPLANE gets blamed.

There are only two bits of code that I can think of off hand where this can happen:

1. Reading X-Plane built-in datarefs.
2. Using the built-in 64-bit Lua allocator by sending plugin messages to X-Plane.

If you can consistently reproduce a crash blamed on XPLANE_PLUGIN_XPLANE, please let me know; it either means we have a bug in our datarefs, or your code is doing something illegal (that we should probably trap with error detection if it’s affordable). Note that we can’t catch the bogus-memory-for-array-dataref case without killing performance, so that’s going to remain a crash.

Crashing In a Worker Thread

Plugins do not have officially registered worker threads in X-Plane 10; therefore if a worker thread crashes, we don’t know if it is ours or yours, and we cannot issue approrprate blame.  Currently X-Plane assumes that all worker thread crashes are sim crashes and auto-reports them.

If you do have a plugin that uses a worker thread and you think it might be crashing, you’d have to run from the command line with –no_crash_reporter to capture native OS crash information.

We (Laminar Research) cannot contact you to notify you of worker thread crashes caught by our crash reporter.  Besides not having contact info for all plugins, the crash reports can’t trace the stack properly to determine who crashed, because we don’t have symbols.

Finding the Plugin At Fault in an Apple Crash Report

If our crash reporter catches a crash, we log the plugin at fault, and we make sure the complete Log.txt is written to disk.  Similarly, if you log something with XPMDebugString, the log is written to disk immediately.

However, sometimes a crash will happen an we won’t catch it.  Causes include a worker thread crashing, running on Yosemite (where, as of this writing, automatic crash detection is inoperative), etc.  If you are on OS X the user will get a crash report, and it can actually be pretty useful.

The crash report does have one weakness: all of the plugins are named mac.xpl.  How do you know who crashed?

It turns out each plugin’s binary gets a unique ID (UUID) which will be logged in the crash report along with its address range.  You can calculate your own plugin’s UUID with

dwarfdump --uuid mac.xpl

The output will look like this;

bash-3.2$ cd DataRefEditor/
bash-3.2$ dwarfdump --uuid mac.xpl
UUID: 0C9A8216-95CB-2F45-D8B6-51B6168AC05D (i386) mac.xpl
UUID: 5A0D5A18-992D-FD89-3CF1-6299CBC8F03C (x86_64) mac.xpl
UUID: 66A96F33-3FFC-A36B-660C-533F006B590F (ppc7400) mac.xpl

This is the UUID for DataRefEditor.  Note that DataRefEditor runs on three platforms (PowerPC, 32-bit Intel, and 64-bit Intel – you can actually run PluginAdmin in X-Plane 8, 9, and 10) and therefore it has 3 UUIDs, one for each architecture.

So if you see a mac.xpl loaded in an Apple Crash Report whose address range covers the crash and has your UUID, then you know you are the one who crashed.

(If I had known that so many crash report tools would delete the path to a crashing DLL, I would have considered a different naming convention for fat plugins.)

About Ben Supnik

Ben is a software engineer who works on X-Plane; he spends most of his days drinking coffee and swearing at the computer -- sometimes at the same time.

View all posts by Ben Supnik →