WorldEditor 1.4 is almost ready for beta, and among its new features (there are several big ones) one is very important to the X-Plane Airport Gateway:
- WED 1.4 can browse airports on the gateway and download scenery packs directly into the WED workspace.
“Direct download” is really important for a few reasons, some of which might not be obvious:
- It saves time. Getting a gateway airport, even if you just want to look at it, is much faster when you don’t have to download the scenery pack, unzip it, install it, then import it into WED. Once you use direct download, you’ll wonder how you lived without it.
- It prevents mistakes. We have seen airport submissions where a user clearly downloaded a scenery pack, imported only the apt.dat (but not the DSF files) and then uploaded it. We cannot take scenery packs like that, because they fundamentally remove the 3-d data from the airport. (A fundamental policy of the gateway is that if you upload a scenery pack and one already exists, yours can’t be worse than the existing one in some way. We have to always move forward.)
- It provides ancestry information. When you download and then upload directly from the Gateway in WED 1.4, WED provides the scenery pack ID of the original pack as the “parent” of the new pack. This means that when Julian goes to look at an upload, he can look at the “original” and more rapidly spot problems. If your pack is the same as the original except for taxi signs, he only needs to inspect taxi signs.
- It prevents data loss. DSF is a slightly lossy data format – that is, if you get your data back out of a DSF file, it won’t be exactly the same as what you put in. (It is like a JPEG image in that regard.)
More on that last point: DSF stands for Distribution Scenery Format – it was meant as a way to make final scenery packs; it was not meant as an interchange format for continuous editing. So users are constantly importing and exporting DSFs to do work, small rounding errors (“bit rot”) will creep into our 3-d, and features that were perfectly aligned might not be well aligned after 4 or 5 edits.
The internal format for scenery on the gateway is not binary DSF files, so doing round trips to the gateway has much less “bit rot” than importing scenery packs.
Finally, DSFs are tiles; if your airport spans a DSF boundary, all DSF features (polygons, fences, etc.) get split along the DSF boundaries.
So if you import a scenery pack that you downloaded, you’re getting the split version, which is harder to edit and work with.
The internal format of scenery packs on the gateway is not split, and thus you can edit the original in almost the same form as it was uploaded.
WorldEditor 1.4 beta should becoming in “weeks” (and hopefully not that many weeks); I’ll post here when we are ready for beta testers.
X-Plane 10.32r1 is now available to beta test – to get it, run the updater and click “get new betas”. Release notes here.
10.32r1 is another small patch aimed at fixing critical bugs. Hopefully soon (E.g. in a week or two) we’ll start a beta of X-Plane 10.35, which will contain new airports from the X-Plane Airport Gateway, as well as smaller feature enhancements that people have asked for.
Right now it looks like fixes for Yosemite will go into X-Plane 10.35 and improved DSF loading and longer DSF visibility will go into X-Plane 10.40 if the code is solid enough.
Hi Guys, it’s Chris. I haven’t written a blog post in ages. They’ve kept me locked in the basement like Milton Waddams, unwilling to let me out to see daylight until I finished X-Plane 10 Mobile. And they stole my #$%#^ stapler!
We recently released X-Plane 10 Mobile for iPhone/iPad and while the Apple users were ecstatic, some Android users were puzzled while others were frustrated.
“Will there be an android version?”
“When’s it coming to Android?”
“Where’s the Android version? 60% of all smartphones run on Android but I guess that Apple-Fan-Boys are more important in your company”
“Why do you guys constantly focus on iPhone first when most users are on Android?”
Before I get into the real point of the blog post, allow me to answer some of those questions. YES we are planning on shipping X-Plane 10 Mobile for Android. YES we have already begun development. We do not have a release date. We do not have any hints. The only thing that I can say, is that we want it out just as soon as you do. NO we do not view Android as a lesser/inferior platform…We value Android customers just as much as we value our iOS customers. A customer is a customer. I think we’ve demonstrated by supporting Windows, Mac and Linux all these years that we’re not trying to play favorites. We want everyone to be able to enjoy our products. BUT, that doesn’t mean that the costs of development and the speed and efficiency of development is equal on all platforms.
Historically, we’ve always developed for iOS first and then Android second. I’d like to be open an honest about our reasons and hope that even if you disagree with them, you’ll at least understand why we have historically developed for Apple first. I will warn you, everything I have to say is completely my opinion, my impression, my feeling based on my experiences. I’m going to sound a lot like an Apple “fanboy”. I will admit, I do have a high level of respect for Apple’s commitment to polish and detail, but I also own a dozen android devices and respect them for their cutting edge features, their openness and their friendliness to customization.
At the end of the day however, I’m paid to be efficient and thorough and my thoughts below explain why that means Apple has historically come first.
I will also warn you…I don’t want this blog post to turn into a flame war between Apple and Android users. We’re talking about phones here people, not religion. At the end of the day, they’re just small piles of plastic and silicon that let us surf the web, make phone calls and play games.
We Can’t Develop Apple and Android In Parallel
Sure, we do this on desktop by releasing Windows, Mac and Linux versions in unison 100% of the time. Developing for desktop is pretty different than developing for mobile. We use very few 3rd party frameworks on desktop and it’s an open environment. On a mobile phone, it’s a very closed environment. What this means is that developing Apple and Android in parallel requires a lot more effort than developing for Windows and Mac in parallel.
Can it be done? Absolutely! Plenty of companies are doing it. But they also have large teams with large expenses. We’re still a pretty small group of individuals and we like it that way. The tradeoff however is that we can only focus on one platform at a time.
One alternative that we could consider is delaying shipment of an Apple product until the Android version is done as well. That’s a loss for everyone. Apple customers lose out on having the latest software and Android customers may lose because…we don’t have the revenue coming in to support the Android development costs. That’s right…Apple sales get reinvested into the company to fund Android development!
As Ben mentioned earlier…Apple and Android mobile sales fund desktop development…and desktop development funds mobile development! This is a very important fact to remember. I’ll admit, we laugh and roll our eyes when desktop users complain about the company working on mobile products, and mobile users complain about the company working on desktop products….and android users complaining about us working on apple products and vice versa.
The company has found equilibrium creating both desktop and mobile products. There’s adequate revenue to fund adequate staffing to continue to develop both.
We Develop On Mac Hardware
This is no secret. It’s been this way since the company started. We just find Apple products allow us to be more productive and don’t get in our way.
Historically, Apple’s Mobile Platform Has Been More Mature
Apple had both a technological advantage as well as a time advantage over Android when they began.
Apple already had an Operating System, supporting frameworks and a development environment to leverage. Making mobile versions of those things required them to port existing, time-tested code to a new platform. From a stability standpoint, Apple had the advantage in that they already had the code, the engineers and the process in place to do this.
On the other hand, Google had to start from scratch. They had to put together a new team to create a new operating system to run new frameworks…and they had to create a set of tools for developers to use.
In addition to all of the technological advantages Apple had, they also had a head-start of well over a year. We were already selling X-Plane V9 for mobile before Android was even announced publicly.
That meant we were already established and familiar with the iOS platform as developers.
When I began the Android port for X-Plane V9, I had to pretty quickly put it down…and wait. Android at the time only supported Java apps. X-Plane is NOT a Java app. 99% of it is written in C/C++ and Android had absolutely no support at the time…and so we waited….and waited….and waited.
Finally, many months later, Android added their NDK which allowed us to have C/C++ support. But it was completely minimal. None of the standard libraries that we were used to using were available. This meant a lot of effort on our part to get anything done. If you’re not a developer, a reasonable metaphor might be a carpenter that’s trying to build a house, but he first has to build his own hammer, nails, square and saw because the tools he’s used to using don’t exist on this job.
Finally it came time to release V9 for Android. For iPhone/iPad, we uploaded our 400+mb app to their store and we were done. On Android however, the store had a limit of 25MB. So that meant we had to buy servers and write code to download the resources from a farm of servers. Again, this added more time and more complexity.
Apple Has Fewer Devices
For this latest release of X-Plane Mobile, we support iPhone 4S/5/5S/6/6+ as well as iPad 2/3/4/Air/Air2/Mini/Mini2 and iPod Touch 5. That’s 13 devices to my recollection. But it’s even simpler than that…because they all have the same GPU manufacturer, they all support the same PVR texture compression, and they all pretty much just work interchangeably from a development standpoint. The only major differences between them are the processor speeds and the screen resolutions. We can literally test on every single device and be sure that the app runs the way we expect it to.
As of the time of this writing, our X-Plane V9 is running on 7,072 devices. You read that right….SEVEN…..THOUSAND…..DIFFERENT……DEVICES. Each device has a different combination of CPU, GPU, screen size, screen density and drivers. We cannot possibly test them all. Admittedly, many of them “just work” and there are of course only a handful of CPU and GPU manufacturers to worry about…but at the very least, it means at least three different texture compression formats. PVR is proprietary and unless the mobile device has a PowerVR chipset, they’re not going to get PVR. So we have to support various formats. That requires three different versions of our app to be created and tested and distributed. That requires three different resource packages to be created and tested.
There’s just no way to have the same level of stability as we can have with the iPhone/iPad platform.
Apple Has Higher OS Upgrade Adoption
Without carriers and other manufacturers getting in the way, Apple can release a new OS with features and bug fixes, and we can be sure that they exist on the majority of the devices that we care about in no time. This means that if there’s a driver issue that needs fixing, it will make it out to the masses and eventually the problem is gone.
Android’s fragmentation has really hurt them in this area. We encountered several devices over the years that violated some OpenGL spec. We worked with the manufacturer to isolate the issue. They release a patch to fix the issue…and most users never had a way to get the patch because their phone carrier dropped support for that phone model.
Now the user’s stuck with an App that they paid for that doesn’t work and there’s nothing that we can do about it.
We like Apple’s Developer Tools Better
As I mentioned earlier, Apple’s developer IDE has been around for ages. We have access to various performance analyzers and can now even analyze an entire OpenGL frame, one draw call at a time. This means we can really tune the crap out of the app before we make it public. In addition, all of the tools come in a single package that just works out of the box. Apple has also always had a simulator that’s hardware accelerated. This means for a lot of things, i don’t need a device plugged into the computer to debug something.
Android’s solution was for less “out of the box” in that they were using various open-source pieces that all had to be installed and fit together just right. Android had an emulator that was not hardware accelerated. It took longer just to boot than it took me to find a phone in my house, get it, plug it in and push an app to it.
Honestly, I think both sets of IDEs are sorely lagging behind features that Microsoft’s Visual Studio has had since 2000, but I digress.
TL;DR
We develop for Apple first because it’s easier and faster for us. It allows us to get the product out the door, running as efficiently and as reliably as possible. When we port the app for Android development, we can be sure that most bugs that come up are specific to Android and are therefore much easier to resolve in a timely fashion.
We are not playing favorites. We have no personal issues with Android and have no personal ties to Apple. The day that Android becomes the faster and easier platform to develop for, it will be the one that we develop for first. It’s just a business decision!
In the meantime, Android users should remember that the way things are currently being done means that they sometimes have to wait longer for new updates, but the updates that they receive will likely be more stable as they’ve been tested harder.
I will also note that we are closing the time gap between iPhone and Android releases. In the past, we were over a year behind on the Android release…because Android didn’t exist. 🙂 Now that it’s becoming more established, the gap should be shrinking more and more.
I want to link to two scenery add-ons that are now available for X-Plane 10:
Alpilotx’s HD Scenery Mesh version 3 is out now. There’s a bunch of good stuff going on here:
- The mesh quality is cranked way up. If your machine can handle this, it makes the DSFs look a lot better.
- This is a recut from the latest OSM data, so the scenery tends to be more accurate. If there’s a tile that’s funky in the default sim, an HD tile replacement can be a nice fix.
- Alpilotx includes details that don’t go into the global scenery for space reasons – better water definitions, more exact forests, etc.
(The blue-ish atmospheric coloring is a third party add-on that Alpilotx installed that cranks up the atmospheric scattering strength.Edit: Alpilotx tweaked the atmospheric scattering himself using a Lua script – see here for more info. But there are third party add-ons that do this if you don’t want to roll your own.)
XFlyer’s Winter Package 1.1 – I’ve been meaning to post this, and it’s a great add-on to combine with the HD meshes.
This add-on replaces the terrain textures with winterized versions. The picture links to some forum posts showing the pack in a number of conditions.
Unfortunately installation of seasonal add-ons is still trickier than add-on meshes, something I hope to fix relatively soon. You’ll find installation instructions on the .org link.
Posted in News
by
Ben Supnik |
We finally shipped X-Plane 10 Mobile! Here’s the official press release:
Laminar Research, creators of the X-Plane flight simulator franchise, has announced the release of X-Plane 10 Mobile, the premiere flight simulation product for the iPad and iPhone.X-Plane 10 Mobile was built using the same concepts in Laminar’s showcase desktop product, X- Plane 10. As with the desktop version of X-Plane, the mobile version utilizes real physics to model the aerodynamic forces found in flying an actual aircraft. The result is a product that is not just another game, but rather a highly sophisticated simulation of how aircraft actually behave in a variety of environments and weather conditions.Version 10 contains numerous features and enhancements not found in the previous mobile version of X-Plane, including:
- 2-person multiplayer over the Internet
- Many additional aircraft – – from a Piper Cub to the Airbus, and it even includes combat aircraft and a helicopter… all with 3D cockpits and available as in-app purchases
- Challenges – – over 20 scenarios including engine failures, mountain top landings, birdstrikes, and even airstrikes, where you dodge incoming anti-aircraft guns missiles
- Flight School tutorials – – including the basics of takeoffs & landings, flying traffic patterns, and how to bomb ground targets
- Brand new, modern user interface – – designed just for mobile devices
- Improved combat system- – – with better missile guidance, more accurate weapon simulation and more realistic explosions
- Realistic airport environments – – including airport buildings
X-Plane 10 Mobile is available as a free App on the Apple iTunes Store and includes a free Cessna 172 starter aircraft. Additional aircraft may be purchased using the in-App purchase feature. An Android version is also scheduled for future development.
We also managed to DDOS ourselves by creating so much traffic back to the company website with the initial announcement, but the site’s back up and stable now. (The developer blog is on the same server as the main website, so if the server’s getting killed again, you’re not reading this. 🙂
Posted in News
by
Ben Supnik |
TL;DR version: if your plugin runs on OS X, you you should be setting the capability “XPLM_USE_NATIVE_PATHS” to 1, like this:
XPLMEnableFeature("XPLM_USE_NATIVE_PATHS",1);
This sets your plugin to use Posix-style file paths on OS X.
I am going through X-Plane looking for APIs that Apple has deprecated and replacing them. Aliases to custom scenery on other hard drives stopped working in Yosemite because we were using the Alias Manager to resolve the aliases, and the API is deprecated; in Yosemite Apple actually made it stop working. So now I’m looking to see what other deprecation problems we might be sitting on.*
One thing I noticed in my search is that kCFURLHFSPathStyle is marked deprecated in OS X 10.9. I don’t know when it will actually stop working, but we’re not supposed to be using it.
And here’s the thing: the only use case we have for it is incredibly silly: if your plugin doesn’t tell us that you can support Posix paths, we’ll convert to HFS paths so that you can then (in your plugin) convert back to Posix paths. In this use case, both the XPLM and your plugin are using a deprecated API to temporarily convert a file path to a silly format, and then back again.
Why Are Unix Paths Opt-In
The original XPLM API dates back to X-Plane 6 and ran on the classic Mac OS under the HFS file system. In this environment, all file system paths were HFS paths, e.g. Volume:directory:then:filename.
For a while, X-Plane could run under OS 9 and OS X using the Carbon APIs and CFM file formats; in this environment, the SDK continued to provide HFS file paths to all plugins at all times.
When we introduced the ability for plugins to use the underlying Posix file paths (which makes life much easier for the plugin developer, since Posix paths are what the OS really wants) we had to make it opt-in; a plugin tells us it wants this new thing by setting a new feature. Plugins that don’t opt in are assumed to be old and are assumed to expect the old convention.
HFS Paths Are Now Obsolete
Here’s the thing: at this point Apple has changed ABIs twice and changed CPU architectures; they have also changed executable formats. Simply put, no plugin code that runs in X-Plane 9 or 10 can possibly be using HFS file paths directly, because all running plugins are only capable of running on OS X.
But because it was possible to write a plugin that worked both ways (by opting in only when X-Plane was ready) there are still plugins that will run in HFS mode if and only if X-Plane can’t support Posix (e.g. if they are running on X-Plane 9.)
So in order to fully dispose of HFS paths, we need your plugin to start opting in to Posix paths. Doing so is really easy – it generally involves adding the one line above and deleting your HFS code (which is fun).
When Can You Use Posix Paths
Posix paths in the XPLM are available to plugins starting with X-Plane 10.0. If your plugin requires X-Plane 10 or the XPLM 2.1, posix paths are always available.
The XPLMEnableFeature APIs are available to plugins with the XPLM API 2.0, starting with X-Plane 9.0. So if your plugin only runs on X-Plane 9 and 10, you can attempt to set this (because the API is always available).
* Because our minimum OS is OS X 10.6 for X-Plane 10, X-Code normally doesn’t tell us about most deprecations. To find all of the issues, I temporarily set our minimum to OS X 10.10 just to see the warning list.
When it turns out that a bug that we thought was in an OpenGL driver is actually in X-Plane, I try to make a point of blogging it publicly; it’s really, really easy for app developers to blame bugs and weird behavior on the driver writers, who in turn aren’t in a position to respond. The driver writers bust their nuts to develop drivers quickly for the latest hardware that are simultaneously really fast and don’t crash. That is not an easy task, and it’s not fair for us app developers to blame them for our own bugs.
So with that in mind, what did I screw up this time? This is a bug in the framerate test that caused me to mis-diagnose performance of hardware instancing on NVidia drivers on OS X. Thanks to Rob-ART Morgan of Barefeats for catching this – Rob-ART uses the X-Plane framerate test as one of his standard tests of new Macs.
Here’s the TL;DR version: hardware instancing is actually a win on modern NVidia cards on OS X (GeForce 4nn and newer); I will update X-Plane to use hardware instancing on this hardware in our next patch. What follows are the gory (and perhaps tediously boring) details.
What Is Hardware Instancing
Hardware instancing is the ability to tell the graphics card to draw a lot of copies of one object with a single instruction. (We are asking the GPU to draw many “instances” of one object.) Hardware instancing lets X-Plane draw more objects with lower CPU use. X-Plane’s rendering engine will use hardware instancing for simple scenery objects* when available; this is what makes possible the huge amounts of buildings, houses, street signs, and other 3-d detail in X-Plane 10. X-Plane has supported hardware instancing since version 10.0.
The Bug
The bug is pretty subtle: when we run the framerate test, we do not set the world level of detail explicitly; instead it gets set by X-Plane’s code to set up default rendering settings for a new machine. This “default code” looks at the machine’s hardware capabilities to pick settings.
The problem is: when you disable hardware instancing (via the command line, explicit code in X-Plane, or by using really old hardware) X-Plane puts your hardware into a lower “bucket” and picks lower world level of detail settings.
Thus when you disable hardware instancing, the framerate test is running on lower settings, and produces higher framerate numbers! This makes it look like turning off instancing is actually an improvement in fps, when actually it’s just doing better at an easier test. On my RetinaBook Pro (650M GPU) I get just over 20 fps with instancing disabled vs 17.5 fps with instancing enabled. But the 20 fps is due to the lower world LOD setting that X-Plane picks. If I correctly set the world LOD to “very high” and disable instancing, I get 16.75 fps. Instancing was actually a win after all.
Was Instancing Always a Win?
No. The origin of this mess was the GeForce 8800, where instancing was being emulated by Apple’s OpenGL layer. If instancing is going to be software emulated, we might as well not use it; our own work-around when instancing is not available is as fast as Apple’s emulation and has the option to cull off-screen objects, making it even faster. So I wrote some code to detect a GeForce 8800-type GPU and ignore Apple’s OpenGL instancing emulation. Hence the message “Disabling instancing for DX10 NV hw – it is software emulated.”
I believe the limitations of the 8800 are shared with the subsequent 9nnn cards, the 1nn, 2nn and 3nn, ending in the 330M on OS X.
The Fermi cards and later (4nn and later) are fundamentally different and can hardware instance at full power. At the time they first became available (as after-market cards for the Mac Pro) it appeared that there was a significant penalty for instancing with the Fermi cards as well. Part of this was no doubt due to the framerate test bug, but part may also have been a real driver issue. I went back and tried to re-analyze this case (and I revisited my original bug report to Apple), but X-Plane itself has also changed quite a bit since then, so it’s hard to tell how much of what I saw was a real driver problem and how much was the fps test.
Since the 480 first became available on the Mac, NVidia has made significant improvements to their OS X drivers; one thing is clear: as of OS X 10.9.5 instancing is a win and any penalty is an artifact of the fps test.
What About Yosemite?
I don’t know what the effect of instancing is on Yosemite; I wanted to re-examine this bug before I updated my laptop to Yosemite. That will be my next step and will give me a chance to look at a lot of the weird Yosemite behavior users are reporting.
What Do I Need To Do?
You do not need to make any changes on your own machine. If you have an NVidia Mac, you’ll get a small (e.g. < 5%) improvement in fps in the next minor patch when we re-enable instancing.
* In order to draw an object with hardware instancing, it needs to avoid a bunch of object features: no animation, no attributes, etc. Basically the object has to be simple enough to send to the GPU in a single instruction. Our artists specifically worked to make sure that most of the autogen objects were instancing-friendly.
When I was very young, it was hard to watch my younger brother get presents on his birthday. I was jealous! Why should he get all of the attention? I was here first!
When I was just a little bit older, I realized that my brother’s birthday was actually a pretty good day for me too. You see, my brother and I had one big pile of toys, so whatever my brother received as a gift would be available to me too; all I had to do was be patient and not snatch his toys for a few days.
The new announcement for X-Plane 10 Mobile is clearly trying to whip up a little bit of sibling rivalry: “X-Plane 10 Mobile will make our desktop users jealous.” Besides being a chance to plug X-Plane 10 Global to mobile users, it is also a reference to the inevitable emails we did get (for X-Plane 9 vs X-Plane 9 Mobile) and will get from desktop users who are jealous of the development resources we spend on the mobile product.
Here are a few notes on X-Plane desktop and mobile and the relationship of the two products.
First, X-Plane 9 Mobile funded the development of X-Plane 10 Global. Had we not shipped X-Plane 9 Mobile, there would not be an X-Plane 10 Global, and I probably would not still be working at Laminar Research. So even if you ignore leverage and synergies between the code and you consider mobile products to be a distraction from the true purpose of X-Plane (desktop flight simulation), you can’t ignore that mobile is part of our business, perhaps a part that should not be discarded.
Second, we have been moving to a “two fronts” strategy where we can actively develop both products at the same time, and we have hired more developers so that we can do so. X-Plane 9 waited while X-Plane 9 Mobile was developed, and then the mobile product was more or less frozen for years while we worked on X-Plane 10 desktop.
The level of ping-pong hasn’t been as bad for X-Plane 10 mobile. We shipped 64-bit support, deployed the airport gateway, ported to Steam and shipped a brand new GPS while developing X-Plane 10 Mobile.* This has not been easy, but I think it indicates that we’re making progress towards “two fronts”. Both desktop and mobile suffer if they have to sit in the penalty box for years on end while the other is developed.
Finally, mobile devices are now powerful enough that we can share code and art assets between the two code bases. A few examples:
- Our minimum OpenGL requirements for the mobile product is OpenGL ES 2.0; our minimum desktop OpenGL version is OpenGL 2.0. And the capabilities required by both (render-to-texture via VBOs, vertex and fragment shaders) really are as similar as they sound.
- I have an iPhone 6 and a 2008 8-core Mac Pro. I ran the particle system performance test code on both and they run at almost the same speed. Obviously the Mac Pro is 6 years old and the iPhone 6 is a top-end phone. But there is no gap. This means that an older but supported desktop device will have similar characteristics to the top end of mobile devices. It’s just one big spectrum of computers now.
A lot of code was moved from desktop to mobile for X-Plane 10 mobile. But code was also developed for X-Plane 10 mobile with the intention of moving it back to the desktop version of X-Plane.
My take-away point: if you use our desktop product, you don’t need to actually be jealous of new toys in X-Plane 10 Mobile. Those toys are meant to be your toys too.
* One thing to note about this list: our ability to do more than one thing at once is mostly limited by who will do the work. Different developers and artists in the company have different skill sets; our developers are not interchangeable robots. Well, one of our developers is a robot, but I’m not going to name names.
In X-Plane 10, we started listing the plugin that crashed in Log.txt when a crash happens. This post is for plugin authors and provides one or two tricks to determine more information.
How Do We Know If a Plugin Crashed?
The mechanism for determining whether X-Plane or a plugin crashed is very simple: every time X-Plane dispatches to plugin code, we make a note that we are jumping into plugin land; when we jump back we clear the note. This way when we get a crash, if the crash is in plugin-land, rather than collect an automatic crash report (which we won’t be able to interpret) we let the OS crash handling go through.
This mechanism is not perfect. For example, when the sim reads an animation from a plugin, it does not mark the dataref read as plugin land, so a crashing dataref handler in a plugin will appear as an X-Plane crash.
How Do We Know Which Plugin Crashed?
The log from a plugin crash will say something like “X-Plane crashed because of a plugin: XSquawkBox.” (This is of course hypothetical; XSquawkBox would never crash. 😉 How does X-Plane know who crashed?
The answer is: it asks the X-Plane Plugin Manager (the XPLM). The XPLM tracks which plugin is executing at any given time; every time a plugin callback is dispatched, the XPLM notes which plugin owns the callback and marks that plugin as current. (This is also how the XPLM knows who owns resources; the plugin that is current when a resource is allocated owns the resource).
Note that if your plugin calls into X-Plane and X-Plane crashes, you still get blamed. For example, if you call us to draw an object (with XPLMDrawObject) but you pass us a bogus object handle, the sim will probably crash and you’ll take the wrap. This is probably the most useful case from a debugging standpoint; if your plugin calls our API and it crashes, we’re not going to look at it without some info about what your plugin was doing. You (the plugin author) get the crash report and can thus know in context how the call was happening.
(If these crashes were reported to Laminar Research, we’d have no idea what was happening in the plugin because we have no plugin symbols.)
Note that dataref reads are tracked by plugin, so if, in your flight loop callback, you call XPLMGetDataRef on a dataref provided by another plugin, and that plugin’s read handler crashes, they get logged, not you.
Who Is XPLANE_PLUGIN_XPLANE
X-Plane is actually registered as a plugin, because it provides datarefs. Therefore if you can crash in callback code registered by X-Plane (that was called from a major callback from a plugin, like a flight loop callback), we’ll log that we crashed due to a plugin, and that X-Plane is the plugin!
An example to illustrate this: your plugin registers a flight loop callback. From your flight loop callback you call XPLMGetDatavf to read an array dataref. The array dataref is owned by X-Plane but you pass an illegal pointer to XPLMGetDatavf, such that saving the values crashes.
X-Plane (the app) will crash inside X-Plane’s dataref handler (because writing to illegal memory will cause a crash). Because we’re in your flight loop callback, a plugin is going to get blamed, and because X-Plane’s “plugin” code (the dataref accessor functions ) were running, XPLANE_PLUGIN_XPLANE gets blamed.
There are only two bits of code that I can think of off hand where this can happen:
- Reading X-Plane built-in datarefs.
- Using the built-in 64-bit Lua allocator by sending plugin messages to X-Plane.
If you can consistently reproduce a crash blamed on XPLANE_PLUGIN_XPLANE, please let me know; it either means we have a bug in our datarefs, or your code is doing something illegal (that we should probably trap with error detection if it’s affordable). Note that we can’t catch the bogus-memory-for-array-dataref case without killing performance, so that’s going to remain a crash.
Crashing In a Worker Thread
Plugins do not have officially registered worker threads in X-Plane 10; therefore if a worker thread crashes, we don’t know if it is ours or yours, and we cannot issue approrprate blame. Currently X-Plane assumes that all worker thread crashes are sim crashes and auto-reports them.
If you do have a plugin that uses a worker thread and you think it might be crashing, you’d have to run from the command line with –no_crash_reporter to capture native OS crash information.
We (Laminar Research) cannot contact you to notify you of worker thread crashes caught by our crash reporter. Besides not having contact info for all plugins, the crash reports can’t trace the stack properly to determine who crashed, because we don’t have symbols.
Finding the Plugin At Fault in an Apple Crash Report
If our crash reporter catches a crash, we log the plugin at fault, and we make sure the complete Log.txt is written to disk. Similarly, if you log something with XPMDebugString, the log is written to disk immediately.
However, sometimes a crash will happen an we won’t catch it. Causes include a worker thread crashing, running on Yosemite (where, as of this writing, automatic crash detection is inoperative), etc. If you are on OS X the user will get a crash report, and it can actually be pretty useful.
The crash report does have one weakness: all of the plugins are named mac.xpl. How do you know who crashed?
It turns out each plugin’s binary gets a unique ID (UUID) which will be logged in the crash report along with its address range. You can calculate your own plugin’s UUID with
dwarfdump --uuid mac.xpl
The output will look like this;
bash-3.2$ cd DataRefEditor/
bash-3.2$ dwarfdump --uuid mac.xpl
UUID: 0C9A8216-95CB-2F45-D8B6-51B6168AC05D (i386) mac.xpl
UUID: 5A0D5A18-992D-FD89-3CF1-6299CBC8F03C (x86_64) mac.xpl
UUID: 66A96F33-3FFC-A36B-660C-533F006B590F (ppc7400) mac.xpl
This is the UUID for DataRefEditor. Note that DataRefEditor runs on three platforms (PowerPC, 32-bit Intel, and 64-bit Intel – you can actually run PluginAdmin in X-Plane 8, 9, and 10) and therefore it has 3 UUIDs, one for each architecture.
So if you see a mac.xpl loaded in an Apple Crash Report whose address range covers the crash and has your UUID, then you know you are the one who crashed.
(If I had known that so many crash report tools would delete the path to a crashing DLL, I would have considered a different naming convention for fat plugins.)
I’ve been fielding crash-related bug reports for X-Plane 10,30, and a lot of them are turning out to be caused by third party scenery packs.
Now there are two possibilities for why a third party scenery pack might crash X-Plane:
- There is a bug in X-Plane. In this case, I want to fix the bug!
- There is something wrong with the data in the scenery pack (that is, the data doesn’t meet X-Plane’s expectations). In this case, I want to change the result from a crash to some kind of clear error report that an author could use to fix the problem.
The Old Rules
During my work on the scenery system for X-Plane 8 and 9, my view was “garbage in, garbage out” – that is, if you feed junk data into the scenery system, the result would be unpredictable and could include a crash. In X-Plane 8, scenery load was completely synchronous – that is, your flight just stopped while we loaded files, so my idea at the time was that validation of scenery files should be done ahead of time, perhaps by the tools that write those files.
That logic (if it was even a good idea back then) doesn’t apply now. Loading is done entirely on other CPU cores and does not interrupt flight; we can take the CPU time to make sure incoming data is valid. And the cost of not validating the data is clear:
- The instability from badly formed data results in a sea of crash reports – if we have real serious sim bugs, it’s much harder to catch them in the noise.
- Because we don’t trap bad data, the end result crash can be in the OpenGL driver (due to cascading errors), and due to the way 64-bit crash detection works, we don’t get any trace of the contributing X-Plane code path.
- Authors who create scenery packs can’t tell if their pack is the source of crash, or how their data induced a crash.
My goal is to chase down crashes caused by third party scenery and both fix any bugs that emerge and tighten up error checking to detect the cause of crashes early on, in a way that is useful to authors.
The New Rules
Because X-Plane has not run validation over all scenery files in the past, there are probably scenery files in use now that contain data that I would consider illegal (either by the documented format, or just by my own view of how the file format was meant to work; I realize that second definition is quite useless to authors). But for illegal data that doesn’t crash the sim, I can’t just go in and change X-Plane to reject that data in the next patch. (“Surprise! Your scenery packs won’t load!” isn’t cool.)
So my thinking for validation is:
- Detected error conditions that would have caused a crash 100% of the time can be handled by any clear means possible, including quitting the sim with an error message. If the sim can handle the error and continue, that’s better, but even a controlled quit with diagnostics is an improvement over an uncontrolled crash.
- Detected error conditions that would have crashed the sim some of the time need to be logged but can’t quit the sim or cancel loading of the entire scenery pack; while a crash is serious, users were probably still enjoying the scenery pack (and merely being annoyed at how unstable their flying experience was). The user experience shouldn’t get worse.
- Detected error conditions that have only minor effects (e.g. out of range values that might cause rendering artifacts) whose legal values are well-documented can be logged, perhaps with a noisy error. The pack works, so quitting or refusing to load is not acceptable, but if the error is too quiet, the condition will never get fixed.
- Detected error conditions that have only minor effects whose legal values were never documented need to be logged unobtrusively; it’s not fair of me to penalize scenery packs for failing to read my mind.
On those first two points: one of the issues that complicates validation of scenery packs is that some errors are not discovered until mid-flight, because scenery is processed while you fly. This means that a large scenery pack might contain areas that are problematic to fly in; if the original author doesn’t cover every inch of scenery in a test flight, the author might not know about the problem.
The ATC system takes a more aggressive approach: it validates every single .atc file at load time even if it doesn’t need to use the data until later. (Virtual air traffic controllers are instantiated near the aircraft as the user flies, but the data in the .atc files that will be used to set up those controllers is checked at load time.)
This approach wasn’t too expensive performance-wise: the .atc files are small and the ATC system has to read them anyway to index them for later use. I am not sure how to best replicate this for scenery; consider Alpilotx’s HD scenery mesh – reading every single DSF in Europe at load time isn’t acceptable performance wise but without that, how can he know that his scenery doesn’t contain areas that could crash X-Plane?
Please Send Me Bogus Scenery Packs
Here’s where I need you to help: if you find X-Plane is crashing, and you find that removing a scenery pack fixes the crash, please report a bug, including the scenery pack (or where to get it) and where you had to fly/start up to induce the crash. Even if it is the fault of the pack (and it may not be – it may be an X-Plane bug!!!!) I still want to clamp down on the failure condition.
