I’ve been working on a conformance test for X-Plane. The idea is simple, and not at all mine: X-Plane 945 can output a series of test images that are the same on each run. The images cover a variety of rendering conditions. If a video driver is broken, the images will be corrupted.
You can learn more about how this works here: I am working on the 945 timedemo tarball now.
The main driver for this is to help NVidia, ATI, and Apple to integrate X-Plane into their dedicated testing. With X-Plane as part of their test systems, they can catch driver bugs the easy way – the day after the code is changed, rather than months later after a series of angry web posts. X-plane 945 includes a number of new features as part of its framerate test to help with this process.
My hope is that this will benefit users (who will see less bugs) and the driver writers (who can get feedback on code changes in a uniform and reproducible manner). Here are the eight images in the sample conformance test I wrote, based on the LOWI custom airport scenery.
Version 2.0 of the plugin API (available in all versions of X-Plane 9) introduces three new routines that, for the first time, allow plugins to work with OBJs.
XPLMLoadObject and
XPLMUnloadObject load an OBJ file (a model mesh) into memory, and then purge it when done. That part most users understand. The routine that causes confusion is
XPLMDrawObjects. When you draw an OBJ with
XPLMDrawObjects, you are not creating anything long-lasting or persistent in the world. Your object will be visible for one frame on screen, and then will disappear unless your draw it again. Your object is not part of any of the physics calculations.
To put this in perspective: when you make a new window using the Widgets API, the window is persistent – it exists until (1) you delete it or (2) your plugin is unloaded for some reason. You don’t have to do anything per frame to “maintain” the window – you make it and it exists.
Objects are not like that. You cannot make an object “exist” in the X-Plane world – you can only draw it once per frame using the drawing callbacks. Essentially the draw-objects API is a lower level API.
Building a Layered System
Plugins operate in a layered environment, with lower level code on the bottom and your plugin at the top. The layer stack might only be 2 layers deep (XPLM on the bottom, plugin on top), or there might be several layers. Consider XSquawkBox:
- The UI is drawn using the XPWidgets API. The XPWidgets API gets its drawing from the XPUIGraphics API, and the XPUIGraphics API changes OpenGL state using the XPLMGraphics API. So we’ve built up a layered system: basic OpenGL supports drawing, drawing supports user interface, user interface supports the plugin.
- Similarly, multiplayer is done using a library that isn’t part of the basic plugin system (but is open source): libXplaneMP. So here the XPLM supports drawing airplanes, libXplaneMP uses that to create a full multiplayer API, then XSquawkBox uses it.
The alternative to a layered system would be a “monolithic” one. Under a monolithic system, the only API for airplanes would be libXplaneMP, and the only way to create user interface would be widgets. Sandy and I usually prefer the layered approach because it provides a lot more flexibility. If you like widgets, great, use them. If not, no problem – roll your own on top of XPLMDisplay.
The Plugin System Is a Foundation
When Sandy and I cannot provide all of the layers, we have a strong bias toward providing the bottom layer, for an obvious reason: if the bottom layer isn’t in the plugin system, it may be impossible for anyone else to create it. So typically if we have a choice between a high level vs. low level API, we’ll put the low level API in first.
This is precisely what is happening with object drawing – we have the low level API (“draw an object”) but not the high level one (“create an object in the scenery system”). Since we have provided the lowest level, it is possible to code persistent objects in your plugin by layering on top of our API. By comparison, had we only provided “create an object” it would be pretty close to impossible to draw an object for one frame – if you didn’t want a scenery object, the API would be inflexible and useless.
I’ve been on the road a lot for work, so my apologies to everyone whose email I am sitting on. Most of my time these days is being spent on new next-gen tech. But there are a few things I’m hoping to get done in the short term:
-
Cut a new time-demo test. This might seem like a low priority item, but it’s not. Apple, ATI and NVidia all run continuous automatic tests of their video drivers, with many applications and games. They have rooms full of computers that continuously run through 3 minute sections of Quake and Call of Duty, etc. If they introduce a driver bug while doing new development, these machines catch the problem immediately.
The new time demo (based on 945) will have a number of features to make X-Plane a more useful test case. If we can make X-Plane into a test case, then they can catch bugs early, and that means you don’t have to see them.
-
Bring WED 1.1 to beta. The only thing holding it back is the DSF exporter, and I did have about two hours to poke at it last week. I’m hoping if I can find just a few more hours, I can finish off the exporter.
-
Examine 950 bugs. I have half a dozen bug reports against 950 beta 1. 950 will be a small beta but also a slow one, because Austin and I have a lot of other things on our plates. If you haven’t heard back from me on a bug report, probably it’s still on my to-do list.
We’ll see how much of that I can get to in the next week.
There’s basically one reason why WorldEditor developer preview 2 is a “developer preview” and not a real beta: the DSF overlay export code isn’t complete.
The problem is that, unlike an airport, an overlay has to be “cut” to the DSF tile boundaries. This is made slightly tricky by the fact that the overlay can have (1) bezier curved segments and (2) a UV map on those segments. My existing toolkit of polygon editing routines doesn’t handle this case yet.
I have no idea when I will have time to complete this code. It is the number one piece of code that, if I had a quiet single afternoon of unexpected time to code, I’d pound it out. If I were stuck in an airport with my laptop, I’d pound it out. It should give you some idea of how busy things are that it still isn’t done.
In the meantime, there is the scenery tools bugbase. By filing a bug, your issue won’t get lost even if it’s a while before I get to it.
A few quick rants about the bug base:
- Most likely the first thing I’ll do when I do get to your bug is just ask you for more info. Consider the bug to be as much a business card so I can make contact as well as a bug report.
- Some bugs may get kicked out.
- Do not file X-Plane bugs in the scenery tools bug base! The scenery tools bug base is not where we store sim bugs.
Do not bother to ask for direct bug base access for X-Plane itself. You cannot have it. The ratio of submitted “bugs” for X-Plane to actual bugs is at least 10:1. That is, 90% of you think that you should file a bug when you have a tech support question. Now you might be in that 10% (particularly if you’ve made it all the way down to this blog post), but we can’t set up open infrastructure with those numbers. My hope is that the scenery tools are self-selecting to the point where people who are using developer-preview tools know what a bug report is.
The file loading code in 950 beta 1 for Windows is slower than 945. Sometimes. This will be “fixed” in beta 2. Here’s what happened:
The scenery system uses a number of small files. .ter files, multiple images, .objs, etc. This didn’t seem like a problem at first, and having everything in separate text files makes it easier to take apart a scenery pack and see what’s going on.
The problem is that as computers get bigger and faster, rather than a scenery pack growing bigger files, they are growing more files. The maximum texture size has doubled from 1024×1024 to 2048×2048. But with paged orthophotos, multicore, and a lot of VRAM, you could easily build a scenery pack with 10,000 images per DSF.
That’s exactly what people are doing, and the problem is that loading all of those tiny files is slow. Your hard drive is the ultimate example of “cheaper by the dozen” – it can load a single huge file at a high sustained data rate. But the combination of opening and closing files and jumping between them is horribly inefficient. 10,000 tiny .ter files is a hard drive’s worse nightmare.
In 950 beta 1 I tried to rewrite part of the low level file code to be quicker on Windows. It appeared to run 20% faster on my test of the LOWI demo area, so I left it in beta 1, only to find out later that it was about 100% slower on huge orthophoto scenery packs. I will be removing these “optimizations” in beta 2 to get back to the same speed we had before. (None of this affects Mac/Linux – the change was only for Windows.)
The long term solution (which we may have some day) is to have some kind of “packing” format to bundle up a number of small files so that X-Plane can read them more efficiently. An uncompressed zip file (that is, a zip where the actual contents aren’t compressed, just strung together) is one possible candidate – it would be easy for authors to work with and get the job done.
In the short term, for 950 beta 2, I am experimenting with code that loads only a fraction of the paged orthophoto textures ahead of time – this means that some (hopefully far away part) of the scenery will be “gray” until loaded, but the load time could be cut in half.
There is one thing you can do if you are making an orthophoto scenery pack: use the biggest textures you can. Not only is it good from a rendering perspective (fewer, larger textures means less CPU work telling the video card “it’s time to change textures”) but it’s good for loading too – fewer larger textures means fewer, larger total files, which is good for your hard disk.
(Thanks to Cam and Eric for doing heavy performance testing on some of the 950 beta builds!)
This blog post is for amateur plugin developers. By amateur I mean: some plugin developers are professional programmers by day, and are already familiar with all aspects of the software development progress. For those developers, the SDK is unsurprising and performance is simply a matter of applying standard practice: locate the worst performance problem, fix it, wash-rinse-repeat.
But we also have a dedicated set of amateur plugin developers – whether they had programming experience before as hobbyists, or learned C to take their add-ons to the next level, this group is very dedicated, but doesn’t have the years of professional experience to draw on.
If you’re in that second group, this post is for you. Explaining how to performance tune code is well beyond the scope of a blog post, but I do want to address some fundamental ideas.
I receive a number of questions about plugin performance (to which the answer is always “that’s not going to cause a performance problem”). It is understandable that programmers would be concerned about performance; X-Plane is a high performance environment, and a plugin that wrecks that will be rejected by users. But how do you go from worrying about performance to fixing it?
Measure, Measure, Measure, Measure.
If I had to go crazy and recite a sweaty and embarrassing mantra about performance tuning so that I could be humiliated on YouTube it would go: measure, measure, measure, measure.
If you want your plugin to be fast, the single most important thing to know is: you have to find performance problems by measurement, not by speculation, guessing or logic.
If you are unfamiliar with a problem domain (which means you are writing new code or a new algorithm – that is, doing something interesting), there is no way you are going to make a good guess as to where a performance problem is.
If you have a ton of experience in a domain, you still shouldn’t be guessing! After 5 years of working on X-Plane, I can make some good guesses as to where performance problems should be. But I only use those guesses to search in the likely places first! Even with good guesses, I rely on measurement and observation to make sure my guess wasn’t stupid. And even after 5 years of working on the rendering engine, my guesses are wrong more often than they are right. That’s just how performance tuning is: it’s really hard for us to guess where a performance problem might be.*
Fortunately, the most important thing to do, measuring real performance problems, is also the easiest, and requires no special tools. The number one way to check performance: remove the code in question! Simply remove your plugin and compare frame-rate. If removing the plugin does not improve fps, your plugin is not hurting fps.
It is very, very important to make frame-rate comparison measurements under equal conditions. If you measure with your plugin in the ocean and without your plugin at LOWI, the results are meaningless. Here’s a trick I use in X-Plane all the time: I set my new code to run only if the mouse is on the right half of the screen. That way I can be sitting at a fixed location, with the camera not moving, and by mousing around, I can very rapidly compare “with code”, “without code”. The camera doesn’t move, the flight model is doing the same thing – I have isolated just the routine in question. You can do the same thing in your plugin.
Understand Setup Vs. Execution
This is just a rule of thumb, and you cannot use this rule instead of measuring. But generally: libraries are organized so that “execution” code (doing stuff) is fast, while setup and cleanup code may not be. The SDK is definitely in this category. To give a few examples:
- Drawing with a texture in OpenGL is very fast. Loading up a texture is not fast.
- Reading a dataref is fast. Finding a dataref is not as fast.
- Opening a file is usually slower than reading a file.
- You can run a flight loop per frame without performance problems. But you should only register it once.
If you want to pick a general design pattern, separate setup from execution, and performance-tune them separately. You want things that happen all the time to be very fast, and you can be quite intolerant of performance problems in execution code. But if you have setup code in your execution code (e.g. you load your textures from disk during a draw callback) you are fighting the grain; the library you are using probably hasn’t tuned those setup calls to be as fast as the execution code.
Math And Logic Is Fast
Modern computers are astoundingly fast. If you are worried that doing a slightly more complex calculation will hurt frame-rate, don’t be. One of the most common questions about performance I get is: will my systems code slow down X-Plane. It probably won’t – the things you calculate in systems logic are trivial in computer-terms. (But – always measure, don’t just read my blog post!)
In order to have slow code you basically need one of two things:
- A loop. Once you start doing some math multiple times, it can add up. Adding numbers is fast. Adding numbers 4,000,000,000 times is not fast. It only takes one for-loop to make fast code slow.
- A sub-routine. The subroutine could be doing anything, including a loop. Once you start calling other people’s code, your code might get slow.
This is where the professionals have a certain edge: they know how much a set of standard computer operations “cost” in terms of performance. What really happens when you allocate a block of memory? Open a file? If you understand everything going on to make those things happen, you can have a good idea of how expensive they are.
Fortunately, you don’t need to know. You need to measure!
SDK Callbacks Are Fast (Enough)
The SDK’s XPLM library serves as a mediator between plugins and X-Plane. Fortunately, the mediation infrastructure is reasonably fast. Mediation includes things like requesting a dataref from another plugin, or firing off a draw callback. This “callback” overhead contains no loops internally, and thus it is fast enough that you won’t have performance problems doing it correctly. One draw callback that runs every frame? Not a performance problem. Read a dataref? Not a performance problem. (Read a dataref 4,000,000 times inside a for-loop…well, that can be slow, as can anything!)
However you should be aware that some plugin routines “do work”. For example, XPLMDrawObject doesn’t just do mediation (into X-Plane), it actually draws the object. Calls that do “real work” do have the potential to be slower.
Be ware of one exception: a dataref read looks to you like a request for data. But really it happens in two parts. First the SDK makes a call into the other plugin that provides the data (often but not always X-Plane itself) and then that other plugin comes up with the data. So whenI say “dataref reads are fast” what I really mean is: the part of a dataref read that the SDK takes care of is fast. If the dataref read goes into a badly written plugin, the read could be very, very slow. All of the datarefs inside X-Plane vary from fast to very fast, but if you are reading data from another plugin, all bets are off.
Of course, all bets are off anyway. Did I mention you have to measure?
* Why can’t we guess? The answer is: abstraction. Basically well structured code uses libraries, functions, etc. to hide implementation and make the computer seem easier to work with. But because many challenging problems are hidden from view (which is a good thing) it’s hard to know how much real work is being done inside the black box. Build a black box out of black boxes, then do it again a few time, and the information about how fast a function is has bee
n obscured several times over!
X-Plane 950 beta 1 should fix the stuttering present in 940. The bug was: the more you flew in one area, the worse the stutters would get – they’d clear out if you kept flying forward enough to scroll DSF tiles normally. Anyway, the bug was a book-keeping error based on the new “paged” airports from 940.* Fixed now – thanks to the users who got me a really clean test case to see this.
(Hint for users: if you can duplicate a scenery-related bug by saving a replay, or even if the bug can be reproduced in replay, that’s really, really useful to know! It makes the bug much more repeatable, since what scenery is loaded varies a lot with flight path.)
* Before 940 we would load the entire apt.dat file into memory on startup. With Robin receiving new highly detailed airport layouts every day, this was starting to use more and more memory. 940 and 950 will load a litttle bit of information about every airport and take note of the file that contains the layout – then when the sim needs a layout, it goes back to disk to pull up just the one airport in question, saving memory. We “page” the airports into memory only as needed.
My blog postings have been a little thin – but not because nothing is going on. Rather it’s a combination of working on next-gen stuff that is still in heavy development, travel, and (perhaps due to the travel, perhaps due to lack of sleep, or perhaps due to certain forms of Brandy brought on site by a certain British developer who won’t be named, but whose handle on the org starts with a P and rhymes with “Cropsman”, cough, cough) a minor fever that I’ve just about recovered from.
A user asked me about the design of the X-Plane updater – so this post is only going to be of interest to the few authors out there who are creating installers for their add-ons and need to update. We try to keep the file formats for X-Plane simple, e.g. “this one folder is your airplane”, so that if you don’t make an installer, the user won’t be overwhelmed with a complex unpack-and-install routine. But if you do need to install or update, well, here’s what we were thinking when we developed our installer.
We had a few needs for our installer:
- It had to be cross-platform to Mac, Windows and Linux.
- It had to update existing products as well as install new ones.
- The updates had to be minimal deltas, e.g. a demo can be 600 MB but an update should only be the 30 MB of files that really changed.
- We wanted the updater to do delta updates from any old version. (Some update systems require installing a number of patches in sequence. We have users who buy the DVD and then update a year later; we wanted to let them update in one shot.)
- Building the patches had to be really, really fast. We use our updater to publish our betas. So while X-Plane probably has 2-3 major patches in a single year and maybe 5 or 6 bug fix patches, we will (due to betas) cut perhaps 100 or more actual “updates”.
- On that last point, cutting separate updates by platform was a deal breaker, as it would triple the number of updates we had to cut. Having a demo install and an update be the same on the server was a big win.
Some of those needs are pretty specific to Laminar Research, but some might apply to a third party. There are commercial installers that will let you generate patches. If you don’t generate a lot of patches, you can probably use existing tools – basically you’ll spend a lot less time up front (since the tool set exists) but you might spend more time in the long run cutting separate patches by OS.
One of the first things we decided was to not use server-side technology or special protocol. That is, we don’t need any kind of smart servers to run the updater – the updates are just files hosted over HTTP on standard rented or owned apache servers. All of the work is done in the client. We did this for a few reasons:
- It lets us throw up the install anywhere – we don’t have complex needs for what’s on the server. Virtually any server will do. (As the company has grown, our server needs have grown too, so this is less of a concern now, but back then we had fewer servers in service, and were buying less overall server capacity.) If a server goes down, we can press another one into service about as quickly as we can move the file set to the server.
- We’re not server programmers. Coding the installer/updater on the client side let us leverage existing company technology, etc.
Here’s what we came up with. The basic idea of the installer is very simple. Note that since the installer uses HTTP files, you can “watch” the installer simply by downloading the files that it downloads.
- First the installer goes to a master server and gets a “version list”, which tells it what it’s going to actually get and what (multiple) mirrors are available. This one master file is the only file that must be hosted at X-Plane.com, and it allows us to change a very small file to press mirrors into service.
- Installs and updates are actually the same to us – it’s a set of files that the user should have once the install or update finishes. This starts with a file directory that lists every file (by path) that needs to be downloaded, as well as its MD5 signature.
- The directory also contains the MD5 of every old version of any file in the version, and lists files that have to be deleted to update the version (e.g. the file panel.png used to have MD5 signature 2934b..23abc2 but is now removed).
- The client, once armed with the file directory, can now download all files in the directory (install) or compare the existing files on disk to the directory by MD5 and only download changes.
- Individual files are on the server in zip form for download and decompression.
That’s pretty much it – you could write an installer in a scripting language using a tool like CURL or WGET – it wouldn’t be very complex because the installer is really just a clever, painted download+unzip tool.
Dealing With Modified Files
There is a bit of complexity in this design that you might not need for a third party installer: handling modified files. Note that the directory contains the MD5 not only of the current version of the file (to detect when no update is needed and save bandwidth) but also to see when the user has modified a file that is “managed” by X-Plane.
Before the installer overwrites/deletes a file that you own, it compares the file’s MD5 to the entire known list of MD5s for the entire version. If this file doesn’t match a known old version, it puts up the warning dialog box that you have modified a file that is about to be overwritten.
While we recommend that add-ons use scenery packs and other “safe” ways to customize the sim, this helps detect when a user has gone in and edited the cloud textures in Photoshop, and prevents us from overwriting them without warning.
Life would be easier if the Earth was a big cube.* I was reminded of this yesterday when I discovered (thanks to a bug report from an author) that X-Plane has been drawing objects with slightly incorrect heading for the last who-knows-how-long amount of time.
The bug (present in 945 and going back who-knows-how-long) is that the object’s heading can be off by up to 1 full degree clockwise or counter-clockwise. (This maximum error occurs at the north pole – average error for real use cases is more likely to be about 0.35 degrees.) The amount of rotation depends on how the scenery system is shifted.
The case the author sent me was a converted scenery pack, where a pair of parallel runways were modeled partly out of draped polygons (which don’t exhibit the bug), but with markings modeled as an OBJ. Depending on the scenery system’s “centering” the markings would be a little bit to the left or right of the draped polygon.
A brief aside: do not model your scenery this way! A bug is a bug, and X-Plane 950 beta 1 fixes this problem, but there are a number of reasons not to use a giant OBJ to model your runways.
- OBJs do not “hug” the terrain, and the Earth is round. You will never really get clean, artifact-free flat surfaces unless they can follow the terrain contour. Even if you “flatten” the terrain, it’s not flat – it’s constant MSL.
- The case where I see this more often is with buildings, where authors build every single terminal building in one OBJ. X-Plane only “connects” the OBJ to the ground at one point (0,0,0 in the object). So to get a good ground connection, you need your objects to be smaller, so they can all “sit” on the ground. (Technically they need to be rotated slightly to wrap around the Earth, but for the scale of an airport this error is only about 1/30th of a degree or so – hard to see.)
- Finally, X-Plane chooses to draw or not draw an object on a whole-OBJ basis. So if you have a huge object, it’s always in view (because at any one time some part of it is in view). In fact, even if the object is not in view, the larger the object, the more the “sloppiness” of the visibility check causes false positives. (The visibility check is optimized for speed, so it really answers “this object is probably in view”, erring on the side of drawing too much by doing less analysis.) So the bigger your objects, the more they are drawn.
Enough ranting – in this case the scenery was the output of FS2XPlane, and the author hadn’t gotten to cleaning the results. Fortunately the interim product showed the rotation bug.
X-Plane 950b1 (in beta now) fixes this problem – rotational alignment of a very large OBJ with a draped polygon should match between X-Plane (with all frames of scenery system reference) vs. WED 1.1 preview 2 pretty much exactly.
*This post is in the long and distinguished tradition of “life would be easier if” musings relating to scenery, e.g. global scenery would be easier if the entire planet was paved in asphalt, cities would be easier if their road grids only ran perfectly north-south and east-west, clouds would be easier if they weren’t translucent, and rendering would be easier if nothing on the planet ever was translucent or cast a shadow. So apparently a programmer’s paradise is somewhere between “Mad Max” and a Dali painting…
There’s a slight performance win to be had by grouping taxiways by their surface type.
Now clearly if you have to have an “interlocked” pattern of asphalt on top of concrete, on top of asphalt, this isn’t an option.
But where you do have the flexibility to reorder, if you can group your work by surface type, X-Plane can sometimes cut down on the number of texture changes, which is good for framerate.
X-Plane will try to do this optimization for you, but X-Plane’s determination of “independent” taxiways (taxiways whose draw order can be swapped without a visual artifact) is a bit limited and can only catch simple cases.
For what it’s worth, interlocked patterns of surfaces were much more a problem with old X-Plane 6/7 type airport layouts, where the taxiways were sorted by size, and there could be hundreds of small pieces of pavement.
