I’ve marked WorldEditor 1.2.1 as final. If you are a WorldEditor user, please upgrade from 1.2 to 1.2.1. Besides fixing a bunch of bugs (including annoying usability bugs), it changes the format of the zip files for submissions to the global airports to a format that is a lot easier for us to handle on our end.
The next version of WED will be 1.3 and will include more usability features; we’re not sure what else will go in that bucket or when we’ll cut it off and kick it out the door.
In a previous post I discussed a new facility in X-Plane 10.25r1 (coming real soon) to disable aircraft-attached objects for performance optimization. There is a second use for this feature: performance analysis.
This post is targeted at aircraft authors, particularly authors who create complex aircraft with Lua scripts or plugins.
The basic idea here is to remove work from X-Plane and measure the improvement in performance. I strongly recommend you look at X-Plane’s framerate in milliseconds. An improvement of 1 frame per second is a big improvement at 15 fps and almost nothing at 60 fps. But saving 5 ms is always a good thing, no matter what framerate.
The GPU Is Confusing
Your graphics card is basically a second computer, with its own memory, its own bus, and its own CPU (in this case called a GPU). Ideally the CPU on your computer and GPU on your graphics card are both working at the same time, in parallel, to present each frame of X-Plane.
OpenGL drivers accomplish this by building a “todo” list for the GPU, which the GPU then does on its own time. Ideally the CPU is off computing flight models and running plugins while the GPU is still filling in mountain pixel shaders on screen.
For any given frame, however, one of the CPU or GPU will have more work than the other, and the other will be idle some of the time.
If you have a lot of GPU work (e.g. 4x SSAA HDR with clouds on a huge screen) and not much CPU work (e.g. no autogen) then the CPU will have to wait. It will give all of its instructions to the GPU for a frame, then the next one, then the next one and eventually the GPU will go “stop! I’m not done with the frames you gave me” and the CPU will wait.
If you have a lot of CPU work (e.g. lots of objects and shadows and plugins) but not much GPU work (e.g. a small screen without HDR on a powerful graphics card) the GPU will wait; it will finish its entire todo list and then go “uh, now what?”. Until the CPU can get more instructions ready, the GPU is idle.
Here’s the key point: your framerate is determined by the processor that is not idle. If your GPU is idle and your CPU is not, you are “CPU bound”; if your CPU is idle and your GPU is not, you are GPU bound. Optimizing the use of the processor that is idle will not improve framerate at all.
Viewing Specific Processor Load in X-Plane
X-Plane’s “frame rate” data output gives you two useful numbers for figuring out where X-Plane is spending its time:
“frame time” gives you the total time to draw one frame, in milliseconds.
CPU load give you the fraction of that frame time that the CPU was busy.
For example, my copy of X-Plane is showing 9 ms frame time and .78 CPU load. That means that the GPU needs 9 ms to draw the frame, but the CPU needs 7 ms to draw the frame – I am GPU bound. (I am also in the middle of the ocean, hence the low numbers.)
Unfortunately if you are CPU bound (CPU load > 0.95 or so) there is no current display for the GPU’s utilization; we are working on that for X-Plane 10.30.
Analyze Performance By Subtraction
Putting it all together:
You can calculate the actual CPU time spent in your add-on from the CPU load and frame time data outputs.
You can disable your add-on to see how much CPU time is now used; the difference is the CPU time your add-on is consuming.
You can change settings to be GPU bound (and confirm that by seeing CPU load < 0.95). Once you are GPU bound, improvements in framerate when you turn off your add-on show the amount of time you used the GPU for.
Armed with this knowledge, you can find the real cost in milliseconds of GPU and CPU time of the various parts of your add-on. You can find out what costs a lot (the panel? The 3-d? The systems modeling) and then focus your optimizations in places where they will matter.
Support Performance Analysis in Your Add-Ons
In order to tell what part of you add-on is consuming CPU and GPU time, you need to be able to separate your add-on into its sub-components and disable each one.
If your add-on makes heavy use of plugin code or scripts, I recommend putting in datarefs that turn off entire sub-sections of processing. Good choices might be:
Bypassing custom drawing of glass displays in an aircraft.
Bypassing per-frame systems calculations.
Bypassing per-frame audio code, or shutting off the entire audio subsystem.
Turning off any 2-d overlay UI.
You use DataRefEditor to turn off each part of your system and see how much framerate comes back. If you get 5 ms of CPU time back from turning off your 2-d UI you can then say “wow, the UI should not be so CPU expensive” and you can then investigate.
X-Plane Supports Performance Analysis
The technic of shutting a system off and measuring the increase in performance is exactly how we perform performance analysis on X-Plane itself, and as a result X-Plane ships with a pile of “art controls” that disable various pieces of the simulator.
This article provides a big list of art controls you can use to disable parts of your aircraft and measure the performance improvement.
Here’s where object-kill datarefs come in: the “cost” of drawing an object is mostly inside X-Plane (in driver and OpenGL code) but also in the time spent calling dataref handlers that drive animation. With an object-kill dataref, you can turn off your objects one at at time and see which ones consume a lot of CPU and GPU time. If an object is found to be expensive, it might be worth an optimization pass in a 3-d editor, or some time spent to improve the scripts that control its datarefs.
I have a lot to cover here – a little for everyone I think.
10.25 Release Candidate 1 Is Up
If you are a third party developer using 10.22, and you haven’t participated in 10.25 betas, please go do so now. You can get the beta by running the installer and clicking “get betas”. (If you run the beta, you are auto-notified to update.)
This build sneaks in object-killing in Plane-Maker; thanks to the aircraft developers who took time to privately test this feature last week!
A Fix to the Plugin SDK
This section is just for the programmers. I investigated a three-way conflict between X-Plane 10.25, Gizmo 64-bit and the new 64-bit XSquawkBox, and what I found was a bug in the C++ wrappers that ship with the X-Plane SDK headers. XSquawkBox was using them, but they were not correctly updated for 64-bit.
They are now. So if you use the “XPC” C++ wrappers in your plugin, please go get the new headers!
I’ve written about this before on the X-Plane dev email list, but the short of it is that ‘long’ as a datatype is not safe for plugin use. A long is 64-bits on Mac/Linux but 32-bits on Windows. If you use long, your data structures change size, which is never what you want.
The SDK widget APIs sometimes store widget IDs (which are pointers) in integer slots. in order for this to work, the slots must be 64 bits wide. The old SDK (and XPC wrappers) use ‘long’ for this, but the correct type is intptr_t. The SDK made this change a while ago, the XPC wrappers made this change now, and you should be sure that your plugin isn’t using “long” with the SDK.
The failure mode of mixing ‘long’ and ptrs on Windows is exciting: the upper 32 bits of the address of the widget get cut off; as long as you allocate your widgets early, your widget address is probably less than 2^32, and you are okay. But if your plugin loads later, your widget IDs (which are secretly ptrs to structs) will be > 2^32 and converting them to long changes the address, crashing the sim.
This is exactly why Gizmo appeared to be “crashing” XSquawkBox: XSquawkBox was using ‘long’; if Gizmo ran first and allocated memory (which Gizmo is well within its rights to do!) then XSquawkBox’s widget IDs would be greater than 2^32 and the ‘long’ bug would kick in.
(I don’t know when Wade will release an updated XSquawkBox, and I do not plan to discuss XSquawkBox any more on this blog. You can follow XSB here.)
Whose Bug Is It Anyway?
The XSquawkBox + Gizmo crash illustrates an important point: if two add-ons work by themselves but crash when used together, we can’t know which add-on is at fault without more investigation.
In this case, the bug was in XSquawkBox. But before I investigated on my computer, Ben Russell reported to me that removing some initialization code from Gizmo also “fixed” the problem (in that it made the symptoms disappear). Yet we know from investigation in the code that XSquawkBox had the bug (using long for pointers on Windows).
The moral of the story is: if two add-ons crash together, we can’t know which add-on is fault by which add-on changes to “fix” the problem. It is very common in the OpenGL world for the driver team to change the driver to work around buggy apps, and for apps to work around problems in buggy drivers. A change to code might be a fix for a bug, but it might be a work-around, avoiding a bug that still exists.
Here’s my take-away point: identifying a conflict between two programs is a way to narrow down a bug, but it is not a fix. We (Laminar Research) almost always ask you to remove add-ons when you see a crash. This is not a fix! We want you to remove add-ons to identify the conflict between X-Plane and a particular add-on (or between two add-ons). The next step is for us to figure out why the add-on might crash X-Plane or vice versa. Typically we prefer to contact the add-on maker directly to get technical information. What we are looking for is an identified conflict with the minimum number of variables.
We’re on release candidate three of WorldEditor 1.2.1. RC3 fixed export of 970 airport files to include frequencies and fixed a bug where you could end up with a two-point hole in a polygon (which would then cause the airport to fail validation).
Hopefully this is the last RC and we can call WED 1.2.1 done shortly.
Huge thanks to Wade for getting the beta going, and to Christian for writing brand new Mac audio support!
Besides being big news for VATSIM users (who were stuck in 32-bit land from now on), 64-bit support is a great fit for XSB because it uses a lot of memory for CSLs, and thus was never very happy as a 32-bit plugin.
This announcement is just that; an announcement. XSquawkBox is not a Laminar Research product. Please do not report bugs to us (Laminar Research). Thanks!
Update: Wade says that this build is time-limited. This is the first public build; he’ll release a time-unlimited build once he’s sure he has the bugs out. So think of this as a public beta.
X-Plane 10.25r1 will go final soon (this week I think) and will be almost the same as 10.25b3. There is one new feature that I am ‘squeezing’ into the RC at the last minute. Squeezing features into an RC at the last minute is not very good project management discipline, but in this case the feature is simple in its implementation and will allow custom airplane authors to improve framerate a bit.
The feature is “object-kill” – that is, the ability to completely disable (via a dataref) attached objects from being drawn. Even when objects are hidden, they have CPU overhead; this features lets authors determine (in their plugins) which parts of the plane are not needed and completely remove them. That big link explains how to use it and what it does.
How the Killers Kill Objects
X-Plane is, for the most part, not like a first-person shooter or racing game; those games have incredibly detail-rich environments, but they also have players that (mostly) stay on the ground, and environments with a lot of near objects blocking far views. By comparison, with X-Plane, once you take off and climb to about 300 feet AGL, you can see nearly everything around you.
For this reason, X-Plane takes a fairly different strategy than shooters when it comes to figuring out which objects to draw, and which not to draw.
X-Plane only eliminates objects that are off-screen (behind you, off to the side), too far away, or too small due to distance.
First person shooting games try to eliminate objects that are occluded by nearby scenery. (They also eliminate the off-screen and too-small objects.) Here are two Wikipediaarticles on the basic techniques.*
We don’t try to eliminate occluded objects because the cost of searching for them would hurt fps at all times; since a lot of the time almost no objects will be occluded, we skip the step, saving overhead.
First person shooters often pre-compute “hint” information to determine which objects can be hidden. This is not possible for X-Plane because the loaded scenery may be a mix of several scenery packs; while we could pre-compute occlusion information for the global scenery, custom scenery might eliminate the occluders (using exclusion zones), rendering our pre-computed information wrong.
But there is one environment where X-Plane looks a lot more like a first person shooter: inside the virtual cockpit. The VC is an environment where there is a ton of detail at high density and a lot of occlusion.
Here’s the problem: first person shooters often set up culling data ahead of time in their modeling environments; the engine only evaluates the precomputed hints. But airplane scan be modeled in any number of programs: Blender 2.49, Blender 2.6, 3DS, ac3d, or any program that can create 3-d.
Therefore the intention of object-kill is to let authors do their own culling and simply tell X-Plane (via datarefs) the result. These techniques could be portal-style occlusion techniques, computed automatically, or simple rules coded in a script.
Inside/Outside Views
The X-Plane 3-d cockpit camera is a completely free camera – you can walk out of the cockpit, through the cabin, out the door, and go look under your wings. I think that this kind of ‘freedom to roam’ adds a nice element of ‘playability’ to X-Plane, but it also comes at a performance cost. We do not provide separate “interior/exterior” objects – every object is drawn in both the interior and exterior views, because the views are fluid. You can always look in the window from an exterior view, and you can always walk outside from an interior view.
If you need distinct inside/outside views to improve performance, you can use object-kill to implement this feature. Simply write datarefs to determine when the view is ‘inside’ or ‘outside’ and wire up each object with a dataref to kill it. Kill your interior views in exterior views and vice versa.
(See below for more subtle ways to optimize objects.)
How Would I Use This Feature?
I am not an aircraft author, so that’s a highly theoretical question, but here is how I would use this feature:
I would wire each object up to its own custom dataref and turn each one off in DataRefEditor. For each object, I would note the change in CPU and GPU usage. From this I can determine which objects are actually expensive.
For each object that is expensive, I would look at my model, and develop simple rules for when I can avoid drawing them.
I would code some ‘predicates’ – that is, functions that tell me about the state of my objects. The idea of predicates is to summarize a set of conditions in question form.
I would then code these rules into simple plugin-based datarefs to hide the objects at the right times.
Here are some of the predicates I would look at:
Is the camera in the cockpit? (This requires looking at the XYZ of the pilot’s head position and comparing it to a box I determine from my model, as well as the current view mode.)
Can the pilot see backward? (This requires looking at the pilot’s head horizontal rotation.)
Is the cabin of the airplane visible? (E.g. for planes with a cockpit door, is that door closed?)
Then I can easily write rules like:
“if the camera is in cockpit and (the pilot cannot see backward or the cabin is not visible) kill interior cabin.”
“if the pilot is in the cockpit and the pilot cannot see backward, kill the entire external model.”
“if the pilot is in the cockpit, kill any details underneath the aircraft (gear, gear door, baggage doors, etc.).”
Two last thoughts on performance tuning by removing objects:
It make sense to spend more time tuning performance on the views that matter most. Note that the above rules are all for the virtual cockpit. The virtual cockpit is where users will spend most of their time in the plane, including hand-flying it, so performance matters.
You need the most fps gains on the views that are slowest. So if the virtual cockpit is slower than external views (which it almost always is) it makes sense to do more work to optimize the virtual cockpit.
* I wrote some very simple anti-portal culling code for X-Plane in an attempt to measure what kind of performance gains we might get. The conclusion: not much.
The problem is that X-Plane heavily groups objects into large clusters so that it can submit them via hardware instancing. When a lot of objects are visible, this optimization makes a big difference in performance – it can be a 10x improvement in speed. But the large groups also mean that, at any given time, at least some part of a cluster is on screen, even when portals are used.
When I turned off the grouping, portal culling cut the object load in half (after some hacking the engine). That’s a big win, but the grouping gives us 10x the object budget, so we hurt object performance by 10x, and then improved it by 2x – not a win in the end.
The addition of airport buildings to the default installation of X-Plane in 10.25 makes it much more important than it used to be to prioritize your scenery packs. It also makes it much more important that custom scenery exclude the scenery below it.
I have heard reports of users going into X-Plane and deleting the airports we ship to deal with conflicts. This is way more work than necessary, will be undone by the updater, and simply isn’t necessary. So this post will explain how you can sort out scenery conflicts without having to change content.
Some Light Reading
I try not to write up documentation in the blog – it’s not really useful to require users to search 900+ posts to find something I mentioned off-hand in a beta four years ago. So I wrote a knowledge base article. Please go read the entire article. I’ll go get another cup of coffee.
Let me summarize how things are supposed to work:
Custom overlay scenery packs should be higher priority than our airports. If you went out and got a custom scenery pack, you almost certainly want to see it, not our work. (Note that if you get X-Plane, run it once, then install custom scenery, this is what happens automatically.)
Custom overlay scenery packs should have exclusion zones to mask out the scenery below them, whether it is autogen, airports we ship, or rogue trees that have decided to invade a runway. (The need to exclude what is below you is not new – this has been in the sim since X-Plane 8.
Mesh scenery packs should be lower priority than our airports if you want to see our airports on top of custom mesh scenery. (This step is not automatic.) Mesh scenery excludes everything (think of the mesh as one giant excluzion rectangle) so you can’t have overlays below meshes.
Where Theory Meets Practice
My favorite Farside cartoon ever is a picture of two sets of engineers standing next to the ends of two highways that were clearly meant to connect – the highways are misaligned and do not connect. The caption: “Where theory meets practice.”
And like the cartoon, the reality of scenery packs doesn’t meet the theory in a few ways:
Some custom airport overlays don’t have exclusion zones. There was nothing under them in the past, so the authors didn’t bother to exclude.
Some users have their global scenery at the wrong priority level. The easiest way to have this happen is to delete scenery_packs.ini – the prioritization is lost and our airports end up on top.
If you install a base mesh into X-Plane, it goes to the top of the priority list, which is stupid; it needs to be below all of the overlays.
What is particularly concerning is that users way of fixing this (going in and editing the actual scenery packs) is probably making the situation more complicated, not less. Here’s what I think needs to happen:
Custom scenery authors need to start excluding their scenery. This isn’t a new rule; I consider it a bug to ship an airport without exclusion zones. If you have a custom scenery pack and it doesn’t have exclusion zones, ask the author to add them. It is not that hard to add a few exclusion zones in WED.
Users need to stop deleting their scenery_packs.ini. The .ini file is there to help you resolve conflicts, but it won’t work well if you constantly delete it. Once you set up your priorities properly X-Plane will not change them! Put your base meshes on the bottom, default airports in the middle, custom scenery in the top, and you are ready to go.
X-Plane needs to get the priority of base meshes correct when they are first installed – the fact that users have to go and ‘fix’ the scenery packs list is really annoying. We are looking at ways to fix this now, but unfortunately we don’t have any plan that we can execute in 10.25. This is high priority for a future update.
X-Plane needs to keep its own overlay scenery (the airports we ship, the demo areas) at a lower priority than any custom overlays that users add. This is the expected behavior and should be totally automatic.
Exclusion Zones and Custom Scenery
A number of authors have rightfully complained that the exclusion zone is a blunt object and doesn’t provide a way to carefully integrate autogen and custom scenery.
I agree, but I consider the problem to go significantly beyond exclusion zones; there is no way for a custom overlay to “edit” base vector data (e.g. to say “I want to move that road over there.”) The underlying data doesn’t have any kind of identifier to refer to it, and the underlying data can change if we recut and take newer OSM data.
I think the future solution to this lies not in more precise or powerful editing in overlays, but rather in the fact that we’re getting more network bandwidth, and the global data we have access to is getting better. Therefore the solution is to (1) push updates to the base scenery and (2) ensure that the base scenery is correct. With this process working, misalignments between custom scenery and the default scenery will be less important, and authors can simply fix small but ugly bugs in the base scenery (via OSM) and the result will be better scenery alignment in an update.
Exclusion Zones and Shared Global Airports
It is possible to include exclusion zones in an airport you share with the global airport database for inclusion in X-plane; some authors have already done this.
There are some cases where an exclusion zone is a must-have:
If the airport gets a new runway, there will be autogen in the location where the old runway used to be; a draped polygon + exclusion zone can “clear” the area for airport.
If the airport has trees on the runway (a bug in the default scenery) you can “fix” the bug with an exclusion zone.
The global shared default airports are one case where I don’t think we need complete exclusion in an overlay – they are meant to be the lowest priority overlay, and therefore don’t need to exclude other scenery.
X-Plane 10.25 beta 3 is now available. If you already have 10.25b1 you will be prompted to auto-update. This build adds some new dry-climate urban textures; here is a comparison.
10.22
10.25b2 with dry urban
If we don’t find any more show-stopping bugs, there will be one release candidate, with some updated level-of-detail distance internal settings.
In the future, I will try to write up the scope of changes for the beta-run when the first beta comes out, so you don’t have to ask “will you fix X in this beta?” For small betas, we try to have all intended fixes available in beta 1, but sometimes (such as with dry urban textures) the fixes go into later betas because they are not ready).
10.25b2 is Dead On Arrival. This happens periodically to betas – they pass internal tests, I upload them, and then immediately find something bad wrong with them that is a show stopper.
So for now 10.25b1 is the current beta (again) – just re-run the updater if you accidentally got beta 2 in the 10 minutes that it was available. Beta 3 will be up as soon as I can get a new beta cut; due to the time it takes to sync our servers, that might be late tonight.
See the beta wiki for what was in beta 2 – beta 3 will be all that plus it should actually work.
I just built a release candidate of WorldEditor 1.2.1r2; if you work on airports, please try it and post a WED bug if something is really borked.
There are a few minor bug fixes, some usability tweaks, and two major ones:
The export format for sending zip files to Robin is modified to make things easier in the collection process. For this reason, we’d like everyone submitting airports to go to WED 1.2.1.
The library view now hides the resources marked private or deprecated in X-Plane 10.25 or earlier. So WED 1.2.1 and X-Plane 10.25 provide a much less cluttered library view. (WED 1.3 will have a nice improvement to filtering as well.)
For more details on what changed, all commits to WED 121 are visible here. (The commit notes are listed, and the yellow tags tell you the version where we then re-shipped the code.)
We are discussing internally how to handle a more permanent airport submission tracking system. My current thinking is that eventually we’ll build file transfer directly into WED – that is, you’ll say “share this airport” and WED will call up our servers directly and send the airport; no zip files and email. But that’s for the future.
X-Plane 10.25b2 should be “real soon” I think – I’m just waiting on one more change to cut the new beta.
