I seem to be having the same conversation with lots of third party developers via email, so I’m going to write up some of my recent thinking on Lua – if nothing else, it will save me typing the same thing over and over.
One thing was very clear in the X-Plane 10.20 beta: while authors can’t agree on Gizmo vs. SASL*, the entire authoring community is surprisingly united around Lua as a scripting language – it’s everywhere in the payware add-on market.
But Lua is being used for a lot of different things – and these levels of usage are paralleled in Gizmo and SASL; my opinion on the use of Lua must be qualified by which usage we are talking about.
- The simplest level is “datarefs and math”. At this point, to fully utilize the aircraft SDK, an author needs to be able to create and do very simple math with datarefs – something that is not possible inside an OBJ or panel.** Right now we don’t have an easy way to do this basic dataref math. Writing a C plugin and compiling for 3 platforms is a huge hurdle to jump just to wire up an animation to a custom rotary switch.
- Gizmo and SASL both provide ‘add-on’ SDKs – custom sound APIs, particle systems, physics, gauges – basically replacement SDKs for parts of the sim’s own extension system where authors wanted more than we provide. This stuff isn’t Lua at all – the underlying systems are coded in C++ and thus can only be maintained by the C++ developer who writes the Lua-based plugin. The development cost (in man-hours) to do a particle system in Gizmo or SASL is abotu the same as it would be to build it into the sim.
- Some authors have written fairly huge scripts in Lua – for example, doing complete systems simulations or navigation code in Lua. (At least, that’s what I am told, e.g. the JAR A320 – I haven’t read the Lua scripts myself.) This is “Lua as language for a huge program.”
This is my current thinking on these three tasks:
- Datarefs and math are a great use of Lua – it lowers the bar for authors, it’s exactly what scripting languages in games are meant for, and the underlying code in C++ is finite, limited, and therefore not a maintenance problem. I don’t know what LR’s relation to Lua, Gizmo, SASL, or scripting is yet, but I have been saying for a while (internally in company discussions) that we need something easier than C for this.
- I think that if an authoring SDK is limited in X-Plane, we (LR) need to make it better. In the most useful cases where people are doing things with Gizmo and SASL, we sometimes have on our road map features to add to X-Plane that are similar. (But note that these features aren’t necessarily coming soon – authors get a time to market advantage by using these outside SDKs.) I consider this plugin code to be a possible maintenance problem. For example, you can write graphics code in a plugin, but it may not integrate well with next-generation rendering engine features.
- I don’t see huge plugins or huge scripts as something LR should get involved in. If you want to make a truly huge or complicated add-on, that’s great, but it’s a big undertaking and it just takes a development team. I don’t know if Lua is good for huge development; the people who say no (people like me) have no serious Lua experience, and the people who say yes have no serious C++ experience. So far I’ve only heard from people who have lived on one side of the grass.
Anyway, one thing is clear: having LuaJIT work in a plugin is a necessity for X-Plane now; with 10.20 we’ve sunk the engineering cost to make this happen. I do not yet know how else (or if) we will leverage Lua.
* Don’t even bother to comment on Gizmo vs. SASL – I will delete any attempts to start a Gizmo vs. SASL discussion so fast that your head will spin around 360 degrees!
** No math in OBJs or panels. An OBJ is a 3-d model – it is data that you view, not a simulation to be executed! We do not want to mix visualization with simulation or data with code!
Edit: Normally I label the overly technical posts as being for a target audience, but I did not this time. My bad — this post is really only of interest (and will only make sense) to serious 3-d modelers – the advanced airplane and scenery nerds!
The brightness of specular hilights is out of sync between HDR and non-HDR mode in X-Plane 10.11. This wasn’t intentional – it’s a bug. 10.20 will fix this; specualr hilights will have the brightness you see in non-HDR mode (which we think is more like they were supposed to work*) but the tinting of HDR mode (non-HDR mode was overly red in 10.11).
I get a lot of requests for ‘hardness’ control – that is, the exponent that controls how wide a specular hilight is. We’ll get this feature eventually – it’s too important to ignore. The main delay in having this feature is that X-Plane uses a deferred renderer, so the specular exponent has to be stored in the G-Buffer somewhere, and storage is tight.
For similar reasons, it is very very unlikely that we’ll ever have RGB tinting of specular hilights. Doing so is 3x as expensive in G-Buffer storage, and we need that storage for other features.
* That is, unless we discover that non-HDR mode looks spectacularly bad in some cases, in which case we may have to push things a little more toward how HDR was.
We have received a number of bug reports on the livery system in X-Plane 10, as well as complaints from both users and authors about how the system works. This post describes what I am thinking for liveries; if this proposal seems hair-brained, the comments section would be a good place to counter-propose.
Two notes on UI discussion before we jump into the details:
- We hate check-boxes. If the UI debate is “which is better, UI A or B”, then the answer is pretty much never “let the user pick A or B with a check-box”, because the point of UI is to present the program in a human-friendly manner. Having two different UIs with selection forces the user to become the UI designer, which is inherently harder. So while we don’t have zero check-boxes in our UI, we really try to keep the ‘have it your way’ stuff to a minimum; X-Plane is a product for sophisticated users and if we put in a check-box every time we get a request for one, the product would be absolutely unusable.
- In arguing for UI, describe what you want to do, not how you want to do it. If you describe an implementation but we can’t do the implementation, the discussion is over. If you describe what you want to do, we can find several implementations and pick the best one.
Liveries Now
As of X-Plane 10.11, you can:
- Select your livery when you open a plane using Open Aircraft…
But you cannot:
- Change the livery without reloading the airplane, mid-flight.*
- Select your livery from the Quick Flight dialog box.
- Change your livery quickly (since a plane reload is slow).
- Change which livery the QF dialog box picks for you – it always pick “no livery” (that is, default paint).
These first two bugs go hand-in-hand. Because you have to reload the plane to change liveries, you lose the initialization that Quick Flight may have done for you.
* There is one bug to note: if you go to the open aircraft dialog box, select a different livery and hit cancel, X-Plane will change your livery on the fly but not reload the plane. Some users have noticed this and use it as a back-door way to change livery mid-flight.
For 10.20
For the short term, I am converting the bug in the Open Aircraft dialog box into a feature – the dialog box will have a “change livery and keep flying” button to change your livery on the fly, and the “cancel” button will not unexpectedly change your livery. Thus the functionality of the old “open livery…” v9 functionality will be part of “Open Aircraft.”
This isn’t perfect, but it at least addresses the issues of changing liveries quickly, or changing them while flying, without having to know about and use a hidden UI bug.
For the Future
There are more features we’d like to do in the future:
- Have a dedicated livery-change UI again – perhaps one that ‘floats’ over the main window while you fly, so that you can see the liveries as you click on them. This would be a quicker way to rapidly preview a large number of liveries.
- Remember the last user selected livery in the per-aircraft preferences (along with quick-look keys) so that if you go to open an aircraft and don’t do anything else, we can pick the last livery you used.
- Someday have ‘drill down’ functionality in the Quick Flight window so that you can set more detailed info about your plane (livery, weight and balance, fuel, weapons) and then go back to the main QuickFlight window. This is a long ways off though.
None of this is going to make it into 10.20 – I mention it only to explain where we’re trying to go – the change for 10.20 are designed to not interfere with this future work. The future work is all based on user selected livery choice, with preferences an easy selection.
Livery Choice in Plane-Maker
This finally brings me to livery choice in Plane-Maker. I have always considered this a bit of a weird feature; it is inoperative in 10.20 and we are looking to get rid of it permanently – that is, setting a livery in Plane-Maker has no use in any of the above plans. Our view:
- The first livery you see if you first open a plane for the very first time in QuickFlight will be the default one.
- Preferences should record the users choice of livery, and this should be done entirely from X-Plane. It should not be necessary to open Plane-Maker to change livery preferences, any more than Plane-Maker is needed to change rendering settings or joystick preferences. User customization should be 100% in-X-Plane.
As I said before in the top of this article, if this seems half-baked to you, please describe what you want to do, not how you want to do it. And please understand that when it comes to UI, less is more; we need to identify a small, highly leveragable set of UI options to cover a wide variety of features. We don’t want to have 6 ways to do everything!
X-Plane 10.20 beta 3 is out. This build went out without new art assets. We have a pile of new art assets waiting to go out the door: autogen from Alex, more terrain textures from Albert, an airplane upgrade from Javier, and fixes to airplanes and aircraft from Tom.
But I held them back and kept them out of this release. Why am I such a bastard?
The answer is: two complex low level changes went into this beta, and I want to capture at least one beta’s worth of crash report data to check for destabilization. The new art assets may use more memory (especially if users crank them up to see what’s going on); if we rolled everything into one beta, we wouldn’t be able to tell art-induced crashes from code-change induced crashes. So we’ll roll things out in steps. As I have said before, the purpose of betas is not to have an awesome flying experience, it’s to find bugs!
Bear in mind that not all of the bugs reported are fixed; we cut beta 3 to get these two specific code changes fixed; more bug fixes are coming soon. Please see the release notes for what is fixed. please do not re-report bugs unless they are listed as fixed but still show up as broken on your machine.
I’m hoping to get a beta 4 out over the weekend with art assets if beta 3 goes smoothly.
So What Did You Do?
So what are the two bug fixes that are major enough to warrant their own beta?
- A change in the load order between X-Plane’s sound code and OpenAL. I realized that the old way we load sound (e.g. in 10.10) makes it virtually impossible for a globally installed plugin (one in Resources/plugins) to share OpenAL, even if it follows all of the rules set out in the OpenAL plugin tech-note. This change fixes that, apparently fixes conflicts between Gizmo and SASL, and hopefully should let plugins do the right thing. Plugin developers, see here for more info. (The original load order assumed that OpenAL would be more robust to two copies being opened; it now appears that this is not the case and OpenAL can be quite picky at times.)
- The OS X 64-bit memory layout has been modified to be compatible with LuaJIT 2.0, the Lua engine inside most Lua-based plugins (e.g. Gizmo, SASL, FlyLua, etc). This does not entirely fix the porting problems developers have been seeing with LuaJIT, but it at least makes the plugins work under light memory conditions. (These plugins would fail 100% of the time in beta 2.) Here’s the technote; I am in contact with most of the plugin developers using Lua, but if you use Lua and haven’t heard from me, email ben at x-plane dot com.
These are not the only bug fixes in beta 3, but they are the two that are weird enough that we want ‘clean’ bug data, without additional art assets to cloud our view.
So: hopefully these two changes will go smoothly and we’ll get on to beta 4 and art assets soon. The artists work their butts off to create this stuff and we want to roll it out!
Will Lua Ever Work With 64-Bit OS X?
A few developers who depend on SASL and Gizmo have asked me whether Lua will work with 64-bit X-Plane on OS X. The only answer that I can give is that I do not know, but I am optimistic that we will engineer a solution. I consider making Lua work to be a very high priority.
Lua is in almost every commercial third party airplane we’ve seen in the last few years; the market has spoken and third party developers have said “we like Lua.” So I think it’s worth doing some serious engineering to keep Lua working. Making Lua work with 64-bit OS X is currently my top priority bug.
I’m trying to migrate our developer documentation from the Wiki to this site. See the menu up top for articles, file formats, and tools.
One document I just posted here: A Checklist for Updating Aircraft to X-Plane 10. If you are working on airplane conversion, I strongly recommend looking over the various items. A lot are “check that X isn’t hosed” – there are very few mandatory changes.
One thought on alpha and HDR and aircraft:
If you need to create a final flat part of a panel out of multiple layers, some translucent, and you are using panel texture, it is better to composite the entire piece of panel in the panel texture than to layer polygons in your OBJ.
For example, consider the annunciator display on the dashboard that many airplanes have. There are two ways to build this panel:
- Model the entire thing in the panel texture. The background can be part of the panel background, and each annunciator is a generic instrument.
- Each annunciator sits on a transparent background, and the background of the annunciator panel is a separate piece of non-panel texture, on a second mesh, directly behind the first.
Choose option 1! Tom had to convert the Baron’s annunciator panel from method 2 to method 1. A few reasons to prefer method 1:
- The sim does not allow for truly emissive translucent lighting in an OBJ, so the results look better if composited in the panel texture.
- You can get shadowing artifacts between the polygon layers.
- Spill lighting on the dashboard will not look quite right if there is alpha.
You won’t use up that more panel texture, but you’ll make your life easier. This technique wasn’t totally possible before, but now with GLOBAL_cockpit_lit you can alwys have your panel handle real lighting.
You know you’ve got a good bug when a user reports that his airplane’s wing disappears when he turns the battery switch off. It turns out that this is the same bug as the 747 rolling over to the side.
X-Plane 10.10 (beta 1 all the way through RC1) has a bug in the IO code that scrambles the electrical system of airplanes on load, sliding the electrical system selections on the “bus 2” page of Plane-Maker over by one slot (except for the last two columns, which get totally scrambled). Then, due to an accident chain that would make the NTSB blush, the results pop out in the flight model, often as an incorrect roll tendency for airliners.
This (and a number of other bugs — thanks to all of the airplane authors who tried their planes and reported bugs against rc 1) will be fixed in rc2, which will hopefully come out “real soon now™”.
If your plane is saved in X-Plane 10.05 format, this bug won’t affect you – the fix to X-Plane will cause your airplane to just work. If you have already saved your airplane with Plane-Maker 10.10, you may have to re-enter some of the amperages and bus choices for your electrical system – the amount of data scrambling depends on how you used Plane-Maker. The good news is that the damage is limited to the second electrical page so at the very worst, you’ll have to re-enter some electrical system data using rc2.
Trusting Beta Plane-Maker
This bug brings up the question: how much should you trust a beta or RC Plane-Maker? The short answer is: “not very much”. Here are my recommendations.
- Never, ever, ever release an airplane against a version of the sim that hasn’t gone final. Things do change in release candidates, sometimes major things when we find a bug like this. Wait until the version is declared “done” before you release your airplane!
- While always saving backups of your work is always a good time, it is especially important when using Plane-Maker betas. Assume a beta Plane-Maker might erase the .acf you are working on entirely; while we’ve never had it do something that bad, it has produced incorrect ACF files before due to bugs.
- Beta Plane-Makers are good for testing and trying new features and experimenting, but not for production work; wait until we go final to permanently change tool chains.
- The warnings about beta go for release candidates too!
I am working on a v9 -> v10.10 checklist for airplane authors; the actual “busy work” of editing the ACF file should be less than 30 minutes per airplane if you know what new values you need to enter for parameters that need updating, so a reasonable work-flow might be to experiment with the new features and report bugs until we go final, then make the actual update on a fresh copy of your plane from an older version.
How Did We Miss This?
The sobering thing for me is that this bug has been in our betas for four weeks and (1) we didn’t notice it and (2) no one reported it. I take a few things away from this: clearly we need to test our own planes more carefully. In 10.10 we did a lot of detailed work on our own fleet, but ironically this hid the bug from us. But I also take away from this that a lot of authors don’t even look at the build until RC.
If you make an add-on, I encourage you to at least look at a few betas before RC. Even if you don’t retest every one, taking a peak early means we can fix bugs that affect your add-on early, which is good for everyone.
This post is a summary of what is going on with the X-Plane 10 starters. I am working on a more comprehensive X-Plane 10 aircraft update check-list that will include starters.
How the Starters Used to Work
In X-Plane 9.70 and 10.05r1, the starter motor applies a constant torque to the engine to increase its RPM. As you motor the starter without adding fuel and spark, the engine’s drag will increase (due to its higher RPM – pretty much all engines move air when turned and thus have higher drag at higher RPM) and eventually you’ll hit an equilibrium: the torque of the starter exactly matches the drag of the engine and you sit there at a constant RPM. This RPM can be quite high because the starter motor can deliver its torque at any RPM.
The torque of the starter is decided by X-Plane using a formula known only to Austin and a few highly trained monkeys that have secure access to the X-Plane source code. The starter “ratio” you set in Plane-Maker is a scaling factor on that ratio. A scaling factor of 2.0 doubles the torque, and a scaling factor of 0.5 halves it.
Note that the default torque (1.0 scaling factor) varies by engine type and engine size! The justification by this is that if you don’t want to have to deal with starters, you set the starter ratio to 1.0 and X-Plane’s “natural” choice is strong enough to start your engine. In practice, the default torque is surprisingly high for jet engines, but at least they start!
How the Starters Work in X-Plane 10.10 Beta 11
Pay no attention to X-Plane 10.10 beta 11 – the changes for RC1 are quite significant.
How the Starters Work in x-plane 10.10 rc1 (coming soon!)
X-Plane 10.10 rc1 is also a torque-based model, with the starter putting out constant torque. There are two key changes:
- The torque is expressed as a ratio of maximum engine torque, not a ratio of an arbitrary default torque. This change the numbering scheme! When you open your aircraft in 10.10 you’ll see the new numbering scheme. For example, the default torque in X-Plane 9 for a jet engine was 50% of engine max torque (see above that jets started really fast), so if you set a starter ratio of 0.6, the net result was 30% of max engine torque. In X-plane 10.10 you will just see 0.3. In other words, what used to be a ‘secret scaling factor’ is now baked in to your starter value.
- The starter now has a maximum design RPM; past this RPM its torque will fall off – very very quickly for electric starters, less so for bleed-air starters. The default will be 100% of engine RPM, so for old planes the starter will continue to motor up to equilibrium. You can turn this number down to model the real world: real starters generally can’t put out their maximum torque up to really high RPMs.
For piston engines with electric starters, I recommend setting the maximum RPM very low and the torque fairly high; a real electric starter for a piston engine is going to put out a ton of torque to get your engine to turn over, but it just can’t run that fast.
For turbines, make sure your design RPM is well above your fuel introduction point. From what a number of people have told me, the starter can often be motored quite a bit past the fuel introduction point.
The net of all of this is: no change from 10.05 for existing aircraft, but the potential to fix a number of weird starter behaviors in 10.10 by limiting RPM. (One advantage of the RPM limit is that you can increase the starter torque without getting an absurdly high terminal RPM.)
Airplane authors: prepare yourselves! I am now very close to closing the last of my fixable bugs filed against the 10.10 aircraft SDK. This means that the next beta will be the one to test your airplanes against. If there is a problem with your work in 10.10 and you tell us after 10.10 goes final, it may be very hard for us to fix it, so if you’ve been waiting on 10.10 betas, please go get the beta now and be ready to test shortly! If you have been waiting and letting your users try 10.10, start downloading now!
One of the main goals of 10.10 is to have the airplane authoring SDK provide similar visual results when HDR is on and off; another is to have these results remain the same for the rest of the v10 run.
Long Term Future Proofing
Lighting model changes represent one of the trickiest engine changes we can put into X-Plane. When done right they add realism, but they also fundamentally change how an author’s work looks. With 10.10, the v10 lighting model is “done” for the version run, but I don’t think it will be the final model forever. The v10 lighting model definitely contains some compromises to make translucency and v9-style authoring techniques work.
A question came up in the comments as to whether HDR mode with the deferred rendering engine and global lights might become the standard rendering mode for X-Plane 10. I think some day it may be – just as we used to have an option: pixel shaders or not, and now they are always on, I expect that most rendering innovations will eventually become standard over a long enough time frame as hardware continues to scale up.
(We have already seen this in first person shooters: the first ones to feature a deferred rendering engine often provided two rendering modes; now new games tend to be deferred-only.)
We certainly view the deferred engine and HDR mode to be preferred, and that’s why so many effects are HDR only. There are effects we can only do with a deferred renderer, and there are effects that are much cheaper to build into the deferred renderer.
Beyond X-Plane 10
If you are working on an aircraft and you’d like it to work well not only with X-Plane 10, but also X-Plane 11, here are a few things to avoid. These features are supported in X-Plane 10, but we see them as compatibility features, not “good things to use.”
- Avoid using the cockpit texture without real 3-d lighting. This happens when you use ATTR_cockpit without GLOBAL_cockpit_lit. The cockpit texture without 3-d lighting is still supported in version 10 but it is not recommended, and you can get some weird artifacts. Over time these problems will get worse (as the lighting model becomes more sophisticated) and a 3-d panel that doesn’t obey the lighting model may not even be possible after X-Plane 10. Fortunately you can use GLOBAL_cockpit_lit to turn on 3-d lighting and in almost all cases it will “just work”.
- Don’t use ATTR_diffuse_rgb and ATTR_emission_rgb. Instead, paint the color into your albedo and lit textures and use ATTR_light_level.
- Make absolutely sure that everything translucent is marked ‘glass’, and that nothing is translucent unless there is a really good reason. Alpha is really tricky for a deferred renderer! This is good advice for v10 as well.
X-Plane 10.10 beta 10 will be out soon, as well as more detailed instructions on airplane updates.
This rather odd 747 picture is from a quick test I did to make sure the shadow options in Plane-Maker were working right after beating the shadow code silly with a hammer. The wing objects have been marked “interior only” for shadows, and since we are in an exterior view…the wings don’t cast shadows. 🙂
Now this is a totally silly way to use the feature, but there is a legitimate use: mark as many of your interior objects as “inside shadow only” as possible; for example, in the 747 the interior passenger cabin object can be marked as interior only – it doesn’t cast meaningful shadows on anything outside the airport.
By marking an object as no-shadow in Plane-Maker you save the sim the work of drawing the object, which is good for fps. If your airplane is used for an AI plane, this makes AI plane drawing less expensive.
In fact, you save it the work of drawing it multiple times. X-Plane using a shadowing technique called “cascading shadow maps”. Basically X-Plane renders different parts of the world at different resolutions, so that the closer shadows (that can be seen in more detail on screen) have a higher resolution. The user’s plane ends up being drawn in a lot of these rendering passes, and as a result the cost of high-geometry-count objects in an airplane can be amplified several times over by shadows. So savings in object-shadow count matter!
(Given the choice of turning off shadows in Plane-Maker or via GLOBAL_no_shadow, use Plane-Maker; it stops drawing earlier and thus saves more CPU.)
