First, I appreciate everyone’s cooperation with the RFC on scenery; we’ve had an ongoing discussion in our developer Slack as well as the comment section, and I don’t think I had to nuke any off-topic comments. The feedback was wide-ranging and there’s no one clear answer but it does give us a really good picture of how the scenery system is working (and isn’t working).
It’s Friday, so let’s do something completely different – her’s some show and tell from a few things people have been working on things week.
Light It Up
Alex has been recalibrating the runway and airport lights for the new photometric lighting engine. This spurred an internal discussion about how best to calibrate artificial light sources. Does the author specify the luminance of the bulb before a tinted plastic housing goes on top (this way is good if you have the bulb specs from the internet) or based on what you’d measure when the finished light is tested? (This way matches FAA specs for airport lights.)
After going back and forth a few times, our answer is “well, both”, and we have a system that now allows this, which should solve use cases for both aircraft (where often the bulb properties are known because you can look up replacement parts) and for airports (where the FAA has standards for the light’s final results).
Something to keep in mind: urban airports are quite dark compared to their surroundings. Ther are very few light sources near the runway that aren’t tightly controlled for brightness and direction. I used to fly over KLAX on a regular basis at cruise altitudes (commuting from San Diego to San Francisco for work) and KLAX was always an inky black void in the sea of lights that is the LA basin; at 34,000 feet no runway lights are pointed up at us.
Wet Surfaces
Petr and Sidney have been working on the weather surface shader, which applies water and other weather effects to surfaces. This is how we dynamically make the pavement wet when it rains.
The shader is tricky because the effect of a surface being wet changes a lot once the water forms a real puddle. When I took my kids to their swim lesson, I couldn’t help but notice the useful reference material all over the place.
A rough wet material – reflections change with angle in X-Plane and real life
Stop Writing on the Windows
I must be a dad, because I get annoyed when my kids get finger prints all over the windows when they “write” things in the frost on a cold day.
Turns out Sidney does the same thing.
What you’re seeing there is programmer art. Programmer art is when the programmers make their own texture files to test code. In this case, Sidney is testing the defrosting system for windscreens, which use a special texture to specify the pattern of defrosting. This lets artists control the defrosting effect and get faster defrosting near vents.
Another “behind the scenes” thing you can see here: that popup window is a set of internal controls for testing, debugging and developing the windscreen effects. The parts of these internal controls that are generally useful will become third party developer tools (like the texture browser and particle system editor in X-Plane 11).
Cessna In Spaaaaaaaace
Daniel rewrote the planet shader. In X-Plane 12, water is treated separately from land (so that it can be 3-d). The new planet shader shows a far view of water and a far view of land at the same time and correctly shows atmospheric scattering, which is normally pre-calculated in a special “froxel cache” for regular scenery.
If you haven’t noticed the pattern, it’s that the art team’s screenshots all tend to look good enough to ship, and the programmer’s screen shots tend to be very, very silly. In this case, the Cessna in space is pretty silly, but what we were looking for was the smooth atmospheric effects all the way out to the horizon.
Here’s one more goofy programmer screenshot:
I was calibrating the runway lights according to Alex’s spec, and typed an extra 0 into one of the internal art controls by accident. The result was this fantastic screen shot.
What you’re seeing is: the billboards for the runway environment are accidentally huge and are filling up the entire reflection cube map. The reflective underside of the Cessna wing picks up this blue lights and it looks like a rave.
[This post is a “behind the scenes” look at the tech that makes up the X-Plane massive multiplayer (MMO) server. It’s only going to be of interest to programming nerds—there are no takeaways here for plugin devs or sim pilots.]
In mid-2020, we launched massive multiplayer on X-Plane Mobile. This broke a lot of new ground for us as an organization. We’ve had peer-to-peer multiplayer in the sim for a long time, but never server-hosted multiplayer. That meant there were a lot of technical decisions to make, with no constraints imposed by existing code.
Amazingly, I think the answer (and I realize I am cursing the next beta by typing this) is we’re getting really close. You’d be forgiven for thinking that’s lunacy given beta 15; here’s a little bit of info about the state of the betas.
X-Plane 11.50 beta 15 is out (and marked “unstable”) on Steam and amongst other things, has two big fixes for beta 14:
Cloud performance should be back to baseline norms for 11.50 – when I fixed a multi-monitor bug in beta 14, I accidentally turned off a major cloud performance optimization,
Fatal errors while resizing the window should be fixed. These squawks were coming from code that now checks much more heavily for error conditions, and revealed a problem when the OS delivers window resizes to us too slowly.
Note the introduction of new bugs in the process of fixing old ones – this is always a risk when a bug fix is intrusive or complicated; in order to get a shippable product, we have to keep ratcheting down the amount of chaos we introduce per beta, making smaller and more surgical changes.
Beta 15 also tried to fix one last plugin compatibility bug that we discovered very late in the game – under Vulkan, there’s no depth/stencil buffer available to third party plugins, resulting in incorrect drawing for previously working plugins.
I think you know where this is going…this new change introduced new bugs, even as it fixed the problem with the original add-on.
We have been working with third party developers over the last twenty four hours on fixes for the regressions here; our hope is to have a new beta on Monday or Tuesday that fixes these issues and has gotten a good third party once-over.
We’re Going to Have to Close the Door
We are reaching the end of X-Plane 11.50 – at this point the number of remaining bugs to be fixed in this beta is small enough that they aren’t that hard to keep track of. To get to a release candidate though, we’ll need to stop introducing major changes.
I am hoping that we already know about all of the third party incompatibilities, because at this point we have to close the door to complex changes to improve backward compatibility. For beta 16 we are fixing what used to work and is now broken (e.g. yes, SkyMaxx will work again), but if anyone is sitting on an add-on problem they haven’t mentioned, we’re out of time to deal with it.
Plugin Developers: Thread Safety!
Plugin developers: in looking at your compatibility with X-Plane 11.50, please re-read this post and check your usage of threads. We’ve had really helpful responses from third parties who we have notified about threading issues we’ve seen by auto-crash reporting, and I think this will help everyone–third parties, users, and us.
I believe that X-Plane 11.50 is significantly more sensitive to threading violations than 11.40, because the Vulkan driver doesn’t spend CPU cycles protecting itself from abuse. If a plugin calls into us from a thread when it’s not allowed to, this can cascade into crazy X-Plane behavior, which cascades into crazy Vulkan behavior that the driver won’t stop. So careful adherence to the threading rules by plugins is critical to Vulkan stability.
With Vulkan and Metal, X-Plane is now firmly in the driver’s seat for VRAM management. This lets us eliminate stutters that were previously present with OpenGL and almost impossible to avoid. It has, however, one big and noticeable downside: when you run out of VRAM, you get blurry textures.
Of course the goal wasn’t to replace stuttering with blurry textures, and we believe that given the normal work load of X-Plane, you should not be seeing this. The fact that so many users are seeing blurry textures, especially on big cards with lots of VRAM, points to the VRAM code being buggy in all the ways beta code can be buggy.
How much VRAM do I need?
Just to get the obvious out of the way, our system requirements have not changed for 11.50. Our minimum VRAM requirement for X-Plane 11 is still 1Gb. We expect these cards to be able to run 1080p with lowered texture resolution, providing an equal experience to X-Plane 11.41. With 2Gb we expect users to be able to run HDR with medium texture resolution on 1080p systems. 4Gb and higher should be able to run HDR at with high resolution textures with 2k monitor resolutions. With 6Gb and higher, 4k monitor resolution should be possible.
A week or two ago we had a very dead beta, and posed the question of how to incrementally test betas in the future. We got a variety of responses, ranging from “private test it first” to “roll it out in a wave” to “full speed ahead, we know betas are bumpy.”
Since then, we’ve been doing one of the easiest and probably most useful things we can: posting the betas early to third-party developers who are in our developer Slack channel.
Beta 7/8 had a ton of changes, and our third-party developers found multiple problems, some of which we wouldn’t see in our internal tests. So we held off on releasing betas 7 and 8 to the public while we fixed those issues. Until today.
There are a lot of changes in this one since it’s kind of 3-in-one. Please see the full release notes here.
Are We There Yet?
X-Plane 11.50 has been similar to X-Plane 11.20 (our VR) release and different from what we normally try to do, in that when we went beta (both private and public), the work for Vulkan wasn’t done yet. We had something that you could fly with, that was delightful for some users (and unstable for others), but we also had a big list of things we still needed to do.
This post is just targeted at plugin developers who are modernizing their object drawing – if you don’t write plugin code, the Cincinnati Zoo has been showing their animals on Youtube – it’ll be a lot more entertaining than this post. (An XPLMInstance cannot tunnel down two feet in fifteen seconds – one point for the zoo animals.)
XPLMInstance makes a persistent object that lives inside X-Plane that is visible in the 3-d world. It changes how you draw from “run some drawing code every frame” to “tell X-Plane that there is a thing and update its data every now and then.”
Instancing is actually a lot easier than draw callbacks! But there are two tricky gotchas:
1. You must create the custom DataRefs for your OBJ’s animation before you load the object itself with the SDK. (If the DataRefs do not exist at load time, the animations are disabled as “unresolved to any DataRef”.)
2. When you create the instance, make sure your custom DataRefs are on the list of DataRefs for that instance.
Here’s the really baffling thing: if you create the custom DataRef and then add it to the instance’s list, your DataRef callbacks will not be called.
Wha?
Here’s the trick: the DataRef you register is a global identifier, allowing the object to refer to what it wants to listen to. That’s why you have to create the DataRef – so that the identifier exists.
But when you create an instance, each instance has memory that holds a different copy of those DataRefs.
For example, let’s say you have a truck with four DataRefs, and you make five instances. X-Plane allocates 20 slots (four DataRefs times five instances) to store five copies of each DataRef’s values.
The instances never look at the DataRef itself. They only look at their local copies. That’s why when you push different data to the instance with XPLMSetInstancePosition, each instance animates with its own values – each instance looks at its own local data.
This is also why you won’t see your DataRef callbacks called (unless you use DataRefEditor or some other tool). The object rendering engine isn’t looking at the DataRefs themselves, it’s looking at the local copies.
In other words, XPLMInstance turns DataRefs from the pull model you are used to (X-Plane pulls on your read function to get the value) to a push model (you push set with XPLMSetInstancePosition into the instance’s memory).
This implies two things about your add-on:
It doesn’t really matter what your DataRef read functions do – they can just return zero, and
You can’t use tools like DataRefEditor or DataRefTool to debug your animations. (That didn’t work well in legacy code either, but it really won’t work now.)
If you try the obvious optimization of not creating your custom DataRefs (“hey, no one calls them”) before you create your instance, you will find that animation just stops working. This is because we need the DataRef to be that global identifier to match your instance data with the animations of the object itself.
One last note: if your old code used sim/graphics/animation/draw_object_x/y/z to determine which object was being animated (from inside a plugin “get” function) you do not need to do this anymore. Because each instance has its own local copies and your DataRef function isn’t called, this technique is obsolete.
In summary:
You must register custom DataRefs.
Their callbacks can just return 0 – they’ll never be called.
Always list your custom DataRefs for animation when you create an instance.
Do not use draw_object_x/y/z; use XPLMSetInstancePosition to create per-specific-instance animation.
I was going to write a post about X-Plane 11.50 beta 5 – what’s new in it, the new ways we are debugging GPU crashes, the crash bugs we’ve fixed, etc. A lot of stuff that we thought was pretty good went into beta 5. Cool new technology! Big bug fixes! Lots of winning!
As it turns out, beta 5 is dead. I hit “go” on the release this afternoon, and half an hour ago, I hit “stop.” The auto crash reporter was showing way too many new crashes in memory management that we had not seen before, and this strongly implies a new and serious bug.
Laminar Installer Users: if you were auto-notified to update to beta five and did so, and you are not crashing, you can keep flying! If your beta five is just a smoldering wreckage of crumpled VRAM and GPU parts, you can re-run the installer with “get betas” option checked, and it will take you back to beta 4.
If you were not auto-notified to update to beta five, that’s probably for the best. Please stand by and keep flying beta four; we’ll post a new beta when we’ve gotten to the bottom of this. We have enough captured crash data to investigate.
Steam Users: we did not release the beta five build to Steam and this is probably a good thing; we’ll try again with a new release that isn’t made of plutonium and unicorn hallucinations.
And if you’re going “why didn’t y’all test it before you released it”…we did! None of our machines show these crashes. But we also have probably a dozen PCs total we can run on. Moving to a new driver stack has meant learning about the weird things that happen on your computers and not ours.
Do We Need a Two Tiered Beta System
This came up in our impromptu beta five post-mortem meeting: do we need to bring people into new betas in stages? With code for new drivers, beta five probably won’t be the last beta where we code something we think is helping and discover that it fails catastrophically, but not on our hardware. We need beta victims^H^H^H^H^Htesters to find these bugs, but once we get a dozen crashes, we don’t need anyone else to stub their toe for us to fix our problems.
So we thought about two possible ways to do this:
A two-tiered system. Early adopters could get an email and hand-update to the new beta before it is put out for auto-update notification.
Send out the beta update notifications over time, e.g. 10% of users get notified immediately, then another 40% if we don’t see crashes, then the last 50%. (This practice is actually industry standard on mobile apps.)
If you are reading this blog post, this far down, you are probably participating in the beta; I’d be curious what approach you’d find most useful.
We were discussing a particularly exasperated sounding bug report on one of the internal Slack channels when I realized that this might not be obvious: a crash with the error message “pipeline must not be null” – it’s one error message that covers a whole category of bugs. We fixed one major case (skycolors were broken) in b1 and added one major case (custom billboard lights on aircraft) in b2 – conservation of pipeline bugs!
Null pipelines are a new category of crash in X-Plane 11.50, so here are a few notes on what this error is and what you can do to help us fix them (and what you don’t need to bother with).
What Is a Pipeline?
A pipeline is just the Vulkan and Metal term for a shader (plus some extra gak (1)) that we use to do our drawing.
X-Plane 11.41 would ask the OpenGL driver to build shaders as it needed them, and then the driver would turn those GL shaders into hardware pipelines on the fly as it got presented with different scenarios.
Not 11.50. We build everything up front. Vulkan has two rules:
Using a pipeline is fast.
Building a pipeline is not fast.
This is a great pair of rules for us – it means if we build our pipelines at load time, we are not going to have stutters mid-frame.
Why Are We Crashing?
There is one down-side to the 11.50 way of doing things: if we don’t build all of the pipelines we need up front during load, then when it comes time to draw, we’re toast. That’s what a “pipeline must not be null” error is – it just means the loading code did not create the pipeline the drawing code needs.
Why not just build every pipeline we could ever possibly need? Load time. X-Plane can build hundreds of thousands of pipelines depending on rendering settings, scenery packs, custom aircraft, etc. We actually did “just build everything” early in our development process and the sim could take half an hour to load.
So we try to build only the pipelines we need. If we build too many, we slow load, and if we build too low, you see this error.
What Do You Do When You See This Error?
On Windows and Linux, it’s really easy: close the alert box and when the auto crash report form comes up, please press “send”. Don’t bother with you email or a message; everything we need to kill this bug is already in the auto report! (Jennifer’s edit: please DO include your email address with any auto report if you want us to be able to confirm we have your specific report! This is the only way we have of identifying who it came from.)
The good news is: the auto crash reports for the pipeline crashes are insanely easy to find and fix.
Mac users: if you see one of these, we need the Apple crash report – please send it in a bug report.
(1) for the plugin developers that know some OpenGL: a pipeline is basically a GLprogram (shader) plus a bunch of the fixed function state that goes with it: blending, depth/stencil, vertex format, FBO format, and some rando stuff thrown in.
The idea is to have the pipeline contain so much information that there is no risk that the driver has to build two hardware shaders for one Vulkan shader (to cope with other fixed function state) no matter how weird the hardware is.
On lots of actual hardware, the pipeline has stuff that’s not actually in the shader, but some surprising things, like vertex format, actually often are.
We have updated the X-Plane plugin SDK and related documentation for X-Plane 11.50. Here are some links:
New Plugin SDK: version 3.02 adds the new modern 3-d drawing callback for OpenGL/Vulkan, and deprecates the drawing callbacks that are unavailable in Vulkan/Metal. Download it here.
(Updating to the new SDK is not mandatory for your add-on to be Vulkan-compatible, but it can help catch drawing callbacks that will not work, and it is necessary to use the new 3-d drawing tech.)
Plugin Compatibility Guide:a short guide that focuses on issues existing plugins might have with X-Plane 11.50. If you are updating a plugin, read this!
OpenGL Drawing Guide:complete documentation for all aspects of drawing using OpenGL with X-Plane. If your plugin uses OpenGL, this is a must-read.
Drawing Over 3-d in 2-d: this sample code shows how to draw in a 2-d window or drawing callback with coordinates that match the 3-d world. Many 3-d plugins that draw need this kind of drawing, e.g. to label routes or aircraft with UI that matches the 3-d world.
Instancing Example Plugin:a sample plugin that draws objects using the XPLMInstancing APIs. Many add-ons will need to transition from object drawing to instancing to be compatible with Vulkan. We strongly recommend moving to instancing – it provides the fastest, most compatible drawing path, particle system support, and in the future will support FMOD sound.
Instancing is actually much easier to use than XPLMDrawObject and bypasses a lot of complexity and chaos. Instancing works with all versions of X-Plane 11, and the sample has everything you need.
X-Plane 11.50 has been out for a little bit more than 24 hours, and things have been a little bit nuts. Here are a few quick notes, in no particular order.
Bug Fixes and Work Arounds
While I don’t have work-arounds for the missing right eye in VR or older NVidia cards that won’t run in Vulkan, the good news is that we have fixes for these already. We are going to start testing beta two on Monday and try to get the fixes we have out as soon as possible. While we don’t have every major reported bug fixed, beta two should make a real difference.
Users who can’t start and have SLI setups: disable SLI in the Nvidia control panel and you will be able to run Vulkan. We are still investigating this – our goal is a bug fix so you don’t need to turn SLI off. (We do not expect X-Plane to leverage both cards – we expect it to run without failing.)
Finally, one thing I should have mentioned in the announcement: if you have scripts that modify art controls, please remove them, and don’t put them back.
The art controls are undocumented and subject to change, and they have changed a lot since X-Plane 11.41. Realistically the authors of these tweak scripts need to go back and re-evaluate every one of their tweaks in 11.50 to see if they actually help or are actually making things worse.
This is not like plugins or scenery; we tell you to get a second installation for the beta not because we want you to run add-on free but rather so that if the beta fails you are not locked out of X-Plane. We expect add-ons to work and we are taking plugin, scenery and aircraft bugs seriously.
By comparison, the art controls are “do what you want, but you void the warranty if you mess with this.” If you are running scripts that hack the art controls, we cannot tell the difference between real bugs in the early betas and art controls screwing things up.
The Road Map For Betas
Looking over the bug reports we have received, I think we are going to take on the 11.50 bugs in three phases:
Stability and compatibility. We’ll start by making sure that we run Vulkan and Metal on every platform that should be able to run them, with add-ons just working in the cases where we expect them to. We’ll start by focusing on fixing crashes, black screens, device lost, unstable plugins, etc.
VRAM use. We’ve received a number of reports that make it sound like VRAM management is not working properly. Once we can run, we’ll dig into blurry textures, running out of VRAM, etc. Sidney has built some great tools to get a good picture of how VRAM is being managed. VRAM management is one of the newest and most complex parts of 11.50 so it isn’t surprising that we’ve seen things that look buggy.
Performance. Once we are running where we should and using VRAM that we should, we can look at the cases where users are not seeing performance benefits from Metal and Vulkan, as well as remaining stutters. Once again, Sidney has built some fantastic tools that should help us dig into this quite efficiently.
This is the only order that we can reasonably approach the bugs. If the app won’t run on all qualifying hardware, we can’t test our VRAM use everywhere, and if our VRAM use isn’t correct, it can bias performance testing.
