This post is a request for commentary (RFC) – that is, it’s the beginnings of a discussion on the specific topic of mesh and overlay scenery packs. A quick note on moderation: unlike normal blog posts, I’m going to kill all off-topic comments for this post. We’ll have non-RFC posts and the discussion will be more free-wheeling, but in this case the goal is to have the comments for the post really flesh out possible solutions to one specific problem.
With that in mind, our topic of discussion: how best to separate mesh and overlay scenery packs?
New Scenery Packs Can Cause Chaos
Here’s how scenery packs work now: all scenery packs in “Custom Scenery” out-rank all of the scenery packs that ship with X-Plane. Except we put some of our scenery packs into “Custom Scenery”, so that strict “customizations bypass default rule” is already a bit broken.
Within the custom scenery folder, the scenery_packs.ini file defines the priority order of packs (and can bypass packs). When the sim runs, new packs discovered at startup that aren’t in the .ini file are added to the front in alphabetical order. So “newly installed wins” is the effective policy.
Here are some things that go wrong:
When a new mesh scenery pack is installed, it goes to the top of the priority list, hiding default Gateway airports below it. Custom mesh authors often want the latest Gateway airports to “show through.”
If a user manually reorganizes their scenery packs, they need to keep custom overlay airports above the default Gateway airports but custom meshes below the Gateway airports. If the user just puts the Gateway airports at the top of the list, custom overlays get replaced.
We regularly see user logs with 500-1000 custom scenery packs, so while I think a nice UI to organize pack priority might be nice, I don’t think it solves these problems. Telling users “go rank 1000 random items in priority order” is impractical.
Separate Custom Overlays from Custom Meshes
So my first naive idea is to simply have two custom scenery folders: one for overlays and one for meshes. Every pack in the overlay folder would completely outrank the meshes folder. This idea begs a bunch of implementation questions:
What happens if a custom scenery pack is put in the wrong folder (e.g. a mesh in the overlay folder or overlay in the mesh folder)?
Where do Gateway airports go? Do they live in custom overlays (and if so are they always ranked last)? Or do they get buried somewhere inside Resources and always rank between overlays and meshes?
Where do library packs go? Library use isn’t mutually exclusive with overlays or meshes.
Is either folder “Custom Scenery” for compatibility? If not, do we simply have no “Custom Scenery” folder, or does that break too many add-ons?
One of the strengths of this idea is that it’s really dumb and simple, and sometimes when the problem is “we have a complex mess,” simple and easy to understand is better than “really clever.”
Automatically Prioritize Packs Upon Discovery
An alternative to two install locations would be to have X-Plane determine the pack type upon first discovery and then rank it in the priority list in the middle if it’s a mesh.
If this were to work, the win would be that it would “just work” with no changes for third parties or users. However, I think in practice it may not be practical – it would be a lot of file sniffing by X-Plane at startup, and the categories of mesh and overlay are a little bit fluid. If a pack contains an overlay and a mesh DSF, how do we categorize it? If the scenery packs are in some scattered order with meshes on top of overlays, where do we put the new scenery?
Author-Selected Rankings
Another lever we can pull is to allow scenery authors to annotate their packs (perhaps with a text file or new library.txt directive) indicating the appropriate installation ranking for the pack. This approach would be similar to automatic prioritization, except priorities would be explicit and cheap to discover.
Authors would opt in; some kind of default behavior would have to be defined for legacy packs. Some open questions:
How would authors define where they want their packs installed relative to other packs?
Would users still be able to customize ranking? If so, would weirdly-ordered packs make it difficult to prioritize new packs?
Author-Selected Sub-Folders Within Packs
This idea came up during some discussions in the third party developer Slack channel: we could introduce a scheme within scenery packs to allow a single scenery pack to include both base meshes and overlays. X-Plane would automatically load all overlays from within these packs first, then global airports, then all base meshes. There’s some nice wins here:
This scheme puts the burden of correct organization on authors, not users, which is better for support load for third party authors – third party authors already need to know how the scenery system works in detail.
This scheme solves a related problem: packs that contain a base mesh and a few custom airports can now be distributed as a single pack rather than several packs that are each installed separately.
Categorization of packs is cheap, as it is simply based on file location.
Thomson and Philipp are at FlightSimExpo in San Diego; at 2 pm PDT (5 pm EDT) he’ll be showing our video presentation. You can watch it here or on our Youtube channel.
Back in June, Ben shared some information regarding X-Plane’s next generation lighting pipeline. It’s now time to go back to the future and talk about another piece of the next generation: Vegetation. In case you have missed it, Chris and Thomson made an awesome preview video for this as well:
If you’ve been experiencing frustration with the airport locking mechanism on the Gateway, it’s probably my fault! The logic behind this feature was changed recently and it kinda caught us out. All of this deserved an explanation, and some further tweaking from us to make things better.
Background
To protect against two artists working on the same airport, some time ago we introduced the capability to ‘lock’ an airport on the Gateway (for the duration of its development cycle). Originally this feature was unlimited, but was later revised to restrict the number of simultaneous locks that could be taken.
Recent changes
The original locking mechanism was better than nothing, but not much. There was no intelligence behind it, that is to say, nothing to prevent an artist from holding an airport indefinitely (by releasing and renewing the lock) and no taking into account previous submission history for a given artist. As a result, we sometimes heard from frustrated artists that a certain high profile airport was being held for a silly long time by a newbie. Another source of frustration was not being able to lock enough airports to keep busy, a direct result of us trying to solve an earlier problem that artists were occasionally holding too many airports!
So, we brainstormed a new design that took into account each artist’s skill set and submission history. Higher skill sets = more airports that can be locked, more and longer extension periods. Locking an airport and then abandoning it without making a submission = fewer airports that can be locked, fewer and shorter extension periods. I signed off on the new design and Daniela (as always) did a beautiful job coding it to the exact specifications.
Shortly after we went live there was a noticeable drop in Gateway submission levels. This sometimes happens, and I attributed it to the normal ebb and flow. However, it has since become apparent there is more to this. The new ‘locking’ mechanism that was supposed to make things better and fairer was having the opposite effect. At first I could not understand this–surely artists don’t need to lock more airports than they can simultaneously work on? But of course this is wrong! The problem is that, during the wait period between submission and moderation, the airport must remain locked, and hence an artist potentially needs to hold onto a bunch of airports that are ‘in limbo’. No doubt everyone in the community already knew this, but I failed to take it into account when signing off on the new logic.
The Solution
The solution is to not tally airports that are waiting ‘in limbo’. This means as soon as you submit your airport, it is no longer tallied against your lock total (but is still locked in an absolute sense, and therefore cannot be claimed by another artist). One thing to be aware of – when an airport becomes approved or declined, it ceases to be in limbo, and WILL be tallied against an artist’s lock count. It’s important therefore to manually release locks that are in effect, but no longer needed.
If you’re reading this, it’s already done. Things should be a little better!
It’s been very busy here internally, but a few things to mention.
X-Plane 11.55 – Crash Fix
We’re focusing almost all of our effort on future technology, but Stairport reported a bug severe enough that we went for a patch: X-Plane would randomly crash when plugins use the new “instancing” drawing APIs.
The instancing APIs are the Vulkan-compatible way to draw objects from plugins, the way to add particle effects via plugins, and will someday support sound as well. Simply put, instancing is meant to be the foundation for plugin-created dynamic content. Third parties did a great job of switching to instancing to be Vulkan compatible when we released X-Plane 11.50, so having this API be rock-solid is really important.
With X-Plane 11.55, correctly written add-ons should “just work.” The interaction between instancing and datarefs does sometimes confuse developers, so I’ll cover that in some nerdy detail in a future post.
The Latest Gateway Airports
While we were cutting a hot patch, we took another set of airports from the X-Plane Airport Scenery Gateway – 11.55 features over 1000 new 3-d airports and 443 brand new airports. I remain amazed at the Gateway community’s progress and results.
I’m excited to finally be able to talk about something I’ve been working on for a while now – the new photometric lighting pipeline. Here’s the preview video Chris and Thomson made:
X-Plane’s lighting and rendering have leveled up several times – in X-Plane 10 we moved to HDR with global lighting, and in X-Plane 11 we introduced Physically Based Rendering.
I know this kills, but it’s too soon to talk about release dates. I can say a little bit about what you’re seeing in the video though.
First, the new lighting pipeline is photometric. What that means is that color values during rendering match real world values (in real world units) through-out the entire rendering process. Rather than say “1.0 is a bright thing, and, um, 4.0 is a really bright thing”, with photometric rendering, there are real answers. The sun is about 120,000 lux. The blue sky might be 8000 cd/m^2. A landing light on the 737 might be 200,000 candela at its peak intensity.
The idea behind photometric rendering is to have all elements of the scene be calibrated to real world values so they all match each other. That cloud should match that sky and that airplane because they’re all in the same units. No more tweaks to try to make things match.
We shipped an HDR renderer years ago, but the new pipeline is, well, more HDR. A lot more HDR. Because we’re working in real world units, we have to maintain a wide HDR image from beginning right to the very end when we tone map. The result is that every part of the scene can have a wide dynamic range.
While our shipping pipeline dims displays during the day by darkening them (to give the appearance of wash-out), the new pipeline simply draws everything in real world units – the camera is simply set for day time exposure (set in real EV units like a real camera) and the displays look dim due to the camera.
The new pipeline features a new tone mapper – one thing you might notice from the videos is that color with the new pipeline are richer. This is partly because the new tone mapper (which works well with real-world illumination values and is HDR-display-ready) does a better job of preserving saturation.
Sun and sky colors in the new pipeline are driven by a new atmosphere and sky simulation – they’re not painted textures. We get the sun color from the composition of the atmosphere and the relative position of the sun and scenery.
Finally, the new pipeline can run screen space reflections (SSR), dynamic exposure, bloom and other effects (still to be previewed), all running with real world photometric HDR values.
Why Photometric Rendering?
The photometric renderer gets us a bunch of visual quality improvements and new effects that were high on our TODO list (and, based on the feedback site, yours too.)
Photometric rendering also serves as a foundation for a bunch of other features that are high on our priority list. That will be a topic of a future preview and a future blog post.
X-Plane can be configured to model real-world weather conditions. It does this by downloading weather report data from a server, operated by Laminar Research. That server, in turn, periodically fetches three total files from two different sources at NOAA: one for METAR-format weather condition data, and another source for two grib2-format files describing, separately, winds and turbulence. Internally, we tend to refer to this system, overall, as Live Weather, or, in the context of anything server-related, just Weather.
Around March 22nd, it broke down.
What went wrong
As is often the case: several things.
X-Plane began reporting that it couldn’t fetch all the data it needed for real-world weather. This was a little surprising since our weather server is supposed to cache the latest data it received, even if it’s outdated, to smooth over any connectivity or availability issues with NOAA’s servers. Further investigation revealed that only one of the three files it needed was both missing from its local cache, and unavailable from NOAA.
The root cause of that data being absent from cache hasn’t been tracked down, for the reason that that part of the system has been replaced (see below). There may be a bug in the caching logic. The server may have suffered a silent crash-and-restart some time after NOAA stopped serving the file. We’re not sure.
Investigation into the causes of the missing data was sidetracked for a little while because, right around the same time, NOAA happened to suffer a widely-reported, significant disruption to their networks and server operations. Naturally, we initially assumed this was the cause of the problem, and our efforts were mainly aimed at mitigating the harm of a temporary outage, not dealing with a longer-term problem.
In fact, what had happened was a longer-term problem, and the outage was, at most, only part of the cause of the disruption we were seeing.
In which we miss a NOAA systems upgrade
Effective on or about March 22, 2021, beginning with the 1200 Coordinated Universal Time (UTC) run, the National Centers for Environmental Prediction (NCEP) will upgrade the GFS and Global Data Assimilation System (GDAS) from version 15.3 to 16.0.
NOAA Service Change Notice 21-20 (Updated)
What that means is that X-Plane’s Live Weather system was about to break.
The system mentioned in that notice is the one that provides turbulence and wind data for the Live Weather system. These are the grib2-formatted files mentioned above.
Some time into this minor crisis, we noticed that both, not just one, of those files were missing—one had been properly cached, as intended, so it’d initially escaped our notice—and that the directory structure and file naming conventions on the NOAA server from which we source them had changed. Then we found the PDF containing the notice quoted above. Nestled deep in that PDF was the line:
Remove WAFS blended product at 1.25 deg
This refers to the file we used for turbulence data. The wind data we’d been using we could still find, albeit in a new location, but here was confirmation that the turbulence data we’d been using was simply gone.
In which various solutions fail
Our situation at this point was: we had found our wind-data grib2 file, and we had several WAFS (World Area Forecast System) files that looked, judging solely from their names, like they might contain the turbulence data we needed.
None of this was true.
We learned that the NOAA system upgrade had included an upgrade to the compression used on their grib2 files. X-Plane couldn’t understand it, and in internal testing actually crashed when attempting to use these files. That meant that though we had found a file containing the same data we needed for wind, we couldn’t actually use it.
Also, none of the other WAFS-related files contained the same turbulence data we’d been using, nor anything readily translatable to same.
So much for the easy solutions.
Fortran: friend or foe?
NOAA provides a program called cnvgrib to convert grib files to other sorts of grib files—for instance, as one might guess from references above to grib2, there is a grib1 that predated it, and cnvgrb can turn one into the other. It can also, most usefully to our situation, change the compression scheme used by grib2 files, so it should let us turn these new grib2 files into something we can use without having to modify X-Plane itself. Missing turbulence data aside, at least we could convert the wind data grib2 files to be usable.
Fortran is a… let’s say venerable programming language, like, for instance, COBOL. It’s a bit obscure these days, outside certain very long-lived systems and niche applications.
Cnvgrib is written in Fortran, and no recent compiled binary version is available. The Fortran (with some support from C) source code is provided by NOAA, though, along with instructions for compilation.
One Linux VM, a couple hours, and 17GB(?!) of Intel compilers and tools later, and we have a compiled copy of cnvgrib. As hoped, it can turn the new-compression grib2 files into old-compression grib2 files that X-Plane can understand.
There’s still the matter of the turbulence data, though.
Enter the Matrix
I skipped ahead a little. At this point in the troubleshooting process we hadn’t yet noticed that the turbulence data from NOAA wasn’t the same as what we’d been getting before, and once we could convert it to grib2 files that X-Plane could handle, it even looked like it was working.
So we deployed it.
By “it” I mean a very low-tech shell script that periodically downloads the two files we need from NOAA and uses cnvgrib to make them usable by X-Plane.
90s legible-clothing Geek Chic slogans at their finest. Also, I did actually do this to about 2/3 of the existing weather server.
Though this apparently worked, we quickly realized something was very wrong when support began fielding reports of way-too-powerful wind in X-Plane, when using the real-world weather feature. This was caused by X-Plane applying inappropriate turbulence modifiers to wind effects, because we were feeding it (from its perspective) bad data.
Having no new source of the turbulence data we had been using, and with the new data causing serious issues, what could we do without cutting a new release of X-Plane?
As it happens, we’d snagged exactly one of the old WAFS-Blended files, containing the kind of turbulence data X-Plane could use, from the NOAA servers, before they dropped off forever.
Yeah. It is.
The clear stop-gap solution, then, was to simply serve the working, reasonable turbulence-data file that we’d saved. All the time. Even though it’s increasingly outdated and definitely not “Real-World” anymore.
That’s right: so far as turbulence data goes, if you’re playing X-Plane with real-world weather turned on, you’re trapped in an old facsimile of the real world. You’re in The Matrix.
Fortunately, the effect of turbulence is minor enough, overall, that this doesn’t throw things off too badly (most of the time) and having some outdated real-world data is better than having none. Still, if you notice things behaving just slightly strangely—or if you have a sense of deja vu with real-world weather turbulence while flying—that may be why. The Matrix has you.
Are we going to keep lying to X-Plane about turbulence forever, then?
Nope! We’re working on translating other forms of turbulence data into something usable by existing copies of X-Plane, and have plans to make future versions of the Live Weather system more robust so that, hopefully, it won’t fail again—at least not in the same way.
Every week for the last ten weeks I’ve thought “I should really write a dev blog post” and then…not done that. This isn’t because all is quiet on the Western front – on the contrary, everyone on the team has been really, really busy, and the dev blog is never the loudest thing shouting for attention. But now we have a new RC available, so here we are.
Mi Memory Es Tu Memory
11.53 fixes one bug, and it’s a rare bug, but it’s “exciting” when it happens. It turns out that if a Lua plugin requests a really huge amount of memory, instead of saying “no,” X-Plane gives the Lua program someone else’s memory. This is not good! If the bank gave you someone else’s money, that’d be a bad bookkeeping error. This bug is too, and the consequences of the bug are typically “really insane stuff happens later,” which is hard to sort out. The plugin that crashes may not be the plugin that requested the memory.
X-Plane 11.53 fixes this – large allocations that cannot be fulfilled are denied, which should cause the Lua plugin to halt the affected script without destabilizing the system.
Script authors, if you’re wondering “now can I allocate a lot of memory in my Lua script,” the short answer is “no.” The longer answer is: when your Lua plugin uses a new version of LuaJIT that can use 64-bit addressing, this limitation will go away via a new plugin, without a change to X-Plane. Since the limitation is in LuaJIT, it’s out of our hands.
G2 Controller Support
Since we were doing a bug fix release, we have included support for the HP Reverb G2. For reasons I don’t fully understand, controller support didn’t “just work” in 11.52, so we had to create a new profile.
G2 users should be able to use their controllers with X-Plane 11.53. However you should also read our KB article for any additional issues with controllers, especially with misalignment. This version also includes a CLI option to adjust this if needed.
Tyler Has Left the Building
After almost seven years, Tyler has joined the ranks of Laminar Research alumnae. You may know him from such hits as:
The X-Plane 11 User Interface
X-Plane Mobile’s global scenery
X-Plane Mobile’s mass multiplayer
He will be missed! It took several weeks just to figure out all the things he maintains.
We Need More Jims
A few weeks ago, we posted a developer opening – I am pleased to announce Jim Keir as the newest member of the X-Plane development team. Jim is already fixing our screwed up code contributing bug fixes and learning the insides and outs of X-Plane’s almost 1 million lines of code. Jim brings our count of Jims up to two, which is still less of a namespace collision than our three Dan*s.
Multicore and Plugins
Most of what we are working on is still in the lab and hasn’t escaped yet. A few weeks ago we did have a discussion with developers in our third party developer Slack channel about multi-core and plugins.
The short story is this: in X-Plane 11.50, Sidney added a widget to the plugin admin window that shows how much main thread time they’re consuming, which in turn reveals how much each add-on is impacting FPS.
Plugin authors responded! Lots of plugins moved their CPU processing time to a worker thread. This is mostly great – other cores tend to be underutilized on high-end machines so this gets us more FPS.
Here’s the concern: a lot of plugins are doing this, and they are each moving work to other cores in their own private way. There is no coordination between plugins, and one day we are going to wake up and X-Plane will stutter because plugins were (just for a frame) using all of the cores and leaving too few for X-Plane itself.
We are looking at a mechanism for plugins to use the background processing system that X-Plane has built in. The win would be that X-Plane could play traffic cop between plugins and the sim itself, and prevent background plugin loading from causing frame stutters.
I will write up a Request For Comments (RFC) as a future blog post, so that a wider audience can comment on this.
X-Plane 11.52r1 is now available & release notes are here. Steam will update automatically if you’re in the proper beta channel; Laminar Research customers will need to run the installer to check for the new beta.
We cut this release primarily to fix our number 1 auto crash report issue, which is a crash in the networking code. We do not yet have a fix for the G2 controllers–that should be next.
Update, 2/23/21: we are no longer accepting applications – thanks to the tons of people who have contacted us! We are in the process of going through resumes, code samples, and interviews now.
We’re looking to add a senior developer to the Laminar Research team in the coming weeks.
As an X-Plane developer, you would work on both our desktop and mobile simulators, and you’d have quite a bit of latitude to work on the projects you find most interesting. At various points, you might find yourself doing things like:
UI development
Low-level performance optimization
Improvements to the X-Plane SDK used by third parties
Networking
Rendering engine work
Platform-specific OS integrations
(We don’t expect you to come into the role with deep knowledge of all those things. We like to hire “T-shaped developers,” people with deep knowledge in one or two areas, who can be flexible and pick up new stuff in other areas as the need arises.) Read More
[This post is a “behind the scenes” look at the tech that makes up the X-Plane massive multiplayer (MMO) server. It’s only going to be of interest to programming nerds—there are no takeaways here for plugin devs or sim pilots.]
In mid-2020, we launched massive multiplayer on X-Plane Mobile. This broke a lot of new ground for us as an organization. We’ve had peer-to-peer multiplayer in the sim for a long time, but never server-hosted multiplayer. That meant there were a lot of technical decisions to make, with no constraints imposed by existing code.
