X-Plane can be configured to model real-world weather conditions. It does this by downloading weather report data from a server, operated by Laminar Research. That server, in turn, periodically fetches three total files from two different sources at NOAA: one for METAR-format weather condition data, and another source for two grib2-format files describing, separately, winds and turbulence. Internally, we tend to refer to this system, overall, as Live Weather, or, in the context of anything server-related, just Weather.
Around March 22nd, it broke down.
What went wrong
As is often the case: several things.
X-Plane began reporting that it couldn’t fetch all the data it needed for real-world weather. This was a little surprising since our weather server is supposed to cache the latest data it received, even if it’s outdated, to smooth over any connectivity or availability issues with NOAA’s servers. Further investigation revealed that only one of the three files it needed was both missing from its local cache, and unavailable from NOAA.
The root cause of that data being absent from cache hasn’t been tracked down, for the reason that that part of the system has been replaced (see below). There may be a bug in the caching logic. The server may have suffered a silent crash-and-restart some time after NOAA stopped serving the file. We’re not sure.
Investigation into the causes of the missing data was sidetracked for a little while because, right around the same time, NOAA happened to suffer a widely-reported, significant disruption to their networks and server operations. Naturally, we initially assumed this was the cause of the problem, and our efforts were mainly aimed at mitigating the harm of a temporary outage, not dealing with a longer-term problem.
In fact, what had happened was a longer-term problem, and the outage was, at most, only part of the cause of the disruption we were seeing.
In which we miss a NOAA systems upgrade
Effective on or about March 22, 2021, beginning with the 1200 Coordinated Universal Time (UTC) run, the National Centers for Environmental Prediction (NCEP) will upgrade the GFS and Global Data Assimilation System (GDAS) from version 15.3 to 16.0.
NOAA Service Change Notice 21-20 (Updated)
What that means is that X-Plane’s Live Weather system was about to break.
The system mentioned in that notice is the one that provides turbulence and wind data for the Live Weather system. These are the grib2-formatted files mentioned above.
Some time into this minor crisis, we noticed that both, not just one, of those files were missing—one had been properly cached, as intended, so it’d initially escaped our notice—and that the directory structure and file naming conventions on the NOAA server from which we source them had changed. Then we found the PDF containing the notice quoted above. Nestled deep in that PDF was the line:
Remove WAFS blended product at 1.25 deg
This refers to the file we used for turbulence data. The wind data we’d been using we could still find, albeit in a new location, but here was confirmation that the turbulence data we’d been using was simply gone.
In which various solutions fail
Our situation at this point was: we had found our wind-data grib2 file, and we had several WAFS (World Area Forecast System) files that looked, judging solely from their names, like they might contain the turbulence data we needed.
None of this was true.
We learned that the NOAA system upgrade had included an upgrade to the compression used on their grib2 files. X-Plane couldn’t understand it, and in internal testing actually crashed when attempting to use these files. That meant that though we had found a file containing the same data we needed for wind, we couldn’t actually use it.
Also, none of the other WAFS-related files contained the same turbulence data we’d been using, nor anything readily translatable to same.
So much for the easy solutions.
Fortran: friend or foe?
NOAA provides a program called cnvgrib to convert grib files to other sorts of grib files—for instance, as one might guess from references above to grib2, there is a grib1 that predated it, and cnvgrb can turn one into the other. It can also, most usefully to our situation, change the compression scheme used by grib2 files, so it should let us turn these new grib2 files into something we can use without having to modify X-Plane itself. Missing turbulence data aside, at least we could convert the wind data grib2 files to be usable.
Fortran is a… let’s say venerable programming language, like, for instance, COBOL. It’s a bit obscure these days, outside certain very long-lived systems and niche applications.
Cnvgrib is written in Fortran, and no recent compiled binary version is available. The Fortran (with some support from C) source code is provided by NOAA, though, along with instructions for compilation.
One Linux VM, a couple hours, and 17GB(?!) of Intel compilers and tools later, and we have a compiled copy of cnvgrib. As hoped, it can turn the new-compression grib2 files into old-compression grib2 files that X-Plane can understand.
There’s still the matter of the turbulence data, though.
Enter the Matrix
I skipped ahead a little. At this point in the troubleshooting process we hadn’t yet noticed that the turbulence data from NOAA wasn’t the same as what we’d been getting before, and once we could convert it to grib2 files that X-Plane could handle, it even looked like it was working.
So we deployed it.
By “it” I mean a very low-tech shell script that periodically downloads the two files we need from NOAA and uses cnvgrib to make them usable by X-Plane.
Though this apparently worked, we quickly realized something was very wrong when support began fielding reports of way-too-powerful wind in X-Plane, when using the real-world weather feature. This was caused by X-Plane applying inappropriate turbulence modifiers to wind effects, because we were feeding it (from its perspective) bad data.
Having no new source of the turbulence data we had been using, and with the new data causing serious issues, what could we do without cutting a new release of X-Plane?
As it happens, we’d snagged exactly one of the old WAFS-Blended files, containing the kind of turbulence data X-Plane could use, from the NOAA servers, before they dropped off forever.
The clear stop-gap solution, then, was to simply serve the working, reasonable turbulence-data file that we’d saved. All the time. Even though it’s increasingly outdated and definitely not “Real-World” anymore.
That’s right: so far as turbulence data goes, if you’re playing X-Plane with real-world weather turned on, you’re trapped in an old facsimile of the real world. You’re in The Matrix.
Fortunately, the effect of turbulence is minor enough, overall, that this doesn’t throw things off too badly (most of the time) and having some outdated real-world data is better than having none. Still, if you notice things behaving just slightly strangely—or if you have a sense of deja vu with real-world weather turbulence while flying—that may be why. The Matrix has you.
Are we going to keep lying to X-Plane about turbulence forever, then?
Nope! We’re working on translating other forms of turbulence data into something usable by existing copies of X-Plane, and have plans to make future versions of the Live Weather system more robust so that, hopefully, it won’t fail again—at least not in the same way.
I have a question and one question only. Is there any plans to add volumetric clouds in the simulator? I feel like volumetric clouds is a huge need! P3D and MSFS has them and XP is the only sim that needs it. I really hope we can get that! Thank you. 🙂
Yes.
Are you guys going to keep us hanging till Sept since Flightsim-Expo has been rescheduled?
Yes.
Thank you for providing some context to this. I was flying VFR online with friends when this happened (via vatsim), leading to some interesting results due to headwind differences between FSX, FS20 and XP11!
Out of interest, have you explored taking your data from the UK Met Office? I believe there are only two sources of WAFS – one it NOAA and the other is the Met Office. Perhaps as a Brit I’m biased towards our meteorological data. However, worth exploring whether they can give you the turbulence data that you are missing.
As an aside, I strongly disagree with the COBOL and Fortran comparison. COBOL is only really used in legacy programming. As an aerospace engineer, I use Fortran almost daily. New cutting edge tools are written in it industry over (especially CFD, FEA, etc) and there are very modern paradigms nowadays.
I did look at Met. Their openly-available data was largely UK-specific (unsurprisingly) and their global WAFS data seems to be under similar access restrictions to that of NOAA’s (after NOAA’s Mar. 22nd update, anyway).
Didn’t mean to trash-talk Fortran, just giving some context for its age-peers. I know it’s still used in scientific communities, in particular, so encountering it in a tool from NOAA isn’t completely surprising, but did make for an… interesting exercise in build-environment configuration.
🙂 For a “ dinosaur language” it is still in the top 10 langauages used today !
Only mentioning it since its weather adjacent. Have you guys seen what DCS 2.7 has done with clouds and weather. I know x-plane is a different beast all together, but I’d love to have that kind of immersion and am willing to pay for the upgrade for it 🙂
The DCS stuff does look good!
Excellent and even funny at times writeup, Paul!
It keeps amazing me how much detail is actually involved with something as seemingly simple as “grab the weather of the NOAA server and inject it into X-Plane, done!”.
any chance when you rework the live weather youll have it blend in weather changes so we dont just get massive changes all at once
Yes.
Fortran bytecode is sometimes ever so slightly faster than the final C/C++ equivalent, but mostly it’s the grey beards in the science/engineering fields teaching their students the old ways of the F(77)orce. Us in the CFD world still cling to it… so from a distant-2nd-cousin fluid dynamics perspective, ya’ll are in good company.
I bet you could simply scale Agent Smith’s sinusoid amplitudes by some local wind velocity factor and fool us all.
Wow. Welcome to LR, Paul! Great read… I wish I could complain, but in between flight plan load tests and debugging code, using LR served weather is the least of my worries. Quite the adventure – Indiana Jones would be proud!
Just glad to read something from the new guy, and that brings a good weather report of another kind. And glad too of Ben’s “yes” and that he’s seen what the lads and lassies over a Eagle Dynamics have done with their atmospheric phenomenon. 😉 And I do remember Fortran. And card punch machines. 😀
I suspect that you’re going to have to migrate to a SADIS or WIFS (https://aviationweather.gov/wifs/products) data source in order to regain the blended product you’re after, and I’m sure a decent case can be put to NOAA to give Laminar Research access to WIFS data.
To be quite honest, the direct WAFS, GFS, GDAS data really aren’t meant for guesstimating flight / simulated flight operations, hence the managed solutions that SADIS and WIFS provide and then a variety of flight planning toolmakers regurgitate.
For example, an aviation provider in Australia may be using AirServices products, JeppPlan, or a variety of other planning products, each of which is ultimately deriving a blended solution from WAFS / GFS data (via SADIS / WIFS, or their own hand-rolled solutions out of SADIS / WIFS data). AirServices will lean heavily on the Australian Bureau of Meteorology for product, but they are using WAFS / GFS in conjunction with local observation and specialist knowledge to derive the same sort of product everyone else is getting (SIGMET, Grid-point forecast, Area forecast, etc).
Having seen the mess where people actually try to flight plan for real off WAFS estimates, I’d strongly recommend the WIFS solution. WIFS will also give a variety of charted products that can be digitally integrated / presented to improve the realism of simulated flight planning cycles (and resulting simulated flights).
Question, when is it time to stop repairing the trusted old Buick and look towards the future and perhaps invent or implement the next cool and efficient thing. Kinda miss progress when it comes to X-Plane. I have no clue how difficult it is or how costly such is to invent, but I cannot believe in these day an age there is no tech out there that will deliver a realistic weather and turbulence model, did not Austin make one for XAVION? Anway, I read above certain things are planned and guess that resides to the next version. In other words isn’t it time for a new Q&A stream? Perhaps a streamed Teams meeting where you guys geek out on new tech ideas? Anyway, thanks for all your dedicated work, repairs and tech feedback 🙂
Re: Fortran
Both Fortran and faxing should have been considered “obsolete” before the end of the last century. Nevertheless, there are reasons why they both persist. As long as they persist, frustration about their persistence will persist. Occasionally, I still get a request to fax something. Although that irritates me, I understand that such requests are going to continue for the foreseeable future.
Thanks for not requiring me to fax this reply.
I think it should be possible to simulate turbulence:
-wind >15 kts below 200 ft
-tops of clouds
-thermals below and inside cumulus
-layers with large wind gradients (edges of jet streams)
-lee side of mountains with high winds
I think the true problem with real weather that there is no offical public API to get the data. So if you have something that works, it may break any time (due to server changes). The other thing is a number problem: I guess the NOAA is not ready for serving thousands of requests per minute or second even. Thus the LR proxy.
Regarding FORTRAN: Maybe some AI can translate it to something reasonable like C…
For the files: Maybe replace the message saying when the file(s) were downloaded with a message saying how old the data inside is.
Despite of all that I was surprised how LR managed to get data out from files that weren’t expected to be used that way IMHO.
X-Plane 9 had also suffered from NOAA changing the interface…
Coming back again to Xp after a few months and I have yet the feeling to leave again. Since I have started with XP10 cloud flickering and other problems/bugs still persist. Thumbs down dear devs.
Here one fred of endless ones: https://forums.x-plane.org/index.php?/forums/topic/225235-cloud-flickering-and-flashing-in-vulkan-any-similar-experiences/
And your X-Plane Feature Requests & Suggestions is now a private space. LOL.