We’ve posted X-Plane 11.32 release candidate two – it contains very few changes from release candidate one. If you think RC2 changed your framerate from RC1 (for better or worse), it is imaginary.
The big item for RC2 is that X-Plane now uses our own replicating METAR servers. NOAA weather was plagued by 404s when the server posted METARs didn’t meet the date/time scheme X-Plane expected. Our own server should serve the latest weather we have, whatever that is.
Over the last few weeks we have spent a tremendous amount of developer time investigating reports of instability, crashes and performance problems, and the results have been quite unsatisfying. We really haven’t found a series of smoking guns we could fix to improve stability. We have learned some things about X-Plane’s performance and stability though. The rest of this post gets into the weeds; if you tune out (and I won’t fault you if you do) the TL;DR is: please turn on our anonymous analytics, and click “send” if you get the crash report form. The more gathered data we get about crashes, the better shot we have at addressing the issues.
Crash Rates and Plugins
X-Plane’s overall crash rate (all causes is approximately 14% – which is to say, for all of our users using analytics, for every 100 times they launch X-Plane, the sim quits in a way we did not expect or want 14 times. This number has been remarkably stable – it’s not a ton different between 11.26, 11.30, 11.31 or the 11.32 beta.
The sim quitting on purpose because of bad content is not considered a crash. For example, if you load a DSF with a missing .ter file, the sim will refuse to proceed and quit. This failure mode is user hostile in that you can’t fly, but it’s not a crash – the refusal to load is the code working according to design. While I would like to make this code less hostile to users, it’s also worth noting that these cases are ones where the author of the scenery pack would have been able to fix this if they had loaded their own work even just once. That is, they are caused by an add-on that should not have shipped.
(There is an in-between area where an add-on is mis-installed because it has a library dependency that the user hasn’t met. This is a deployment problem that really needs to be solved, but it’s orthogonal to true app crashes.)
We categorize crashes into plugin and non-plugin crashes; starting with 11.32 we actually get a statistical picture of this. A crash is a plugin crash (and you see the “we crashed because of a plugin: XSquawkBox” or whatever) if the sim crashed while executing code on behalf of a plugin or inside the plugin on the main thread.
There are a bunch of cases where plugins do not get correctly tagged – in particular, we regularly see crashes on random worker threads spawned by plugins; since we don’t know whose thread it is, we can’t blame the plugin. For example, the FF A320 crashing inside CEF on a worker thread is registered as an X-Plane crash (and we see it in our auto-reporting view) but it’s not our code and there’s nothing we can do about it.
One thing I’d like to do in future patches is improve diagnostics. The rate of actual blamed plugin crashes appears so far to be quite low, and given that we do see uncaught plugin crashes in our data on a regular basis, I think this is a case where add-on authors can only fix what they can see. If we can attribute all plugin crashes to plugins, then the plugin authors can catch their own bugs. Better diagnostics also helps a user remove a troublesome add-on in the case where that would help.
“Known” Crashes
We have a few cases where X-Plane hits an error condition and deals with it by crashing. This is pretty bad, drives up our crash rate, and is something we need to fix to be less user hostile. For example, if a PNG file is bad (either corrupt contents or the sector on disk that backs it has gone bad) then X-Plane’s response is usually to mysteriously crash. Besides being rude, the crash gives an end user no idea which file is bad, and thus no way to fix it. X-Plane ships with something like 9000+ PNG files and over 2500 DDS files, not counting add-ons, so if we don’t tell you which file is bad, you’re not going to find it by poking around.
FMOD sound bank incompatibilities is another example – if you have two aircraft sharing a byte-wise copy of the same FMOD data (e.g. command-D duplicate the C172) loaded at once – then our FMOD loading code fails and then registers as a crash. The code is working exactly as we designed it, but the design isn’t robust enough. As we’ve learned, in the real world users duplicate aircraft (and their FMOD packs) on disk all the time.
The crashes in this category are things where we’re being user-unfriendly and better code would make these problems go away or leave users with a way to actually fix them. But they’re not case of “weird stuff inside the sim blew up.”
Persistent Stability Problems
In the crash data we do also see a few persistent stability problems. There’s some kind of crash in the ATC system that we’ve seen for a very long time but we don’t know how to reproduce. It’s a case where if enough users do enough random stuff, we hit an edge case in the ATC system that isn’t handled correctly. The solution here is to embed more diagnostics at the crash site until we can understand it from the reports we get from users. Please hit “send” when you crash – don’t worry about filling in the fields – it’s the report itself that we need.
We also see a lot of crashes inside the OpenGL drivers, from all of NVidia, AMD and Intel. Because the IHVs don’t share symbols and source code with us, we really can’t tell what went wrong in these cases.
My hope is that with Vulkan we’ll have better options for in-driver crashes. With Vulkan, they redesigned the error checking model: error checking is a feature you enable (via a configuration option at app startup) that brings a layer of code in on top of the driver to check what the app is doing. With error checking off we get the fastest framerate, and with error checking on we get slow framerate (more error checking means more slow) but some really great diagnostics.
(To put this in perspective: when Sidney ran the Vulkan version of Airfoil-Maker on Linux with the wrong driver installed and no error checking, it rebooted his entire Window server! So no error checking really means: no error checking.)
Since error checking is optional and selectable when the app runs, we could put an option into X-Plane to run in “safe mode” – if a user is hitting persistent stability problems in the driver, that user could turn on validation and possibly capture an error in X-Plane itself that would otherwise just be “the driver crashed.”
Running Out of Memory
I’ve worked with a few users to try to track down the out of memory problems we’ve heard about, and there isn’t an obvious pattern here. Some users report running out of memory in 11.30, but when put back on 11.26, find that they still run out of memory. We get more out-of-GPU-memory complaints on 11.30, but that might be because a few cases in 11.26 that were crashes due to running out of memory now report the problem in an orderly way. In 11.26 they were just mysterious crashes with a “send” form.
In the cases we’ve seen, the user running out of memory was often…actually running out of memory – that is, the surprising thing is not the crash but that X-Plane ever worked at all on those settings. The fundamental problem we face is that we have no visibility into what the OpenGL driver is doing with GPU memory. The OpenGL tries to manage memory no matter how much we ask for, and if it fails, we don’t know what went wrong.
The good news is: we have much better options for Vulkan. With Vulkan, we manage memory, which means we know what’s going on, and we can take steps to avoid out of memory crashes. If we do run out of memory, it should be for much more obvious reasons. We’re still analyzing what we can do about memory with Metal, but the choices should still be better than OpenGL.
The only advice I can offer now if you are seeing persistent memory crashes is: turn your settings down or use less add-ons. If you push X-Plane to the limits of your hardware, it may work for a while and then fail.
Performance
Sidney has looked at a lot of performance data from users who reported low framerate, and in almost every case, the performance has been as-expected. The most common case we see is users with relatively low single-core CPU performance hitting low framerate at high rendering settings while their GPU is bored. To put some numbers on this, if your CPU’s single-core geekbench score is down around 2000, you are almost certainly CPU bound, way at the bottom of what’s okay for X-Plane, and a new GPU won’t help.
As of now, X-Plane cannot use large numbers of cores (e.g. a 32-core machine is useless) and gets only limited performance boosts for framerate with multiple cores under some circumstances. This is something we are working to change in the future, but it’s not going to change quickly. If you are looking to improve performance with hardware, single core speed is still the most important metric.*
In a few rare cases we saw performance that was disproportionately bad compared to what we’d expect from an old CPU. We’re trying to gather more data from these users but the case is rare enough that we haven’t gotten a useful report yet. If we find a smoking gun, we can act on it.
Like memory, Vulkan will help with diagnosing performance. With Vulkan, more of the code is written by us and the Vulkan code we run has very predictable timing. So when we get complaints about performance, we’ll be in a much better position to understand what is slow and why.
What’s Next
I’m actually not sure what the next patch will be, but we do have a bunch of bug fixes to 11.30 waiting to go out once we have stabilization under control. I also have a pile of bugs that I have not yet fixed that are high on my priority list where something in 11.26 stopped working in 11.30. So if you have filed a bug that’s not fixed, we have not forgotten about it – it’s either next to come out or possibly on the short list.
* Yes, we realize that this dependence on single core speed is bad. It’s just going to take time to move to Vulkan and then offload the single thread.
Thanks for the feedback, this is very important for us!
Regards, Igor.
That quit on a bad scenery tile is a right royal pain in the ass, even for people who don’t use commercial scenery, I have lost track of the amount of times I’ve been halfway through a long flight, over scenery created in Ortho4XP and, bang, the sim quits. As a pretty regular user who has the entire United States in Ortho4XP, it’s sods law that there are a few dodgy tiles here and there, hardly my fault. Reading the first couple of paragraphs of this post made it sound like you guys crash the sim on purpose, as a punitive measure against bad third party products ! I always assumed there was no way around this crash, now I’m starting to think differently. I will admit that as an Oculus user I can count on one hand the number of times I’ve fired up X Plane in the last year due to performance issues and having to end the program in Task Manager everytime it closes (no other VR title has ever forced me to do this, yet XPlane seems to blame Nvidia) I’m still hoping you pull something out of the bag with this move to Vulkan that makes XPlane playable, I would like to think all that computing time making Ortho tiles for the entire U.S. wasn’t wasted !
Hi Ben
How does on get the r2 beta patch on steam – I’ve already got R1 beta patch and steam hasn’t downloaded anything new for Xplane?
r2 is not out on Steam yet but will be ‘real soon’. I actually spent the last 3 days automating the steam build process and tying it to the LR build process. So in the future, the Steam build will be ready for us to release as soon as we think it’s ready. (Until now we’ve tried to improve Steam availability times mostly through heroic manual builds by Philipp.)
Sometime, you could just change the release number and declare “20% perf improvement”, without touching the code. And then see half of users going: “I got +50% fps! LR rulez”, and the other half going: “my fps halved! LR sucks!”.
I joke about this at every company meeting. Placebo is a powerful thing.
So, when you say Vulkan will take time, it means we are far from a Vulkan Beta? Not this year.
*Not complaining, just want to know how close it is. Thanks.
No no no – Vulkan beta this year or Austin cuts my finger off. I’m saying that _even_ after we ship Vulkan, the cost of rendering “the frame” won’t be multi-core just from that. It -may- be faster just by being Vulkan (and we do hope Vulkan makes a real improvement) but the port to Vulkan isn’t a port to multi-core.
The port to Vulkan does make a multi-core renderer possible, because Vulkan has multi-core friendly rules, while Opengl has multi-core-hostile rules.
*Imagines finger cutting with a maniacal Austin laugh*
Yeah: You promised “Vulcan will be much betrer than OpenGL, but you won’t get it soon”. should we be happier now, or less happy? As we say at home: A sparrow in the hand is better than a pigeon on the roof”. Do maybe kerp thr sparrow (OpenGl) healthy while trying to tame the pigeon (Vulcan) for all of us.
Would the sparrow be happier if there was no vulkan?
you guys have spent what 18+ months now and this its time give up and move on to XP12 and use another engine its clear Austins 20 year old engine is just not up to the task now with out nearly rewriting it all and the time and code you put in that could be better put in to basing XP12 off stock engine that already does every thing you want
this just getting silly now so your going to move over to Vulkan but not use any of the stuff in Vulkan that makes it better? why again im thinking because the of APPLE YOU REALLY REALLY NEED TO DUMP MAC its only holding things back and many devs have said they wont support metal any way on top of that Metal is a mess to start with still
face it Apple is not the place for the desktop sim just port Mobile over for the Mac and then move XP12 to linux/windows and use Unigine and save your selves 2+ years of work
great so itll run on the Mac but with no 3rd part support
Loading dsf files will suddenly cause fps hugely drop.Can it be noticed by the developers and will be fix in the future?
Excellent update, thank you Ben.
I am using 1132 R2 on one drive and 11.30 on another and have to say the only time i have had a problem is when i have tinkered with something myself or added something that didnt install well. the current XP is easily as solid as anything else in the sim market, no it isnt perfect by a long shot, but it is perfectly useable, and we all enjoy and appreciate all the work that is being done to improve it, Rome wasnt built in a day!!
Hi
Would you benefit from a METAR server in Australia? I run a hosting company and can host one for you, maybe even for free (depending on how big your server needs to be)…
Actually still having old X-Plane 9 around, I was thinking about writing a METAR downloader myself. I’d prefer that solution over a Laminar-hosted METAR proxy.
Hi Ben
I would like to report that I haven’t gotten any crashes using 11.31 (knock on wood) and I also haven’t noticed any errors using real weather those times I did. Normally I use Active Sky for parsing weather into the sim, even with its qwerks and twitches I have not seen any CTDs yet related to weather. When checking the output folder I do however see 3 dump files present but not any crash reports. When it comes to performance, I have noticed a decreased hit in 11.31 vs 11.30, but I think most of those are due to weather injections by third party and cloud texture. I do have a clean version of X-Plane with latest beta updates. I use this installation purly as a test platform. So when an error occurred I move into that an re-test. If I the error can be reproduced, I tend to send a bug report. Luckily Jennifer does an excellent job investigating. In any case, just wanted to let you know there is some different views on the matter of crashes. I would say in 100 boots I would get if any not more than 1 or 2 crashes. In most cases it has to do with scenery, plugins or any other third party error.
Best of luck with hunting down those crashes and making X-Plane better.
Hi Ben, on the last live stream you mentioned Vulcan was going to be run for the first time in X-Plane itself. Just wondering how that went?
Great summary by the way. I found the information really informative.
Lee
What about AMD cards? For me the most import thing with Vulcan is that I hope it will fix AMD cards performance which has compered with NVIDIA verly low performance, I think similar card of AMD is 40% slower than NVIDIA and as far I understood is because AMD has terrible OpenGL drivers?
J uses the VR function a lot.
In order to avoid a loss of FPS would it be possible not to have Oculus rift at each launch of the VR?
No particular bug Xplane works properly.
Ben, cou;ld you please ask Julian to authorize uploads to the Gateway if somebody is on 11.32 R2? Thanks, Matt
You can’t upload airports to the Gateway from a beta version of X-Plane. For this reason I keep a second non-beta copy of X-Plane on my drive and that’s where I do my airport development and uploading.
Julian authorized it. Thanks
With 32GB ram and 12GB vram, I never run out of memory UNLESS I reload scenery or change location during an XP session. If you are interested, I can produce the Out of memory error and file a bug report.
Happens to me as well. 32GB and 8GB VRAM but after two or three times reloading the scenery (Developer Options) I get a message that XP is totally out of memory. However my task manager claims otherwise and shows roughly 16GB available.
Been wondering about this for quite some time.
Hi,
Thank you for these explanations Ben, it’s always a pleasure to be immersed in the development of X-Plane.
I write plugins and tend to crash X-Plane due to my own mistakes (not that often but still…). Should I re-activate analytics or keep avoiding sending you “polluted” data?
NOAA servers are notorious like that. I’ve had amusing issues w/ their GOES13/15 imagery servers in the past: e.g. the server would fault Fri night and nobody was around to hit the reboot button until Mon morning. I would expect aviation weather products used by XP would receive more round-the-clock care than that.
That said, the transition to GOES16/17 has coincided with use of an AWS repository for said sensor data and I haven’t had a problem since. I wouldn’t be surprised if the data XP pulls down is on there somewhere too.
One dream of mine is to use that sat data to distribute XP cloud coverage via a plugin. A dream for a time when days have 25 hrs…
Nice to see LR so involved in keeping us simmers updated on the progress of Xplane. I usually get a crash when a new or updated plugin is installed. I reboot and all goes well afterwards……sensitive huh!!..
Has there been any discussion about how far behind the METARS are even with the new setup? X-Plane was reporting no rain and dry runway conditions as a storm front moved through this weekend because the METAR was two hours old (I checked the METAR text file). And yes I had “refreshed” multiple times and the frequency is set to 30 mins. Avaitionweather.gov had the current METAR with storms.
It’s like saying X-Plane has real world weather as long as the weather hasn’t changed from 1-3 hours ago.
AFAIK METAR time stamps are UTC. So did you set up the UTC/GMT offset correctly (to let X-Plane select the correct file to download)?
I did a check right now.
EDDS UTC 07:12 local time 08:12, no summertime,
METAR 0600
Changing localtime to 09:12 results in UTC 0812 and METAR 0700
Looks like METAR is behind
My fault!
I mixed up METAR and ATIS!
Please delete my last posting, Thanks.
I would like to bounce back on the performance.
-1. I won in FPS by retouching my file “sittings” a file that you should repackage, a name notice there are too many unnecessary parameters..
-2 My average FPS 60/70 for the aircraft “Aerobask retouched Laminator” and on the Cessna 172SP 100/110 with the following configuration:
high visual effects (HDR)
Textures high
Anti-Alliassing 4 X SAA
objects medium
-3 For the hardware I have a mobile Predator “PH517-51/1070” and a Bullguard program with a game booster that optimizes activated games. As for the graphics card “Intel Vulkan” It is for me that a commercial war
-4 – The VR 25/40 depending on the type of aircraft but I still believe that the loss is the Oculus that turns in the background. The also you should try to override..
-5 – Overall X-plane 32RC2 works well and in VR a treat except on Airbus and Boeing or there are too many gadgets they are good for professional training.
VR loss isnt that as there is no loss any other VR aps like Elite and Project Cars
the issue is on LRs end they REALLY need to update to Rift SKD 2.0 the other issue is that they are drawing everything TWICE its like they didnt even bother to ask Oculus for help which they offer devs for free
no there something else going on that kills the frame rate you can tell this by when your in the VR hanger and frame rate is fine
Bits of intelligence to add: Do not select the glider (ASK) as AI traffic on foggy weather. As the glider foes not have sn artificial horizon (gyrocompass), it’s a challenge even for real pilots when in the clouds…
Similar for strong winds, heavy rain and balloons…
Do not set the glider as AI at all – it has no engines!
Is it possible to use another CPU core for AI caculation?
We already do that – have since v10, at least!
Vulcan in 2019, then Raytracing in XP12?
News about steam version?
The weather is always wonderful 😀 does not accept the noaa weather plugin and the manually weather…
11.32r2 is available as a steam beta.
Since you already mention the FF A320 in regards to crashes, as an example. Just a lingering thought. It would be very interesting to know what kind of plug-ins or addons that statistically bring XP out of balance. Especially since some of their creators or commercial moderators claim the product is 100% stable, or as often stated «We have thousands of customers not experiencing crashes»
Hi Ben 😀 !
I’m a X-Plane noob from Germany, and just want to send you a warmly “Hello” out of the core of my heart. Right now I tumbled across your blog-site (never knew before that this exists), and ‘m still AMAZED !!!
You are doing a wonderful job, seriously, very straightforward, and I love it !!!
Nevertheless transferring difficult computer stuff in such a enjoyable manner and transparency, that even computer cretins like me enjoy it to read, even beeing capable to understand most of it too surprisingly.
Pleas take care of your health, Ben. I’m drinking a lot of coffee too (and do a lot of swearing/sometimes).
But what really helped me was to change my habits of drinking it from instant coffee to freshly ground, high quality coffee products (its a way lot better for your stomach).
I know you US guys have a lot of bad habits regarding food and drinking ;).
All the very best for yourself, your daily life-balancce work, and your sweethearts.
THANK YOU! THANK YOU! THANK YOU!
Thanks for the update. How about the idea of coding the ‘Exception Headquarters’ to deal with all errors and missing resources? The primary purpose being to Not let the sim die a sudden death. Let’s keep XP alive.
“* Yes, we realize that this dependence on single core speed is bad. It’s just going to take time to move to Vulkan and then offload the single thread.”
and this is why you dont roll your own engine in 2019… you let some one else that has full time team of 100’s do it like say Unigine
they seem not have had any issue with this kinda thing nor any other modern 3d engine
again maybe its time that you just stop wasting time on XP11 and start on XP12 using a off the self engine like Unigine that EVERYTHING you could ever want in it already
this limping along trying to rebuild an engine that has its roots still in the fixed pipeline GPU world is insane
you still dont have water thats not a flat texture in 20 freaking 19 or deformable mesh things other engines have had for 10+ years now
and dont give me the but its a sim thing Unigine has been used already by Bell as sim engine along with many others if you bother to look on there website
this would save LR 1000’s of hours of work to improve other things like flight model with out having to mess with low level things
Good. When you start your own company and create your own flight sim software, you’ll be free to use Unigine as you see fit. Until then, it appears Laminar will continue down the path it sees fit for them and their software.
Hey Elios,
This topic has been brought up so many times already… even by me. The reply is always the same. They have considered it and decided to stick with their own engine. Be it because of pricing, or the effort of porting everything, they still prefer not to use a “game engine”.
Water is still awful, yes. But everything else is coming in nicely, no? 🙂
At the moment the greatest “let down” for me is the lack of mediterranean autogen, as I fly mostly in Portugal and Spain, but MisterX is about to release a product where this is addressed, so I’m a very happy camper right now 🙂
Cheers,
Bruno
there are other things they need work too not just the water lack of 3d trees, volumetric clouds, clouds effected by winds… scenery loads… weather loads…. VR issues… i mean just look at the things Unigine does out of the box with no extra code mean wile LR is hear re inventing the wheel thinking its the greatest thing ever
all of the this gets solved by going to an off the self engine and Unigine has already been using in hard core simulations putting XP’s flight model on top would not take that much work
YES they say the looked at this before but have they in the in the last year? and did the they look at Unigine?
Why u mad tho bruh? You think anyone at Xplane is going to respond positively to that tone? Didn’t you ever learn you catch my flies with honey than vinegar?
Either you know what you’re talking about but no one will consider your idea because you’re so abrasive or you’ve got no clue what the inner business requirements are and you sound like a fool.
Send me your address, if xplane won’t give you your money back, I’ll give you $60 to delete the product and never comment here again.
Be a fan, not a fanatic.
Dang this post is cringe.
“This is why you don’t roll your own engine in 2019”
“Unigine has everything you could ever want in already”
– I would like to see the $10k+ you are forking for a license to this engine, and also donating your time and resources to porting over 20 years of content to XP12
“still dont have water thats not a flat texture”
– Ben literally had water tesselation working in early XP11, and writing a water shader is a damn near trivial thing. It is quite clear that there is work being done to XP to support more advanced water in the future i.e. Austin working on new water physics, Water DSFs obtaining rasterised DEM data for potential depth and bathymetry later.
“If you bother to look on there website”
Yes, we can all read
Bottom line, the attitude here stinks. Constant suggestions that these companies like Nvidia or Outerra are going to do the work for Laminar for free, if they submit their code is just outright laughable.
Fix up and suggest something more productive to post here.
“Ben literally had water tesselation working in early XP11, and writing a water shader is a damn near trivial thing. It is quite clear that there is work being done to XP to support more advanced water in the future i.e. Austin working on new water physics, Water DSFs obtaining rasterised DEM data for potential depth and bathymetry later.”
then where is it if its so easy over a year later? see what i mean engines ARE NOT easy and frankly XP12 should be clean break any way
Great article for those of us who are big-time XP users but not IT experts! Thanks very much Ben. Cheers.
Ben, quick question: does the Installer Updater find and correct bad png files?
From our sim itself, kind of. If the file is bad on disk, the installer might report an error trying to check for an update, but at least the log will tell you which file and you can delete it and try again. It would be nice in the future to have it re-download the file, but it’s really hard for us to test these features because we can’t easily re-create failing hard drives to test this case. If the file just has the wrong bytes for whatever reason, it will detect that.
It won’t help for third party scenery though.
Hi Ben,
is there any way/any chance that X Plane 10 will get the real weather download fix as well? (XP11 is great but my system just isn’t up for it in any satisfactory way).
I’m currently using a plugin to get some semblance of real weather but, as good as it is, it isn’t as smooth as the original way……or is this a ‘snowball’s chance in hell’ type request 😉
Regardless, keep up the great work, X Plane just gets better and better.
Cheers
No – we haven’t patched X-Plane 10 in years – it’s basically end-of-lifed. I think someone could make a plugin to download the weather from our new server and it would work in xp9 and 10.
Thanks for the reply, I know it hasn’t been patched for a long time!
A new plugin sounds interesting….and a learning experience 😉
So here’s the thing about why many of us choose X-Plane… it provides the most realistic experience of flying an aircraft, from piloting to aircraft dynamics, performance and physics. The graphics of the outside world are indeed way off arcade style games such as GTA5, and I guess they always will be (although 3rd party scenery such as Orbx TrueEarth is stunning addition). Incidentally, when you’re flying at 35,000ft in real life, water does actually look ‘flat’.
If you also consider a level D (full flight) simulator, the graphics aren’t any better than X-Plane (most I’ve seen are actually a lot worse!), but that’s because pilots use them to learn how to correctly pilot an aircraft, rather than look at how realistic the outside world appears.
I guess it comes down to how you’re intending to use X-Plane… do you want it to be a game, or a simulator?