The next beta will be out in a day or two, and it will be 10.50 release candidate 1 – that is, I’ve just finished off the last few bugs holding us back. A few quick bug notes:
- Beta 7 has a major bug in how AI airplanes can park at airports – this should be fixed in the RC, resolving a lot of confusion by authors.
- The KLAX tall-buildings-in-the-approach path are fixed – this is an art fix.
- Alignment of water reflections with the terrain is slightly better.
On this last bug, three things can go wrong with the water reflections, and only one of them is fixed:
- The Earth is round, but X-Plane has to pick an arbitrary flat plane to pick as “the water”. If this flat plane isn’t pretty close to the triangles where the reflection is closest to land, then this can cause error.
- Similarly, since the Earth is round, any time the water isn’t perfectly flat, weird stuff happens. This is normally not a problem, but sometimes if airports have been flattened this can cause crazy reflections.
- There was a small permanent offset in the reflection noise texture, partly due to mipmap rounding error and partly due to a 0.19% rounding error.
This last one is what’s fixed in 10.50 and it should remove small errors where the reflection plane is picked right in (1) and there isn’t goofy shaped water (2) – which is to say, it should help some of the time.
X-Plane 10.50 beta 7 is out. There are a few fixes that I hope make this finally a “solid beta”: no more flashing airport lights during the day, normal maps back on aircraft, and liveries should work correctly.
We have a fix in the works for tall buildings blocking the approach path at KLAX; I’m hoping to get that into beta 8 some time next week.
Creating New Apt.Dat Layouts
WED 1.5 beta 1 is out; please do not upload revised airports with WED 1.5 yet. If there are still bugs in X-Plane’s handling of the new apt.dat data, then you have no way of knowing what your airport will look like after we fix those bugs.
X-Plane 10.50 adds a new rule: AI aircraft will not be born or land at airports where there isn’t a parking spot wide enough for the aircraft to park. So if you have an airport with a 11,000 foot runway and no size-E or F parking spots, the 747 would have landed in 10.45 but will not land in 10.50.
AI aircraft will land even if the parking spots don’t match the needed aircraft type – in other words, if you have a size E parking spot for helicopters only, the 747 will still land (and try to park there). Here’s why I didn’t stop this case: without static aircraft, the equipment codes on parking spots is likely to have errors; it’s hard to know you get everything right. So if we start requiring equipment code matches to land an aircraft, we may end up with no AI aircraft at lots of airports.
Once static aircraft have been deployed and authors update their airports to use them, the equipment codes should become much more accurate, and at that point we can prevent the AI from landing if it can’t find an equipment code match on a parking spot.
We also do not limit landing by taxi route width; the assumption is that if you have a width-E parking spot, you have a width-E route to the active runway somewhere on your layout.
Starting with beta 7, you can now debug ATC AI placement decisions by setting the art control atc/debug/log_spawn to 1. After it is set, future spawning decisions will be heavily logged under the “ATC” log tag. If an AI isn’t placed at an airport, you’ll see exactly why.
Note that currently the takeoff/landing requirements for the aircraft tend to be inaccurate. (They are computed from some properties of the aircraft, not flight testing.)
Flow Problems
From my examination of the small set of airports that I run across while debugging (with something like 3000 airports, I am at best “sampling” the gateway airports), it appears that authors don’t understand how ATC flows work. We’re working on new documentation to try to explain this, and I’m thinking that some very basic flow errors could be caught by WED. For example, I saw one airport that had this in WED:
Airport XXXX
Flow "east/west"
Runway Use: 9 (arrivals, departures)
Runway use: 27 (arrivals, departures)
This is almost certainly a mistake. This is a single flow for the airport that says “when this airport is in east/west mode, aircraft may take off or depart from either runway 9 or 27!” In other words, airplanes can use the runway from either direction at the same time.
I know of no real-world example where this actually happens, and if there is one, it has to be super-rare. The author probably meant to do this:
Airport XXXX
Flow "east"
Wind rule: wind must be coming from the east
Runway rule: 9 (arrivals, departures)
Flow "west"
Wind rule: wind must becoming from the west
Runway rule: 27 (arrivals, departures)
In this case, the airport is using either 9 or 27, but not both at the same time. A few things to remember about flows:
- Only one flow is ever in use at a single time. That’s why a flow can have more than one runway.
- All of the runways within a flow are in use at the same time when that flow is in use. So never put conflicting runways in a flow!
- X-Plane picks the flow by looking at your flows in the order you put in WED. It takes the first flow where all of the restriction rules (time, wind, visibility) can be met. So put the ‘preferred’ operations for your airport first in the list.
- If you have more than one rule of a type, only one must be passed. So for example, you can have a “rush hour” flow with two time rules, and it will be picked if either is picked. So you can make morning and evening. Similarly, you can make a wind rule for “strong from the east” and “calm wind” and if either is picked, the flow is picked.
- If you provide no rule for a category, the flow is never disqualified by that rule. So if you have no time rule, the flow can be picked at any time.
Here’s two more art controls: atc/debug/rwy_flow debugs how X-Plane picks its flow – turn it on and then go to your airport and you can see in the Log why the flow was picked. atc/debug/rwy_selection shows how a runway was picked for an airplane given an existing flow.
The first beta for World Editor 1.5 is now available to download.
This version features numerous bug fixes, along with major improvements to make editing airports easier and faster by providing more visual clues. It’s also the first 64-bit version of WED!
Some highlights of this version include:
You can see the full list of bug fixes, improvements and new features in the README.WorldEditor file found in your downloaded WED folder.
Please try the latest version as soon as you can and let us know if you find any bugs by filing a report on the Scenery Tools tab of the Gateway (not the desktop bug reporter for once!).
X-Plane 10.50 Beta 6 is now live. Here’s the full notes. As always, please report bugs via the bug reporter, not in the blog comments. (And if you blog about a bug in the forum, you might as well be reporting it to your cat – we do not have the resources to scrub dozens of forums for bug reports! You are welcome to discuss bug with other users in forums, but if everyone assumes someone else is going to write the actual report, X-Plane stays borked.)
A few high profile fixes:
- The black texture/crash loading airplanes bug is fixed (again – hopefully!). If you see any pitch black textures, crashes loading aircraft, or cryptic error messages about “negative reference counts”*, please report this ASAP!
- I found a race condition in the art asset loader. This is the kind of thing that would cause a crash very rarely, but the results would be inexplicably bad when they did happen. So that was exciting.
- A bunch of random crashes that we found by the auto reporter are all fixed. If you are on Windows and X-Plane 10 crashes, please click the auto-report – while we don’t have time to look at every crash report, we can look at the highest frequency crash reports and fix them, so complete data really helps.
Async ATC
For X-Plane 10.50 I restructured the code that loads ATC “controller cabs” (that’s the simulation of the actual control tower, and it contains the data X-Plane needs to route AI aircraft on the ground) to be fully asynchronous. In X-Plane 10.45, building cabs can cause the sim to stutter for up to 1/4 of a second as the cabs are loaded.
What I discovered last week is that while I had written the code, I hadn’t actually included it in 10.50. So it’s in beta 6; I’ll keep an eye on the automatic crash reporter; this is the kind of thing that is supposed to go into beta 1 (so that once we are stable, the sim stays stable), but it’s also a key fix.
The new ATC controller cab loading code is stutter free. But – please do not report “the sim stutters” as a bug. The rest of the sim is not guaranteed to be stutter free; it depends on the scenery you use, the rendering settings, your hardware, etc. Until we have evaluated everything the sim does while flying and rewritten everything that can stutter, we don’t guarantee locked fps and smooth flight. The rewrite to ATC controller cabs is a big victory (we also removed a stutter for pro customers using USB keys and made the static aircraft stutter free) but there is still more to do.
Becoming completely stutter free using multi-core is a long term goal that we need to keep making strides toward, but it’s not a “do it in one patch” kind of thing.
Third party developers – please try your aircraft on beta 6! If the black texture bug really is fixed then our bug list is now very short, and the beta won’t last much longer. Please make sure your add-on works, so that you don’t find out about bugs after we go final.
* For the programming nerds reading, when it became clear from automatic crash reporting that we had a reference counting bug that was causing crashes, I made the reference count sanity check assertion run in full release builds, so we could get a guaranteed reliable crash and auto-report instead of depending on all hell breaking loose due to doubly-freed objects.
Note: beta 5 came out before the long weekend and then I managed to not publish the announcement. Sorry about that! Since the original writing of this it has become clear that only some of the no-texture objects were fixed, and the throttle reversers are borked. Beta 6 should be out in a day or two addressing both.
Update: just to clarify, since this post kind of freaked people out: We are not closing off the scenery system. We are not changing or hiding any file formats. We are not removing any scenery tools. If you’re using a supported part of the scenery system now, you can keep doing what you are doing.
X-Plane 10.50 beta 5 is here – see the release notes for a complete list of bug fixes. There are some big bug fixes here including a few big ones:
- Fix for fuel consumption (which was tied to the number of physics cycles you had set).
- Fix for backward control surfaces (which was intermittent based on what aircraft you had loaded before).
- Black textures and crashes loading airplanes are now fixed. (This case was getting hit only when the right combination of untextured objects was used.)
- Fixed some crashes loading scenery.
On this last bug: it looks like the w2xp “net” sceneries are responsible for virtually all of the crash on scenery load bugs I’ve seen in beta 5. The problem is that w2xp doesn’t pre-process and sanitize raw OSM data before sending it into the sim; the result is that a small number of weird OSM roads are causing a cascade of ever-weirder data in the rendering engine, eventually resulting in a crash.
I’ve gone to the extraordinary step of programming X-Plane to attempt to clean up the road data on DSF load. You’ll see a log warning with w2xp net scenery when this happens telling how many road chains had to be deleted. Even with this, I can’t guarantee that there isn’t some OSM data out there, when dumped directly into the simulator, that won’t crash.
I’m not sure what the long term fix to this is, but it did make me question our scenery strategy. We’ve always tried to keep the scenery system open, and I think it’s fantastic to see third party developers doing things that we’ve never done with X-Plane. But I also don’t know how we can guarantee X-plane’s stability when this kind of unmetered low level access to the rendering engine is provided.
My son is five and he really wants to participate in conversations at dinner. “Daddy, what are you talking about?”
So I tried to explain to him that we had worked all week to get X-Plane 10.50 beta 3 ready and that we wanted it to be really good, and then at the last minute we discovered it was broken, so we had to go back and fix it and try again.
Gabe: What are you going to do if it’s broken again?
Me: Probably cry.
Gabe: No! Daddy, you’re not going to do that! I said no joking!
Me: No, I’m being totally serious. I’m going to sit down in that corner and wimper with my head in my hands.
X-Plane 10.50 beta 3 fixed a lot of big bugs – several crash bugs fixed, etc. It unfortunately also had a small problem where it wouldn’t start without blowing up. (This one got by us because it only happens at some airports, and it requires a clean release machine to see- it doesn’t happen on developer machines or some of the more heavily loaded installs that Jennifer tests with. The breaking change also went in late in the game, and thus missed screening.)
So we turned around beta 4, and if it’s unusable, I’ll have to reconsider my life choices.
Crash Fixes!
We fixed three high profile crashes in X-Plane 10.50 beta 3 (now available to you without brain damage in beta 4):
- Crashing at DSF load time if a DSF mesh has a crack in it.Apparently some G2XPL-made meshes have tiny hairline cracks in them. They’re almost impossible to see but they were enough to crash the road code. DSF mesh cracks are illegal (per the DSF spec) but X-Plane now handles them anyway; I don’t think anyone wants to stop using this category of scenery.
- Crashes reading real weather when the .grib2 files are missing on Windows. I’m not sure of all of the different ways this was happening, but it should be fixed.
- There was a memory scribble in the scenery loading code for some scenery pack configurations.
That last one is in its own category of badness. A memory scribble is when a piece of X-Plane code just goes off and writes over random memory (in X-Plane’s memory space – X-Plane can’t splat the memory of other apps on your computer). Once this happens, pretty much anything can explode at any times ,and often it does.
We can’t know how many of the crashes we were seeing via auto-reporting were due to the scribble, so:
- Please do get X-Plane 10.50 beta 4!
- If you reported a crash bug and you still are getting a crash in beta 4, please re-report it.
- Please keep on auto-reporting crash bugs on Windows and Linux – this is hugely helpful. The auto-crash reporter gives us a statistical picture of what’s going on with the beta.
Scenery and Aircraft Authors: I think 10.50 beta 4 should be stable enough to look at your add-ons; airport lights should be fixed and working normally, scenery should not be crashing, and missing taxiways are fixed.
Scenery authors in particular: 10.50 contains extensive changes to the rendering engine under the hood, so please take a careful look at your scenery packs and make sure they haven’t mysteriously changed in how they look compared to 10.45.
I tried to rush through X-Plane 10.50 beta 2 over Friday and Saturday, in an attempt to get past a number of high profile crash bugs and installer errors quickly, despite having our Q/A person out of the office.
This didn’t work too well – beta 2 introduced a new stupid bug (the airplane lights are always on), and of course there are still plenty of other bugs.
So my plan now is to slow down a little bit and try to take time to verify beta 3 a bit before releasing it, which means it won’t make it out until mid next week. There are bugs we’ve found in beta 2 that are not quick fixes that are still quite severe, so it’s time to measure twice, cut once, not take two steps forward and one step back.
In the meantime:
- To everyone who has reported bugs via the bug reporter, thank you! We’ve gotten a lot of bug reports, which is great – it gives us a rapid picture of what’s working and what’s not.
- To everyone who has also posted their bug report in the comments section, please stop doing that! I have deleted a number of recent comments that were either “I also posted this bug to the bug reporter, but (bug report goes here)” or “this is not a bug report but (“bug report goes here”). We really need the bug reports in a single place where we can carefully track them. That place is not the blog comment section.
Tall Buildings and Data
I believe that X-Plane 10.50 beta 2 fixed the bug where the tall autogen was missing from the installer. Unfortunately this means that if you still don’t see tall buildings, it’s because the DSF doesn’t contain tall building information. A good check is to go to New York City – if it’s tall with max autogen, it’s the underlying DSF data, not the autogen.
The tall building height data that is in the global scenery DSFs now came from a mix of FAA data and user collected data from a very long time ago. Therefore tall buildings are very likely to be missing or in the wrong place outside of the US. Our future plan (which alpilotx has already made progress on; I need to examine some of the files he sent me) is to use OSM for the height data too, which should be a huge improvement.
While we do plan to use OSM data to provide DSF height data, that is not going to happen soon, and is not going to happen for X-Plane 10.50.
In the meantime, I will try to post soon the DSF information on how the height data is encoded. This will at least give third party developers the option to create DSFs that take advantage of the new autogen.
Posted in News
by
Ben Supnik |
The good news is: 10.50 beta 2 is out, and it fixes a pile of big problems with beta 1.
The bad news is: it has its own problems – apparently some of the airplane lights are on all the time. This will have to get fixed in beta 3.
If you get a crash on startup, you’ll need to run the installer to get beta 2 – auto-update might not run due to the crash.
A quick note on scenery: I was able to fix one scenery-related crash in beta 1, but there are others still in the code. If you are seeing scenery crashes, I need a Log.txt showing the problem -and- info about what scenery and what settings you combined to get the crash. With the complete reproduction information, I can fix the crash.
Posted in News
by
Ben Supnik |
This morning I found out why certain plugins (e.g. SeaTraffic, etc.) are crashing X-Plane 10.50 beta 1 on startup. Beta 2 was being uploaded when I found this, but I’m going to fix this bug first and recut the beta, as it’s a big bug (since it renders many users’ systems to be dead-on-boot). Beta 2 should be up by tomorrow morning but might go live some time today if things go well.
The plugin-crash bug is a regression bug – X-Plane used to do something right and has now “regressed” and is doing it wrong in X-Plane 10.50 beta 1. This is our bug to fix, and we will fix it in beta 2; no plugin will need any changes, and the plugins crashing in beta 1 will start working again in beta 2.
That’s everything you need to know about the plugin crash bug. What follows is a very long and verbose write-up of the crash (think of it like an NTSB accident report). Perhaps it will provide plugin authors with some insight into where compatibility problems come from, and why early betas can be unstable.
A Proximate Cause: Loading an OBJ “Too Early”
We’ll work backward from the proximate cause (e.g. literally why did the plugin crash) and work our way backward to eventually understand the “why” of this crash. The fix for the bug is relatively simple, but you can’t know if a bug fix is correct without understanding the complete why; bugs like this take a lot more time to diagnose than to cure.
The crashing plugins are all calling XPLMLoadObject from their XPluginStart callback – that is, they are asking X-Plane to load an OBJ at the earliest possible time they can. This is totally legal! But it turns out that:
- X-Plane isn’t ready to load an object at plugin start and
- The penalty for doing so is a lot worse (a crash always) in 10.50 than in 10.45 (a crash rarely).
Why isn’t X-Plane ready to load objects? The answer lies in a little-known part of the OBJ file format called “conditionals.” Conditionals are a lot like C preprocessor macros – they are IF-THEN statements in an OBJ that let you make parts of your OBJ file only run in certain settings. For example, we use conditionals to remove the artist-drawn shadows from the static aircraft when X-Plane is in a mode that can draw shadows dynamically; this prevents “double-shadow”.
Conditionals are weird – they work not by evaluating the condition when we draw, but rather when we load; if a conditional is false in the OBJ, that text in the file is literally not used.* This means we need conditionals to work at load time, which in our case means when XPluginStart is called.
Unfortunately, conditionals are set up when we load preferences, and we load preferences after we call XPluginStart. So when these plugins load an object, the conditional system isn’t inited, and the OBJ loader crashes. Since we called the OBJ loader from a plugin, the plugin gets blamed. (A plugin author looking at the plugin crash can easily tell what happened – “I called XPLMLoadObject and it exploded!”.**)
Is this typical? Yes. Often it’s the combination of special rules for initialization, plugins using code in a way the sim doesn’t, and multiple subsystems that cause crashes.
Why Now?
When I found the code path that was causing the crash, my first reaction was “this is a mess…how did this work in X-Plane 10.45???”. So I went back to the X-Plane 10.45 code and found that this was broken in X-Plane 10.45 too! So what changed?
The answer is that in X-Plane 10.45, the conditional system would only crash (due to not being inited) if it was used; in other words, any object with IF SCENERY_SHADOWS (for example) would crash if loaded from a plugin at XPluginStart. But if you didn’t use the conditionals, things worked fine. And my guess is that very few plugin authors use conditionals in their objects.
What changed is that X-Plane 10.50 uses the conditional system for every object load, even if you don’t have an IF statement. So now the crash is 100% reproducible, not a rare “five things must all go wrong at once” kind of crash.
Is this typical? Yes. This is a bug where something was fundamentally broken for a long time, and an incremental internal change in how our code works changed the symptoms from ‘rare’ to ‘always’. This is very typical of beta bugs.
There may be other bugs like this too – it looks to me like objects loaded from XPluginStart with named lights might not get their named lights.
Why Do You Need Conditionals If I Don’t?
The next question this raises is: why did you monkeys change the OBJ loader code? What was wrong with what we had before? What kinds of X-Plane features cause this code change that break plugins?
The conditional code changed to fix a bug that shipped in X-Plane 10.45. The bug is: if you start the sim with HDR off and then turn HDR on, some spill lights don’t appear. The cause is that the OBJs are loaded with spill lights stripped out for performance*** (since HDR is off). In X-Plane 10.45 when you turn on HDR, we reload a lot of objects – but not necessarily the ones we need to, and the spill lights are lost. Rebooting brings the lights back.
To fix this, I modified the OBJ engine to track which conditionals any given object uses. If you add “DEBUG” to the end of your object to view the diagnostics, you’ll now see this in the output. When you change a rendering setting in X-Plane 10.50, only the objects that use a conditional that was affected by the settings change are reloaded.
That’s a big improvement in loading rules compared to X-Plane 1045. Turn on HDR and only objects with spills are reloaded; turn on shadows and only objects with optionally baked shadows are reloaded. But it means we need to use the conditional system on every object load to set up those flags; that’s what exposed the bug.
Is this typical? Yes! What we have is a simple refactor making different use of an internal API to fix a bug, and the results make a separate existing bug worse; that separate bug is reproducible only via plugins.
Is There an Ultimate Cause To All of This?
When fixing bugs like this I have to ask myself “is there a way this could have been avoided?”
The root of this bug is the root of a large number of plugin quirks and edge cases: XPluginStart (the first thing a plugin does) is called insanely early in X-Plane’s load process; as a result, a lot of the SDK isn’t actually available.
The decision to call XPluginStart early is a bad decision, and it is one that I made, well over a decade ago, to solve one specific problem. At the time, X-Plane (6!) had no option to save the selected AI aircraft to preferences. XSquawkBox needed specific AI aircraft loaded to support multiplayer, and it was really slow to let X-Plane load 20 random aircraft, then reload them all later.
To “solve” this (and I use the term loosely, since this one fix has been the source of so many bugs) I put XPluginStart before almost all parts of sim load, so that the XPLMAircraft API could let a plugin pick the AI aircraft before they were loaded, influencing the first load.
If I had a time machine, I’d go back to 2000 and kick myself in the ass. This early in load, virtually all rules of how X-Plane work are wrong since so much of the sim is not yet loaded. Plugin authors already cope with this by “deferring” their work until the sim has fully loaded; our advice for a while has been to not touch any aircraft before this point.
So is this typical? Yes – it’s yet another edge case introduced by plugins starting unnecessarily early.
* If you are wondering why conditionals work at load time and not run time, the answer is instancing. X-Plane does the complex analysis to categorize an object as instancing-friendly when it is loaded; attributes disable instancing. The conditionals run at load time so that if attributes like ATTR_poly_os are removed for shadowing, are removed by conditionals, then the object becomes instancing-friendly (because the conditional is like removing the text).
In other words, by having conditionals “pre-process” the OBJ file, you don’t pay for what you don’t use.
** If you call a plugin API from your plugin and the plugin code crashes, it could be X-Planes fault or your fault; plugin APIs may crash if given bad arguments (e.g. a junk pointer for a string argument).
*** This is the same idea as above; by removing stuff you don’t use like a spill light that isn’t seen with HDR off, we can get X-Plane onto a faster path. For example, if an object contains a spill light with a dataref, it can’t be instanced; if the spill light will never be drawn, deleting it makes the object instancing-eligible.
So…beta 1 is buggy as a bed in New York. I’m going to list these here in the hope of stemming the tide of bug reports:
These are broken in beta 1 and will be fixed in beta 2 – I’ve already fixed the code.
- The new autogen is missing! 🙁 There was a bug in the code that makes the installation; 10.50b1 diligently installs the old autogen.
- Crashes with some custom scenery.
- Missing taxi lines – they’ll be back.
- Floating objects – I did see this one during pre-beta; it’s now fixed.
The static aircraft are also not working; I am investigating this now, but I expect to have it fixed for 10.50 beta 2.
Finally, users are reporting crashes on startup with plugins; if you have this on OS X, please include the Apple crash report and not just Log.txt. I’m not sure what’s going on here, but it’s almost certainly a bug in 10.50b1 – we’re not changing any plugin compatibility rules, and the set of plugins affected is quite wide. (This bug is also not easily reproduced; I haven’t seen it with any plugins, it was not reported in private beta, and in one case the developer of a crashing add-on can’t crash his own add-on.)
Beta 2 will be out some time in the next few days.
