You know you’ve got a good bug when a user reports that his airplane’s wing disappears when he turns the battery switch off. It turns out that this is the same bug as the 747 rolling over to the side.
X-Plane 10.10 (beta 1 all the way through RC1) has a bug in the IO code that scrambles the electrical system of airplanes on load, sliding the electrical system selections on the “bus 2” page of Plane-Maker over by one slot (except for the last two columns, which get totally scrambled). Then, due to an accident chain that would make the NTSB blush, the results pop out in the flight model, often as an incorrect roll tendency for airliners.
This (and a number of other bugs — thanks to all of the airplane authors who tried their planes and reported bugs against rc 1) will be fixed in rc2, which will hopefully come out “real soon now™”.
If your plane is saved in X-Plane 10.05 format, this bug won’t affect you – the fix to X-Plane will cause your airplane to just work. If you have already saved your airplane with Plane-Maker 10.10, you may have to re-enter some of the amperages and bus choices for your electrical system – the amount of data scrambling depends on how you used Plane-Maker. The good news is that the damage is limited to the second electrical page so at the very worst, you’ll have to re-enter some electrical system data using rc2.
Trusting Beta Plane-Maker
This bug brings up the question: how much should you trust a beta or RC Plane-Maker? The short answer is: “not very much”. Here are my recommendations.
- Never, ever, ever release an airplane against a version of the sim that hasn’t gone final. Things do change in release candidates, sometimes major things when we find a bug like this. Wait until the version is declared “done” before you release your airplane!
- While always saving backups of your work is always a good time, it is especially important when using Plane-Maker betas. Assume a beta Plane-Maker might erase the .acf you are working on entirely; while we’ve never had it do something that bad, it has produced incorrect ACF files before due to bugs.
- Beta Plane-Makers are good for testing and trying new features and experimenting, but not for production work; wait until we go final to permanently change tool chains.
- The warnings about beta go for release candidates too!
I am working on a v9 -> v10.10 checklist for airplane authors; the actual “busy work” of editing the ACF file should be less than 30 minutes per airplane if you know what new values you need to enter for parameters that need updating, so a reasonable work-flow might be to experiment with the new features and report bugs until we go final, then make the actual update on a fresh copy of your plane from an older version.
How Did We Miss This?
The sobering thing for me is that this bug has been in our betas for four weeks and (1) we didn’t notice it and (2) no one reported it. I take a few things away from this: clearly we need to test our own planes more carefully. In 10.10 we did a lot of detailed work on our own fleet, but ironically this hid the bug from us. But I also take away from this that a lot of authors don’t even look at the build until RC.
If you make an add-on, I encourage you to at least look at a few betas before RC. Even if you don’t retest every one, taking a peak early means we can fix bugs that affect your add-on early, which is good for everyone.
What is the last stable release of x-plane X?
as of this writing: 10.05r1.
“real soon now™” oh no’s, that’s a very taboo word around Impatient Chris.
I actually noticed this bug earlier in the 10.10beta run, but I couldn’t think of anything that could’ve caused it, I checked some other planes and these didn’t show this behavior.
So I played around in Plane-Maker, removed wings, brought them back and looked at the electrical pages. The bug went away from my aircraft.
But it came back with the RC1(if I remember correctly) and I thought: “What, you again, what have I done?”
Sorry Ben, next time I won’t try to workaround potential bugs!
Yep – if there is a lesson here it is: “report it, don’t work around it”! And this bug is a classic illustration of why: basically the only people still affected by the bug are people who worked around it.
(Which includes us — we updated our own planes because we thought the esys damage was due to an old bug in 10.00 and not a new one. Then we realized what we’d done and I had to go reset the esys pages again.)
Ben, one of the things I really appreciated back in the days of yore when I was a programmer analyst at a major credit card processing firm was the QA department. I could test my own code to the 9’s and they would still find stuff. Laminar is a lean mean coding machine, but with the growth and complexity of X-Plane, at some point do you think you’ll actually establish a formal QA department? I shudder to think at what regression testing of the entire code base would be like. But there’s nothing like a little formality to catch a ton of bugs before they get out the door. I know this would really cramp Austin’s style, and in light of other recent shenanigans over which you guys have little control it would be hard to even consider adding another mouth or two to feed. But at some point the annoyance of a QA cadre will really save your bacon.
I agree that having non-engineering test would be a win. At the same time, with so few people, every head count added is a big deal, and it’s hard for us to add “specialists”.
At a major firm, the capitalization exists to fund a QA department.
Given the size of X-Plane as a business entity, I think they took the only step they could and that made sense. Providing an open Beta program still accomplishes the required task of having unfamiliar eyes (to the underlying code) catch the issues for which the programmers are too close.
We’d do an open beta even if we had dedicated QA. When you have one programmer QA is an impossibility. When you have ten, it’s a must. Somewhere in between life is just awkward.
Laminar Research: the pimply-faced teenager of engineering organizations. 🙂
Getting involved in a beta program gives me a better appreciation for what it takes to get a complex product out to market.
I enjoy it, pimples and all!
A “department” here? No. A dedicated dude? Yes…well, maybe. Open beta’s are not regression tests. The frequency with which something unexpected comes along with something new and good is part of X-Plane lore. I think the key advantage would be a shorter beta cycle, allowing the more fun stuff – the new stuff – to get coded sooner. No one likes fixing bugs, and not many users enjoy finding them, especially when they slip through the open beta. I’m not arguin’ here, just raising a timely point at a timely time. At some point QA will be affordable, especially when Laminar is not in the middle of being sued. 🙁
The flip side of shorter betas is that in each cycle we only get fraction of that time to put up real features without the circus of a beta. So for 10.00 -> 10.05 where we were doing bug-fix patches it was really hard to get anything done. 10.10’s cycle ran longer than I wanted it to, but not by a lot. I think an 8-week dev cycle followed by a 4-week beta isn’t a bad aspiration.
Injecting a QA person with the resulting shorter betas would certainly disrupt “the way things are™.” 😉 Frankly, I know it would be like moving to Mars! And we don’t want to do that. Not just yet. No criticism intended, by the way. Just being an olde farte. I’m really looking forward to 10.3 – 4 or so. You’re dancing as fast as you can right now, and you’ve got a really big lady on the card for 10.2x. But in the middle of the run is where you guys really seemed to hit your stride in 9, and I’m optimistic for the future of 10. 😀
does that possibly mean 10.20 wont show till november? if based on 8 week dev, 4 week beta
As I have said approximately 300 times already, I am making NO statements regarding the future dates of ANY releases. Please do not ask again, either directly or indirectly.
Yeah i caught my self asking something dumb after submission. My bad
I Can’t see the sense in shorter beta cycles either. To my mind the whole point of beta’s is to eradicate known bugs and allow new bugs to be found and eradicated, which is where the focus should be. Shouldn’t new features be added after the bug run has finished and a stable build established? Then the new beta run would eradicate the bugs in the new features?
Hi Nick,
90% I agree re: keeping bug and feature dev separate. The one exception is: beta gives us a rare opportunity to rapidly iterate with users and third parties. So for a feature where I want a lot of user feedback I might put a prototype in beta 1 and then revise quickly, taking advantage of the beta to easily get new builds to lots of people. But that kind of thing shouldn’t be happening in late beta, and it’s not an excuse to not have features done.
This is clearly off topic, and for that I apologize but I couldn’t find anywhere else to put it.
We haven’t seen any news on the ATC since February. I was wondering if you guys might have any commentary on the ATC? I’m not asking for a features roadmap or anything of the sort. It’s just really fun to follow the little updates on scenery as the process roles on, it would be fun to see the same thing with the AI development in ATC…
ATC is and probably always will be on the roadmap for new features but we have to balance our focus in the same way you don’t work on only biceps in the gym or you’d end up with giant arms and skinny legs. It’s a lot like the scenery system too in that it will continue to evolve and grow in realism over time but what’s next and when it’ll be here is still unclear at this moment.