Having such a small development team, it’s very difficult for us to test ALL conditions in which X-Plane is being used. Each of our offices are littered with joysticks, laptops, desktops, loose video cards, hard drives and various other hardware. We do our best to always be making steps in the right direction but it’s a losing battle to keep our testing internal. A prime example is the recent Joystick debacle. I rewrote the entire Joystick subsystem on 3 platforms and tested it on all 3 platforms on all joystick hardware I own, and Ben and Austin did the same. We worked out all known bugs and then shipped it…and immediately we found hardware that caused crashes 100% of the time. Once those were fixed, other devices were still broken…some devices didn’t even meet USB compliance and we had to work around that issue as well. I can’t possibly own every USB Joystick device on the market so how on earth do we get the product to be as stable as everyone wants it to be?
The answer is simple…we rely on ambitious and courageous users to take the plunge and help us test it. Being a Beta tester is a bit of a thankless job. Sure you get an early look at the new goodies but it can be very frustrating since often, you can’t enjoy them without crashes, artifacts and other annoying bugs. Most users are happy to provide their feedback via the bug form, and now with the new automatic crash reporting system, we’re finding and fixing crash bugs in record time. A prime example is the Joystick crash. Within minutes of releasing 10.10Beta 1, I was aware of the crash and had all of the information that I needed to solve it. And within 24 hours, we had a new build that took care of it. But that’s only useful for crashes, it doesn’t help us learn about visual artifacts or other problems. That’s where we need YOU.
After reading through the forums, it’s become clear to me that many people are just not exactly sure what it means to Beta test software so I’ve come up with my own list.
- Keep two copies of X-Plane. If you care to ever fly X-Plane for enjoyment, ALWAYS keep two copies of the simulator on your hardware. One on a stable release that you use for enjoyment, the other on the latest beta for test purposes only. Your sanity will thank you later.
- Don’t use the beta for recreation time. If you’re only in the mood to fly and have fun that night, stay away from the beta copy unless you don’t mind your flight getting cut short. Too many users expect to fly for 6 straight hours and become enraged when the sim crashes on short final. That’s Murphy’s law!
- Always stay on the LATEST beta. Users often say “The latest beta broke XYZ, how do i go back to the previous beta?”. The answer is you don’t and you shouldn’t. That defeats the whole purpose of beta testing! During a beta process, the code is often changing very rapidly. If we’re on Beta 6 and you’re still submitting bugs on Beta 2, you’re wasting your own time and you’re not helping the product. It’s important you keep up with development.
- Blame us, not your system. If it worked in Beta 3 but it’s broken in Beta 4, do NOT tear your machine to pieces trying to figure out the cause of the problem. Report it and let US figure it out. Time after time, I see users change their drivers, install service packs, uninstall and reinstall X-Plane, update their OS etc trying to solve the bug…when it’s probably OUR bug…not your computer’s.
- Beware of the placebo effect! _EVERYTHING_ affects frame rate. Time of day, clouds, location, aircraft, view angle, view direction, addons, plugins etc etc. Sometimes we’ll release a simple patch that does nothing but fix one bug that was unrelated to the rendering pipeline. I’ll poke my nose in the forums to get some feedback and see 10 users who all of a sudden claim to see some massive fps increase in performance and 10 users who see a massive decrease in performance. I role my eyes at the claims because it’s not a controlled experiment. For example, if you’re the type of person who flies with real world weather enabled, flying one day with clear skies may get you 60fps while the next day you may see 40fps with overcast. Heck, even departing from a different runway than the previous day can result in different frame rates. Running with the command line fps_test IS a controlled experiment. THAT’s the only way to be sure you’re seeing a change in performance. Don’t trust your instincts, gather data in controlled experiments.
- Always read the release notes (http://wiki.x-plane.com/beta) for each new version. They’ll tell you what bugs are supposed to be fixed as well as what new features have been added. If you see a bug that’s claimed to have been fixed yet it’s still happening to you, that’s when you write a new bug report. If we didn’t claim to fix it, it’s probably not fixed in that particular beta.
- We need a Log.txt! When submitting a bug, ALWAYS ATTACH the Log.txt from THAT sim run. Even if you don’t think the Log.txt is applicable, attach it anyway. Tell us in as much detail as you can what you were doing at the time.
- We already know about your crash. Because of the new automatic crash reporter, there’s no need to submit a manual bug report for crashes! Let the system do its thing. If you remembered some details after the fact that you want to tell us, then go ahead and file a report. Of course, if the auto-crash reporter didn’t load up for whatever reason, that’s a good time to file a manual report.
- Tell us who you are and what you know. We really appreciate users who enter their email and comments into the crash report form. It gives us a way to contact you to get more information. This was invaluable to me when attempting to fix the joystick issues earlier in 10.10. A handful of really helpful users made the research a breeze.
- Be objective, not emotional. No one hates bugs more than we do but writing a rant instead of a bug report may make you feel better, but it makes me grumpy. And when I get grumpy, I attend seminars…and when I attend seminars, I feel like a winner. And when I feel like a winner, I go to Vegas…and when i go to Vegas, I lose everything. And when I lose everything, I sell my hair to a wig shop….Don’t make me sell my hair to a wig shop.
- Stop staring at the fps counter! It seems that many users are obsessed with their fps counter. “It went up this beta. It went down this beta. It stayed the same this beta.” Yes, the framerate is ONE useful metric to determine the software’s performance but it’s not the ONLY one and often it’s not the right one. Turn it off and fly! Enjoy the simulator. Sometimes there’ll be a bug in one beta that increases fps as a side effect because the sim is no longer drawing what it’s supposed to. Suddenly, some users are thrilled to see a 20% boost in performance and perhaps they don’t notice that half of the usual streets aren’t being drawn any longer. Ben fixes the streets and in the next Beta, all of a sudden they lose their 20% performance gain and are outraged that we “broke things again.”
Damned ! I can underline every word you say. Although my personal “feature list” for X-Plane is rarely filled with “Checked” Items, I must admit, that you guys are _doing_ your job. Slowly but surely (where “slowly” means a compliment, nobody wants a piece of software, that was rushed without thinking)
For me its kinda refreshing experience, that we users are somehow part of the develop-process, not like years back, where simulators (lets not talk about names or companies) were smashed on the market with some “take it or leave it” mentality…
At your number 2.) I disagree. ONLY when creating a very personal setup with flightplan, weather, plane, ATC and all the stuff that is in my fantasy, there is enough room for a random stone i may stumble over accidentally (a bug). Being aware using a Beta piece, I dont mind crashing on short final. The counter side -for me- would be using a “stable” version where all the exciting new features are missing? Boring…
Greetings
For clarification, does this mean that reading the FPS counter in the corner of the window is NEVER a preferred method of reporting a potential bug? For example, I recently submitted a bug report describing a FPS drop from 40 (forty) to 3 (three) after turning on HDR when sitting at a grass landing strip in the middle of the trees in the mountains (i.e. no lights, roads, buildings, etc.). To me, that was a big hit. Will it make your lives easier if we ALWAYS submit FPS results that have come from the framerate tester, regardless of our opinion of what constitutes a FPS-creaming bug (whilst doing our best to quell the tears of joy or sorrow that occasionally manifest during beta testing)?
This is not a hard rule, my intention on that bullet point was that the average person simply cannot trust the FPS counter as a sole indication that one version is any better or worse than another version. Only a controlled experiment like the FPS test can compare apples to apples. No matter how hard you try otherwise, I think the best you can accomplish is comparing Macintosh Apples to Granny Smith.
Using your bug as an example, if you tell me your fps go from 40 to 3 after enabling HDR, I don’t get excited by that because those are absolute numbers, not relative numbers. If you told me that in Beta 3 under identical conditions you could run HDR at 20fps but now in Beta 4 you only get 3, that’s something substantial…in that case we have a reference to go by. For all I know without a reference, maybe an earlier beta got you 1fps in HDR mode meaning you got a 200% gain. 🙂
frame rates are one of those things that shows up in many different Hobbies that are used as a performance gauge . In flight simming its FPS in Amateur radio its SWR (standing wave ratio) . sometimes you just have to forget sbout them and enjoy .
Your article practically qualifies as a rant. Congratulations! Nice read. Always stay with the betas indeed… some folks don’t know what they are missing. The TEN run has been great for me — a joy. Yeah we all whine from time to time, but from my perspective it’s never about the big things but the little things, like why does this manip/light/thingy not work as good as before. Of course, when it comes to the big things like a joystick suddenly doesn’t work or the sim doesn’t load, that’s a different matter.
If you had to sit and read through some of these bug reports, you’d rant too. 🙂
Chris, I thank you for this post!
As one of the “brave and foolish” always testing your betas, I got used to nearly everything about bugs and strange behavior. But that only shows how complex the sim is and how tight things are intervonnected to each other.
The bug reporting system is a great gift, making it easy and reliable to test your software with the knowledge of helping to make things better.
I can absolutely understand Your opinion on framerates, but I understand the ramps of users, too, as I am a user, too…
Nothing is more frustrating than to poke around with rendering settings and testing and trying for hours to get higher framerates – or usable framerates. The thing is, nothing keeps the fun out of you that much when you look at a stop motion film while trying to have fun (I’m not talking of betas right now).
People would accept lack of functionality and minor bugs alot more if they could easily find rendering settings that worked for their system.
So what about taking the framerate test system and pushing it to a framerate calibration system? I personally don’t care to let my system run 2 days in a row and let it find the best settings (which would be a min of 20 fps under the worst conditions, so full weather, my preferred airplane, my desired resolution), but I absolutely do care about sitting two days in front of my pc and doing all the little “lets lower this a bit and fly around and see what it does under condition xyz” things.
Giving an automated-benchmark-render-settings-thing to the users would stop the rumors about framerates immediately, at least in my opinion.
It is not the fact of having too low framerates what causes the users to get angry, it’s their inability to find good settings for their system quickly without spending days of time.
Anyway, please keep on going, you are definitvely moving into the right direction (as you are doing it all the time), but never forget that developers and users have (sometimes) different needs and wishes.
All the best
Marc
Would it be possible for your bug reporter to return an identifier for each submission? At the moment, if I think of some extra information that might be useful, I don’t know how to refer to my previous report.
This is something that Ben requested as well. We’ll look into it.
only, the fps_test does not work in 10.10 so far. Bug reported
It works fine, it just requires arguments that you’re probably not passing correctly. I think we need to update the docs. Perhaps I can convince Ben to write a blog post on the topic too.
Evening Chris,
Thanks for taking a moment to encourage us to step far away enough from our own isolated experiences to look at the global picture, literally. Some of us enjoy journeys, and others enjoy destinations. Beta versions are a privilege to those in the former category, yet a frustration to those in the latter. Regardless, eventually everybody wins.
Al.
Very nicely said.
Hi Ben,
Totally agree with users being obsessed with the frame-rate counter.
I’ve missed many a nice view from starring at it. Sometimes, my wife gets jealous and asks me if her frame rate is as good as the one I keep starring at. I tell her that nothing compares to hers, but you know what women are like!!
Keep up the good work mate.
Dom
oops typo, LOL
Can I ask something for your beta page?
POST YOUR BLOG UPDATES WITH DATE (and maybe even time)!
Is there a reason we don’t know when something was posted?
There’s no particular reason other than not really seeing the need. How would that help users to know the date that old Betas were posted?
How large can attachments be with the bug reporting tool? I recently had zipped some screenshots in order to show some strange graphics that only showed up with the fighter planes. That was a big file and the summary page of the bug reporter did not appear.
I’m not sure off-hand. It’s probably whatever the PHP limit is…10-15MB perhaps.
Wouldn’t that issue be solved by uploading your screenshot to something like Imageshack or Photobucket and just posting the links to the image instead of trying to post the entire image itself?
don’t know. whatever is more convenient to Laminar. There is no statement about attaching screenshots any way. I just tried.
From the bottom of the sample bug report page:
“If the bug causes a visual problem, please take a screenshot. Please send the actual PNG file X-Plane creates; do not convert it to JPG or resize it. If you have a paint program, consider circling the problem area in “red pen”. This will make it completely clear to us what is wrong with the screenshot. (It is not always obvious, particularly if a screenshot shows two bugs — which one are you reporting?)
You can attach up to four files (up to 4 MB each) in your bug reports. If you need to attach an archive, please use ZIP, not StuffIt or RAR.”
You have those strange white out effects to ?
Thanks for the LR perspective on my favorite hobby “beta testing”, and the into to command line fps_test. Question do you have any LR standard situation files (*.sit) for the latest beta’s you could let us have (I think there used to be some somewhere, but don’t know if they are still compatable with the latest beta)? Also anything like a standard LR approved weather file METAR.rwx (I use my own at the moment).
Thanks again, for such a fun hobby. “What do you mean I must be mad!”
cessna729.
p.s. I’ve got another WD VelociRaptor waiting for Beta5!
The fps tester can work with the .fdr files that ship with the sim – I used .fdr files to avoid having the fps test route be broken when the replay format inevitably changes.
I am interested in “command-line FPS test” but the linked page does not have V10 package. Would you please add that? Thanks.
Yeah I realized after the fact that the link was kind of stale. Stay tuned for a blog post on the topic.
Hi Chris
Software development can be a very thankless task, especially when your software is available for sale! I guess it’s about the consumer mentality… if you buy a car you expect it to get you from A to B and not break down or you send it back to the manufacturer. Although I understand that to a degree, being a programmer myself i really feel for you guys. You have the goal of making the best flight sim package available and you have evolved a culture of continuous improvement to reach that goal. You said yourself that you are a relatively small development team and, no, of course you cannot be able to pre-empt every failure scenario.
I’m glad you made this last post because I hope that it will open the communities eyes as to what really is involved in software development and what beta software testing actually means. I rarely post but I do follow this blog and I notice many moans and groans about what needs prioritising, why not 64 bit, why is that tree coloured blue etc? The fact is, and what people don’t understand, it’s a bit like building a tower block. Before the pretty lights and decoration can be put on the penthouse you need a very solid and reliable base to build off.
I’m guessing version 10 was nearly a complete rewrite for you guys and because of that your finding new sets of problems to overcome (the joysticks) because of the new methods involved. But people, this all adds to making those foundations the very best they can be so the icing at the top will be that much sweeter.
Hopefully now people have read your post chris there can be much more constructive feedback in the spirit of continuos improvement! I for one (and I’m sure one of many) applaud your efforts and wish the whole team the very best and will be sure to send you constructive comments whenever something comes up.
Keep up the good work!
Beta’s are fun as you push the limits, but sometimes it pushes your limits as well, It is not hard not to notice the frame-rate drop when it goes from 50-40 frames to 6 or 7 frames and mostly when you look out of the window at custom scenery with a VC panel, then speeds up when you look lower on the panel then totally crashes (you have my auto-reports) when you look up through the window again, one minute you have everything and then you have nothing, This has been the issue through out Ver10 and not just the latest betas, start up XP again and it will run smooth as silk with high framerate, running “out of memory” I think it is called .
But this combination of VC cockpits and Custom scenery is making the platform unstable and masking other issues….
And other beta testers should be aware of the issue, don’t get me wrong a lot has been sorted but this one never seems to be….now in the betas this issue been highlighted more than ever.
another thing maybe off-topic: recently I came up with some suggestion (even with a bit of code in C#) about the flow-Items (wind rules) . Would be nice to know where/to whom I can adress this?
Ron
Are you talking about the new ATC flow feature?
exactly, the new ATC flow feature !
You can email it to me directly if you’d like.
Good post and thanks for all your had work!
just a suggestion, now you’ve got that superb automatic crash reporting system setup could you not do the same thing for bug reporting within the main program? For example, in the x plane menu select bug report, we fill it out and send and the reporter automatically collects the files you require to monitor and send with the report. It’s not that we are lazy or anything but it would be nice if we see a bug we can report it there and then and you’ll have the last save of the log file etc. You could at least do this for the beta’s?
I’ve thought about this as well. I’ll keep it in mind in the future. We can certainly expand it’s usage.
I agree. In addition to attaching the log.txt, perhaps another automated bug report could be one that can include screen caps. Like you choose “file graphical bug” then it tells you to hit a key for screen cap. Maybe it even overlays a red ellipse for you that you can position and stretch. before hitting the capture key. Then you can accept/reject pictures until you have up to 4 (or skip the rest if what you have is enough).
The next dialogue could then just have the free form text (filled in where appropriate–like OS version and X-Plane version). You type in additional info, and hit submit. Bingo bango bongo!
just installed mountain lion and ran xplane. seems to work better and a little faster no crashes.