EXPERIMENTAL USE ONLY!!! -- [What?] -- Login to add public annotations [How? >>YouTube].

X-Plane 11.10 is the first major version release with a lot of code changes and new features. By design, this version has a lot of big changes in it that we’ve held back for weeks or even months. This also means it is more likely to be unstable and buggy, especially for the first few beta releases.

We’ve recently progressed to internal beta testing. Now we should have a public beta sooner rather than later, so I wanted to bring back out some great tips on beta testing that Chris wrote way back in 2012. Nearly all of the info remains as relevant today as it did 5 years ago, and I’ve included his best practice tips below.

  1. Keep two copies of X-Plane. If you care to ever fly X-Plane for enjoyment, ALWAYS keep two copies of the simulator on your hardware. One on a stable release that you use for enjoyment, the other on the latest beta for test purposes only. Your sanity will thank you later.
  2. Don’t use the beta for recreation time. If you’re only in the mood to fly and have fun that night, stay away from the beta copy unless you don’t mind your flight getting cut short. Too many users expect to fly for 6 straight hours and become enraged when the sim crashes on short final. That’s Murphy’s law!
  3. Always stay on the LATEST beta. Users often say “The latest beta broke XYZ, how do i go back to the previous beta?”. The answer is you don’t and you shouldn’t. That defeats the whole purpose of beta testing! During a beta process, the code is often changing very rapidly. If we’re on Beta 6 and you’re still submitting bugs on Beta 2, you’re wasting your own time and you’re not helping the product. It’s important you keep up with development.
  4. Blame us, not your system. If it worked in Beta 3 but it’s broken in Beta 4, do NOT tear your machine to pieces trying to figure out the cause of the problem. Report it and let US figure it out. Time after time, I see users change their drivers, install service packs, uninstall and reinstall X-Plane, update their OS etc trying to solve the bug…when it’s probably OUR bug, not your computer’s.
  5. Beware of the placebo effect! _EVERYTHING_ affects frame rate. Time of day, clouds, location, aircraft, view angle, view direction, addons, plugins etc etc. Sometimes we’ll release a simple patch that does nothing but fix one bug that was unrelated to the rendering pipeline. I’ll poke my nose in the forums to get some feedback and see 10 users who all of a sudden claim to see some massive fps increase in performance and 10 users who see a massive decrease in performance. I roll my eyes at the claims because it’s not a controlled experiment. For example, if you’re the type of person who flies with real world weather enabled, flying one day with clear skies may get you 60fps while the next day you may see 40fps with overcast. Heck, even departing from a different runway than the previous day can result in different frame rates. Running with the command line fps_test IS a controlled experiment. THAT’s the only way to be sure you’re seeing a change in performance. Don’t trust your instincts–gather data in controlled experiments.
  6. Always read the release notes for each new version. They’ll tell you what bugs are supposed to be fixed as well as what new features have been added. If you see a bug that’s claimed to have been fixed yet it’s still happening to you, that’s when you write a new bug report. If we didn’t claim to fix it, it’s probably not fixed in that particular beta.
  7. We need a Log.txt! When submitting a bug, ALWAYS ATTACH the Log.txt from THAT sim run. Even if you don’t think the Log.txt is applicable, attach it anyway. Tell us in as much detail as you can what you were doing at the time.
  8. We need different things for crashes on Mac vs Windows. Because of the automatic crash reporter, there’s no need to submit a manual bug report for crashes on Windows. If you remembered some details after the fact that you want to tell us, then go ahead and file a report. Of course, if the auto-crash reporter didn’t load up for whatever reason, that’s a good time to file a manual report. Mac users on the other hand, need to send us a copy of the Apple crash report. These files are located in your username/Library/Logs/DiagnosticReports/. The file name will have X-Plane & the date in it, and will end in .crash.
  9. Tell us who you are and what you know. We really appreciate users who enter their email and comments into the crash report form. It gives us a way to contact you to get more information. A handful of really helpful users can make the research a breeze.
  10. Be objective, not emotional. No one hates bugs more than we do but writing a rant instead of a bug report may make you feel better, but it makes us grumpy. It also makes us unlikely to take you seriously or follow up on your issue.
  11. Stop staring at the fps counter! It seems that many users are obsessed with their fps counter. “It went up this beta. It went down this beta. It stayed the same this beta.” Yes, the framerate is ONE useful metric to determine the software’s performance but it’s not the ONLY one and often it’s not the right one. Turn it off and fly! Enjoy the simulator. Sometimes there’ll be a bug in one beta that increases fps as a side effect because the sim is no longer drawing what it’s supposed to. Suddenly, some users are thrilled to see a 20% boost in performance and perhaps they don’t notice that half of the usual streets aren’t being drawn any longer. Ben fixes the streets and in the next beta, all of a sudden they lose their 20% performance gain and are outraged that we “broke things again.”

Being a beta tester is a bit of a thankless job, but we rely on our ambitious and courageous users to take the plunge and help us test each new version of X-Plane. Beta testers help us find those subtle or rare issues so that the sim is polished and enjoyable for everyone.

About Jennifer Roberts

Jennifer came to X-Plane to update the manuals and stayed for the bug testing. You'll most likely see her answering bug reports or making video tutorials.

86 comments on “Beta Testing 101”

  2. I agree with Bruno… must link to this directly from the .Org!

    It sounds like 11.10 is going to really knock our socks off, or knock our heads off our shoulders. (At least at first… 😉 ) Let the games begin (when yer ready)!

  4. Well you have my attention…… Great Info!

    When ever your ready to let the code out the cage for the wild simulator animals to tear into like it’s the last meal they “may or may not get to eat just yet,” let us know 🙂 ….. I’m looking forward to test diving again and sending in reports as needed.

    2. By the way, X-Plane always has two decimal places in the version number and hence version 11.1 does not make sense. If you think about the first version after 11..00 then that was 11.01 Now we are already at 11.10 while previous one was 11.05

      Also if we will reach version lets say 11.20 and 11.30 then they are not versions 11.2 or 11.3, they are versions 11.20 and 11.30

  9. Can we expect that the new SDK will be released simultaneously with the beta? Many thanks!

        1. Xchecklist is built with new SDK and tested on Linux X-Plane 11.05r2 waiting for 11.10b1 to complete my testing.

          Bill

  10. ready for testing 🙂

    Will the new “profiles” feature also allow to save “Control Response” and “Stability Augmentation” on a per aircraft basis?

  12. I don’t know how you do it! The combinations and permutations of variables that can affect performance and ‘bug free’ operation is absolutely mind boggling! You not only have to fine tune your product but also ensure that it works in harmony with L.R and the myriad of other third party addons. It’s like being a family member in counselling. Everyone needs to do their part to make things go smoothly and if one doesn’t it often screws it up for everyone else!
    You have a very interesting job though and I am grateful for your endeavours!

    1. We’re trying to work more kinks out with a small, private beta right now. Once we release the public beta, anyone will be able to test it by running the installer and checking the box to get betas. Steam users can opt into betas in the X-Plane 11 Properties > Betas section.

      1. How can we keep two copies of XPlane if we are Steam users?

        might be a dumb question.. 😛

        1. You can’t, really, but Steam allows you to quickly switch between the release and the beta version, and the Steam client will switch you reliably. With the X-Plane installer, down-dating from beta to stable is a hassle you don’t want to do.

  14. Ben said he could probably talk abut what is coming to 11.10. Do you still have plans to share a xp 11.10 overview about all changes, features in advance? or should we wait for the first beta release notes?. Very excited about 11.10.

    1. I don’t think we will have many more announcements before the first public beta. We have some marketing plans lined up to cover some features more in-depth as we go through the beta process.

    1. You could copy your X-Plane folder and then put a symlink to the custom scenery folder in one to another. You’d use the mklink command on Windows.

  17. Please work on the multiplayer connection please,some of us will fly with friends,like we used to in FSX, so make multiplayer more friendly rather then opening ports 49000 etc, please make multiplayer like any other platform ,look for a server and boom join in or create our own server in a way that anybody could join in without the need of knowing each other IPs
    Not like Vatsim,Ivao,Pilotedge etc,for that the average guy like me needs a lot of studying and time to actually enjoy flying in it,most of the time a bunch of friends only wants to get together and enjoy the xp11
    Reply

    1. Internet multiplayer is unfortunately not going to be supported any time soon. It’s a big project, and one we’d like to tackle in the future, but it’s not on the short list.

      1. Might not be for you to answer, but didn’t Austin say that one of the two new Laminar Research hires’ focus on networking/multiplayer?

        Obviously does not mean that this is the focus of development anytime soon though.

        1. Indeed! Jörg is our new resident networking expert. He’s been killing off networking bugs—mostly surrounding the weapons systems—for the better part of 3 months or so.

          2. he might want to look at why xp11 doesn’t work with the android version of flight plan go wile xp10 does

          3. Brief question on this, as was briefly addressed in one of the Q&A’s, will the issue with multiplayer DE-SYNC be worked on before the major overhaul of multiplayer? Many of us have been enjoying a port forwarded multiplayer environment, however since v11 release there has been positional sync issues for the visual representation of other players that was not present in most of the v10 run. It is not a connection issue as Austin seemed to suggest in the Q&A, it is some other as of yet unknown (to us mere mortals) issue. Has Jörg made any headway on tracking that one down? Its a rather big buzzkill for lots of us who like to fly in formation 😉 Cheers and keep up the good work!

          4. I’m not aware of anyone looking into this.

            The thing is… there are soooo many things that can go wrong between your computer and one across the internet. We don’t offer any official support for internet multiplayer for that reason. If you can track down the issue and show it’s not due to “the usual suspects” (e.g., high ping times between you and your internet-connected peers, DNS lookups times if applicable, etc.), we can look into it.

          5. Hi Willzah,

            If you can run the X-Plane 10.50 demo and latest X-Plane 11 on the same machines and you see different sync in formation flight between the two, please email Austin _directly_ to discuss what info he needs to sort this out.

          6. Hi, I saw there is a discussion on desync in multiplayer. I can confirm i have also seen this happen to me both with systems and aircraft position, those i’ve flown with have seen the same.

          7. Hello again! So I did what I would consider conclusive testing that it isn’t an issue with connections. I just tried it on all 3 home computers (1 MAC and 2 WIN) with clean demo installs V11 on our LAN (gigalan) all 3 systems had fps over 40, and default transmissions per second of 20 set in networking. The same exact issues are present. Desync for sure due to some unknown bug(other bugs present as well, altimeter/compass jittering, see here for example someone else’s vid https://youtu.be/f1-kojVE2FY?t=355). The strange thing is that when all clients pause, the aircraft go to their actual synced positions, but when the simulation is running it is very out of sync, and visibly stuttering. With regards to v10 I used it for hundreds of hours with many friends some as far as thousands of miles away (Norway, UK, ect. while I’m in USA) in the past and had no sync issues(albeit depending on the release version, there were “phases” of bugs there too). So I don’t really feel that would be necessary to test v10 again personally, because when it worked, it worked beautifully. Since others in my crew of about 10-20 online flight sim enthusiasts have certainly confirmed v11 has this issue independently of my own multiplayer testing, I think that the “cat’s out of the bag” for us at least. If you would still like myself and/or others to e-mail Austin directly about this we can. Thanks again guys!

    1. As soon as we can get it ready. Believe me, we’re working nearly non-stop to get there! But those pesky bugs keep popping up. 😉

  21. I wanted to ask: has the new particle system made enough progress that we may be seeing it in 11.10 or is that almost definitely not going into this update?

    I would say it is now on of the few jarring things remaining in XP11’s otherwise beautiful world.

  22. What’s the password for the release notes, where can we find it?

    Also does that mean the public beta is released this weekend or as soon as you work out the kinks?

    1. You can’t! 🙂 The people with private access to the beta at the moment also have access to the release notes. Everything will become public once we have a public beta. It doesn’t look like that will be this week at the moment however.

      1. It will be ready when its ready, Jennifer. No worries. 🙂 It’s going to be “fun” enough with the bugs that do get into 11.10 for the user community to find. Stomp away!

        “We will release no beta before it’s time.” (With apologies to Paul Masson.) 😉

  23. Hi, I can oly assume WED 1.7 will be released at same time, looking forward to teh new building blocks.

    Thanks

  24. Hiya, I noticed in the Cosford presentation there is mention of utilising Vulkan moving forward. For MacOS users, has any consideration been given to utilising a framework such as Molten (https://moltengl.com) to allow Vulkan apps to run on MacOS? Just a thought 🙂
    Thanks!

    1. Geezes! Someone’s been feeding Ringler! Ambitious plan–thanks for the link. Missed it as my main machine is down 🙁

  29. I’m excited about the updated physics adding more wind forces and supporting aircraft flying backwards to help with helicopters and tail slides, etc. In terms of aerobatic maneuvers and gyroscopic motion. Will the updates support more of this type of maneuvering? Or any future updates?

    It’s my understanding that many of these gyroscopic maneuvers are not yet supported?

    Thanks…

  30. Can anybody tell us what is the release date of 11.10? Is that day October 9, or 10, or 11??? Please write us an answer. Thank you!

    1. We can’t tell you because we don’t know yet. We’re still testing internally and we’ll release the public beta when we have it ready.

  31. Quick question. If i would like to test the Beta and at the same time to continue to have 11.05 release, how can i do that in a MacOS? Is it just copy the 11.05 to a different folder and run the installer from that same new folder?
    Thank you,

    1. On any OS you would make a full copy of your existing X-Plane install, then only update one of them to the beta version.

      1. Thank you. And regarding the digital license, do I need to do something to have the copy working?

          1. Is it legal to run 2 xplane processes with same serial? Same pc. I understand that’s not legal. Am I wrong?

          2. You would need two separate licenses if you want to run X-Plane at the same time on different computers, and network them together for example. It is ok to have multiple versions/installs of X-Plane on one computer and use one key for them all since you would only be running one at a time.

  32. While we are waiting for the 11.10 update, can you tell us which of the default planes will be equipped with the new G1000? Thanks.

    1. It was already confirmed that the SF50 will be changed to the G1000, the C172 will receive an additional G1000 variant.

  33. With the Vulkan implementation will the new ryzen cpu or intel new coffee lake have any advantages over one another? Thank yoi

  35. Do we need to download a new version of the X-Plane installer to install the upcoming X-Plane 11.10 beta, or can we use the same version of the installer that we used for 11.05?

    1. You will use the same, existing installer. You only need to change installers when we release a new version since it will contain bug fixes.

      1. * that is, when we release a new version of the _installer_, you will want to update to it to get installer bug fixes.

  36. Jennifer could you tell us please what day the X-Plane 11.10 could be download from server? The release day, please 🙂

    THANK YOU Jennifer

    1. I would guess that Jennifer can´t give you what Laminar doesn´t know.
      When the product is more or less ready depends on the number and severity of the new and known bugs, not on a date.

Comments are closed.