In a past life, before I worked for Laminar Research, I was a plugin developer. So I’ll admit up-front that I might be the most guilty individual when it comes to this particular problem: log spam.
What do I mean by log spam? By log spam I mean printing information to Log.txt that isn’t necessary.
Log.txt exists so that we (and third party developers) can look at a single machine-generated file and get a strong idea of what a user did with X-Plane for diagnostic purposes. Therefore log.txt needs to contain three kinds of information:
- Basic configuration information. We print your graphics card, your OS, etc. (In 10.30 we’ll even get your CPU type right on OS X. 🙂 It’s more reliable to log this than to send a user into ten different control panels to gather information.
- Very basic event logging, e.g. what did the user do? Where did he start his airplane, what scenery was loaded.
- All abnormal conditions, including error alerts that came up.
In particular, what we do not need to do is log the detailed blow-by-blow progress of loading in excruciating detail.
If I may use my own sins as an example, older versions of XSquawkBox shipped with debug code on that dumped the contents of its entire ICAO dictionary – thousands of lines of text. This information isn’t configuration (it’s not unique to particular installs), it isn’t an event – it is part of the event of loading XSquawkBox, and its not an error condition. What a mess! (Fortunately Wade has fixed this and turned such logging off in the newest XSB betas.)
Programmers write this kind of logging code to help trace program execution, but it’s important to turn it off – if we don’t, the log files get so verbose that everyone spends a ton of time just fishing through ten tons of junk.
Some more guidelines:
- Log the presence of files or sub-modules that are user-configurable, but do so briefly. It’s reasonable for SASL or Gizmo to list scripts that do not come with the base package, or for XSquawkBox or Pilots Edge to list CSLs found. (This information lets a developer see what a user has done to modify the plugin.)
- Don’t log the presence of required files unless an error condition occurs. An empty log can simply mean “everything went okay.” (The sim will already confirm successful plugin loading on a per-plugin basis.)
- Log rare events, e.g. a flight starting, a network connection being opened. Don’t log events that occur at high frequency, like receipt of a single network packet.
X-Plane is guilty of log-spam too; I’m slowly trying to cut down on this. For example, spurious autogen-tree warnings are now gone from the log in 10.25r1.
Support Debugging
Sometimes you need all of that logging. My suggestion is: if your add-on is complicated, make a specific interface to get verbose diagnostics that users can turn on as needed. For example, X-Plane has an option in “operations and warnings” to dump verbose play-by-play network data to Log.txt. When you enable this, you can get a ton of information about network operations, but the log will also be totally over-run. The option is meant only for developers who are integrating with X-Plane’s UDP networking, and thus it has to be turned on by hand.
The OBJ engine actually also has such a trick – if you put the single word DEBUG (all caps) at the end of your .obj file, the entire OBJ contents will be output in detail to Log.txt. Again, this is a ton of information, available when needed, but off by default.
You can support verbose logging in your add-on with a script variable, a dataref, a change to a preferences file. Pick an easy mechanism, so that you can have users turn on verbose logging only when they need it.
Not Everyone Has Grep
Finally, a quick aside to my fellow professional programmers. You might be rolling your eyes and going: “Ben you idiot, it’s easy to filter the log. Everyone should just prefix their log entry and use grep.”
If we were all programmers, I would agree 100%. But please understand that aircraft and scenery authors have to look through the log to get diagnostic information about art-asset problems, and this is an audience that is sometimes not comfortable with command-line tools, huge 25MB text files, massive amount of text, etc. So we are trying to keep the log.txt more human-readable than, for example, dmesg on Linux.
X-Aviation just posted an update to SkyMaxx Pro – the new 1.1 patch brings big performance improvements and fixes rendering problems with HDR. From the user reports I’ve read, performance in HDR mode is good. (I hate to see users have to pick between HDR and third party add-ons; we want HDR to be the basis for superior next-generation aircraft and scenery.)
Getting a plugin that draws in 3-d to work with HDR mode requires some caution; the plugin APIs for drawing were designed in X-Plane 6, when ‘deferred rendering’ didn’t even exist as a concept.
I have updated the X-Plane SDK tech-note on 3-d drawing to contain modern guidance on how to cope with 3-d drawing callbacks. In particular, you get called twice when your plugin requests to draw in 3-d:
- The first callback is for drawing “solid” stuff that will be affected by spill lights; the only safe thing you can do from this callback is to call XPLMDrawObject.
- The second callback is after lights have been mixed; at this point normal OpenGL drawing works reliably (albeit without spill lights being mixed in). This is the time to draw translucent prop discs, coach marks and labels, clouds, smoke and particle systems, etc.
If your plugin does any 3-d drawing (e.g. custom particle system drawing or any kind of effects code), please review the tech note, and email me if you have questions. The next-gen CSL code sample that is linked from the article is tested and works correctly too.
Update: this bug has been fixed – see here for details.
For X-Plane users with an AMD graphics* card on Windows or Linux: there appears to be a bug in the 13.11 drivers** that cause corruption in HDR mode; the symptoms are square artifacts of missing clouds and missing windshield in aircraft.
If you can run the 13.9 drivers, stick with those; if you have one of the new Rx 290 cards and you must use newer drivers, turn HDR off for now.
These bugs are Win/Lin only – the Mac AMD drivers don’t have this problem.
* AMD seems to have fully rebranded ATI’s GPUs at this point, so I guess I should get used to writing ‘AMD Radeon’.
** The bug could very well be in our code, but since the code worked on old ATI drivers and a number of other driver stacks, it may very well be a regression in the driver itself. We won’t know until it’s fixed. And even then, maybe we won’t know.
Posted in News
by
Ben Supnik |
The HDR check-box turns on HDR mode in X-Plane 10. But what is HDR mode?
High Dynamic Range( HDR) mode in X-Plane actually enables many features at the same time. We put in a single check-box because the features all fit well together, and we wanted to keep the interface simple.
- High-dynamic range rendering. In HDR mode, the world is rendered across a wider spectrum of brightness (just like real HDR photography)*, to better capture lighting levels. The result is then compressed to low dynamic range (which is what your monitor does).
- Tone mapping with real-time exposure. The HDR image is converted to LDR via a tone mapping algorithm that attempts to preserve the character of the image while keeping the dynamic range within limits of your screen. The level of exposure changes in real-time: if you look at a bright scene and then duck the camera behind a building, you’ll see your eyes slowly adjust up, then everything gets dark when you stare at the sun. (The level-adjustment algorithm runs entirely on the GPU for performance.)
- X-Plane uses deferred shading for all rendering except for translucent glass in aircraft. Deferred shading is a 2-pass approach to rendering that is commonly used in today’s shooter games, because it allows us to efficiently draw huge numbers of real 3-d spill lights. In the case of X-Plane, you get a lot more night lighting.
- The individual puffs in our 3-d clouds have soften intersection with the terrain to avoid ugly cloud-ground collisions in low-cloud conditions. (See page 1 and 4 of the article for before/after pics.)
- We emit heat blur from the engine particles. The ability to add blur effects is a capability of the deferred rendering engine; something I’d like to do more with in the future.
- Optional atmospheric scattering. This effect enables a more advanced lighting and fog shader that runs on the GPU and models the diffusion of light through the atmosphere as it hits oxygen and water particles. Atmospheric scattering is what makes the far view ‘bluer’ than the near view.
So the moral of the story is: the HDR check-box actually enables a lot of effects!
Why Did You Put Everything in One Check-Box?
We had a few reasons to put a lot of effects into one bucket:
- Some of those effects require other effects. For example, you can’t have the heat blur or soft cloud puffs without the deferred renderer.
- We wanted to keep the UI simple. Our view is that the rendering settings screen is way too complicated and hard to tune, something we’re still looking to improve. So any time we add features, we have to ask: can we make this simpler?**
- There are fewer combinations for us to test if there are fewer settings. Sometimes we can have bugs based on combinations of features, so having the features all enable at the same time simplifies development of testing, which is extra important when we have to test GPU features against different hardware and drivers already.
* GPU nerds: The final render surface is in 16-bit floating point format.
** If you really want to get tweaky, most of the sub-effects within HDR mode can be accessed by settings.txt. But if you edit settings.txt and your computer transforms into an alien cyborg and destroys humanity, don’t hold me responsible.
If you’ve survived downloading the rather large 10.25 patch and you want to download even more data, good news! Alpilotx has just released version 2 of his X-Plane HD Mesh Scenery.
His scenery pack is a donation-ware recut of North America and Europe. Alpilotx’s HD scenery use the latest OSM and other global data sources, and are cut with the latest DSF generator; thus they contain better DSF generation with newer data. Alpilotx then cranks the mesh settings well beyond what we can ship with the global scenery (which must fit on a DVD set and not fill an entire SSD), bringing out wonderful terrain definition and other detail that we have to cut down for the global scenery that ships as a default with the sim.
One of the cool effects of cranking up the base mesh detail: it brings new detail to the terrain shading. In X-Plane 10 we moved the process of altering terrain with slope (from forest to grass to rock, or whatever the artist defines) from a CPU process calculated ahead of time to a shader process run on the GPU. The result is that more detailed meshes actually produce better shading, as the shading responds to the 3-d detail.
If you have a computer that can run them, these meshes look great, and they get you a source of recut DSFs now.
The recut DSFs that Laminar ships will be based on the same DSF generation code improvements and the same new data, but the resolution will be reduced to produce files of similar size to the default scenery we already ship. We have not decided exactly how we will ship recuts; I think that really important fixes (e.g. Sydney with the harbor fixed) will be automatic updates; I don’t know about a wider scale yet.
Posted in News
by
Ben Supnik |
X-Plane 10.25, out now, is a big update by size – 1 GB compressed, if you were current to 10.22. The result on our servers is, well, higher than normal load. (That nearly flat top-out at 100 mbits is 10.25 kicking.) We’ll keep an eye on things; I expect that server load will settle down in the next few days, once the initial rush of auto-updates goes through.
Posted in News
by
Ben Supnik |
X-Plane 10.25 went final today; if you’ve launched X-Plane you already know that from the auto-update notice. With this update we shipped the first set of lego-brick airports, and new urban and natural terrain. We also updated WorldEditor to version 1.2.1 during this beta run.
The Necessity of Saying Too Little
This post, like all “road map” posts, will almost certainly be too vague for your liking. What you’d really like to see is a list of features and a list of dates, e.g. “we will ship X on this day, then Y on this day” for future X-Plane patches. Unfortunately I cannot provide that. I also can’t expect you to be happy with only vague release notes. But I have to hope that you can at least see what things look like from the other side of the table; if you were in my position, you might not post too much either.
There are a few problems with announcing new features way in advance:
- The penalty for being wrong is a lot higher than the benefit of being right. If I can predict our future features directly, at best everyone gets excited and impatient for the new release. But if my prediction is wrong, it will make everyone very angry. So if there is any risk that a feature might not come out when we think it will, I think everyone is better off without pre-announcements. (We are not the only tech company or even flight simulation to maintain this policy.)
- It is really hard to predict when new features will come out for technical reasons. Or put another way, if what we are trying to do is new and novel, there aren’t good things to compare to in order to estimate schedules. We thought we’d ship real-time shadows in X-Plane 9.0, but when 9.0’s release date came around, they just looked awful; the technology we had picked was not good enough. So we put them on ice, re-coded them and released them for X-Plane 10.0. That kind of thing is hard to predict; we had working shadows for v9, they were just horrible. In hindsight I’m really glad I didn’t pre-announce “X-Plane 9 will have real-time shadows” – everyone would have been disappointed by the lack of shadows, and it would have drawn away from the real-time water reflections we shipped.
- It’s also hard to predict when new features will come out for business reasons; if a feature isn’t being worked on “right now”, there’s the risk that some kind of high priority emergency will come up. For example, during the X-Plane 10.25 beta run, we realized we had to recut our installer, because it had a crash bug on certain OSX versions. That time wasn’t in the estimate, but the installer crash was an emergency. Had I said “X-Plane 10.25 will go final on November 15th”, I’d have been wrong, and you’d be mad that we had mislead you.
So my first point is that software is unpredictable, and I think it’s better to keep quiet until we’re sure we know what will happen. Thus, no statements about what we will release six months from now.
Guidance Is Important
If you develop third party aircraft, you’re probably jumping up and down at this point, going “are you freaking kidding me? I can’t plan my business this way!!!” And you’re right. What I can try to do is provide guidance about technical challenges that you might face as an author.
Our overall goal is to maintain add-on compatibility within a major version run. If your add-on follows the rules, we’ll try to avoid changing the rules. But sometimes add-ons bend the rules, and updates can affect those add-ons. I try to provide hints as to what is going to matter, far ahead of time.
For example, X-Plane 9 introduced the concept of ‘glass’ objects in an aircraft. Glass objects solves the problem of getting your cockpit windscreen and 3-d interior to render in the right order, which is nice. But it also is necessary to get high quality translucency in HDR mode in X-Plane 10. So we tried to get the ‘glass’ option into v9 and then encourage developers to adopt the new scheme, as it provides both better results and future proofing.
Big and Small Patches
Our patches to X-Plane come in two sizes these days:
- Big patches have long betas (4 weeks or longer) and make significant change to the sim; we need the beta time to find and fix bugs introduced by the new changes.
- Small patches change a very small amount of code; we test them for only 1 or 2 weeks. Because the amount of code change is very small, we don’t have to worry as much about strange bugs we didn’t expect.
X-Plane 10.10 was a (very) big patch, and 10.20 was a big (albeit not very chaotic) one. The bug fix releases (10.11, 10.21, 10.22) were all, of course, tiny – they just fixed 1 or 2 bugs that made it past beta.
X-Plane 10.25 is actually a small patch by our standards – while it provides a lot of great stuff for users (airports, cities, etc.) and it’s a big download data-size, the code changes are incredibly small; it’s mostly art files.
Future Releases
At this point our next “big” release planned is 10.30. We have a ton of stuff planned for it; it’ll be a big release, offer significant new features, and probably take quite a while in beta.
If you are a third party aircraft or scenery author, you will want to re-check your add-ons on 10.30. I know that this takes time away from working; we try to not have a “retest everything” patch that often; the last one was 18 months ago. Remember that if you find something broken, you will need to report a bug, not change your add-on!
(As a side note, I hear this from third party developers all the time for bugs that have been in the sim a while: “You mean no one else reported this bug?” Everyone thinks someone else will report a bug; some bugs that are visible only on third party content thus never get reported. Assume that no one has reported a bug – you’ll often be right!)
We may do at least one more small release before 10.30, either for DSF recuts, more lego brick airports, or to fix any crash bugs that we find. But at this point no code features are going into the sim before another “big” patch – small patches will be for art.
In terms of timing: I can say with some certainty that 10.30 will be next year, not this year – there are less than six weeks left in 2013, and we won’t be ready to public beta a big patch that soon. I think it is possible that we’ll have one more small patch in 2013, but we don’t have a firm plan yet.
The DSF recut work is mostly done, but I am going to wait for AlpilotX to release his latest HD scenery; his scenery is developed using the same tech as the global scenery, so it should give us a good idea as to whether the recuts are ready or need bug fixing.
Posted in News
by
Ben Supnik |
I’ve marked WorldEditor 1.2.1 as final. If you are a WorldEditor user, please upgrade from 1.2 to 1.2.1. Besides fixing a bunch of bugs (including annoying usability bugs), it changes the format of the zip files for submissions to the global airports to a format that is a lot easier for us to handle on our end.
The next version of WED will be 1.3 and will include more usability features; we’re not sure what else will go in that bucket or when we’ll cut it off and kick it out the door.
In a previous post I discussed a new facility in X-Plane 10.25r1 (coming real soon) to disable aircraft-attached objects for performance optimization. There is a second use for this feature: performance analysis.
This post is targeted at aircraft authors, particularly authors who create complex aircraft with Lua scripts or plugins.
The basic idea here is to remove work from X-Plane and measure the improvement in performance. I strongly recommend you look at X-Plane’s framerate in milliseconds. An improvement of 1 frame per second is a big improvement at 15 fps and almost nothing at 60 fps. But saving 5 ms is always a good thing, no matter what framerate.
The GPU Is Confusing
Your graphics card is basically a second computer, with its own memory, its own bus, and its own CPU (in this case called a GPU). Ideally the CPU on your computer and GPU on your graphics card are both working at the same time, in parallel, to present each frame of X-Plane.
OpenGL drivers accomplish this by building a “todo” list for the GPU, which the GPU then does on its own time. Ideally the CPU is off computing flight models and running plugins while the GPU is still filling in mountain pixel shaders on screen.
For any given frame, however, one of the CPU or GPU will have more work than the other, and the other will be idle some of the time.
- If you have a lot of GPU work (e.g. 4x SSAA HDR with clouds on a huge screen) and not much CPU work (e.g. no autogen) then the CPU will have to wait. It will give all of its instructions to the GPU for a frame, then the next one, then the next one and eventually the GPU will go “stop! I’m not done with the frames you gave me” and the CPU will wait.
- If you have a lot of CPU work (e.g. lots of objects and shadows and plugins) but not much GPU work (e.g. a small screen without HDR on a powerful graphics card) the GPU will wait; it will finish its entire todo list and then go “uh, now what?”. Until the CPU can get more instructions ready, the GPU is idle.
Here’s the key point: your framerate is determined by the processor that is not idle. If your GPU is idle and your CPU is not, you are “CPU bound”; if your CPU is idle and your GPU is not, you are GPU bound. Optimizing the use of the processor that is idle will not improve framerate at all.
Viewing Specific Processor Load in X-Plane
X-Plane’s “frame rate” data output gives you two useful numbers for figuring out where X-Plane is spending its time:
- “frame time” gives you the total time to draw one frame, in milliseconds.
- CPU load give you the fraction of that frame time that the CPU was busy.
For example, my copy of X-Plane is showing 9 ms frame time and .78 CPU load. That means that the GPU needs 9 ms to draw the frame, but the CPU needs 7 ms to draw the frame – I am GPU bound. (I am also in the middle of the ocean, hence the low numbers.)
Unfortunately if you are CPU bound (CPU load > 0.95 or so) there is no current display for the GPU’s utilization; we are working on that for X-Plane 10.30.
Analyze Performance By Subtraction
Putting it all together:
- You can calculate the actual CPU time spent in your add-on from the CPU load and frame time data outputs.
- You can disable your add-on to see how much CPU time is now used; the difference is the CPU time your add-on is consuming.
- You can change settings to be GPU bound (and confirm that by seeing CPU load < 0.95). Once you are GPU bound, improvements in framerate when you turn off your add-on show the amount of time you used the GPU for.
Armed with this knowledge, you can find the real cost in milliseconds of GPU and CPU time of the various parts of your add-on. You can find out what costs a lot (the panel? The 3-d? The systems modeling) and then focus your optimizations in places where they will matter.
Support Performance Analysis in Your Add-Ons
In order to tell what part of you add-on is consuming CPU and GPU time, you need to be able to separate your add-on into its sub-components and disable each one.
If your add-on makes heavy use of plugin code or scripts, I recommend putting in datarefs that turn off entire sub-sections of processing. Good choices might be:
- Bypassing custom drawing of glass displays in an aircraft.
- Bypassing per-frame systems calculations.
- Bypassing per-frame audio code, or shutting off the entire audio subsystem.
- Turning off any 2-d overlay UI.
You use DataRefEditor to turn off each part of your system and see how much framerate comes back. If you get 5 ms of CPU time back from turning off your 2-d UI you can then say “wow, the UI should not be so CPU expensive” and you can then investigate.
X-Plane Supports Performance Analysis
The technic of shutting a system off and measuring the increase in performance is exactly how we perform performance analysis on X-Plane itself, and as a result X-Plane ships with a pile of “art controls” that disable various pieces of the simulator.
This article provides a big list of art controls you can use to disable parts of your aircraft and measure the performance improvement.
Here’s where object-kill datarefs come in: the “cost” of drawing an object is mostly inside X-Plane (in driver and OpenGL code) but also in the time spent calling dataref handlers that drive animation. With an object-kill dataref, you can turn off your objects one at at time and see which ones consume a lot of CPU and GPU time. If an object is found to be expensive, it might be worth an optimization pass in a 3-d editor, or some time spent to improve the scripts that control its datarefs.
I have a lot to cover here – a little for everyone I think.
10.25 Release Candidate 1 Is Up
If you are a third party developer using 10.22, and you haven’t participated in 10.25 betas, please go do so now. You can get the beta by running the installer and clicking “get betas”. (If you run the beta, you are auto-notified to update.)
This build sneaks in object-killing in Plane-Maker; thanks to the aircraft developers who took time to privately test this feature last week!
A Fix to the Plugin SDK
This section is just for the programmers. I investigated a three-way conflict between X-Plane 10.25, Gizmo 64-bit and the new 64-bit XSquawkBox, and what I found was a bug in the C++ wrappers that ship with the X-Plane SDK headers. XSquawkBox was using them, but they were not correctly updated for 64-bit.
They are now. So if you use the “XPC” C++ wrappers in your plugin, please go get the new headers!
I’ve written about this before on the X-Plane dev email list, but the short of it is that ‘long’ as a datatype is not safe for plugin use. A long is 64-bits on Mac/Linux but 32-bits on Windows. If you use long, your data structures change size, which is never what you want.
The SDK widget APIs sometimes store widget IDs (which are pointers) in integer slots. in order for this to work, the slots must be 64 bits wide. The old SDK (and XPC wrappers) use ‘long’ for this, but the correct type is intptr_t. The SDK made this change a while ago, the XPC wrappers made this change now, and you should be sure that your plugin isn’t using “long” with the SDK.
The failure mode of mixing ‘long’ and ptrs on Windows is exciting: the upper 32 bits of the address of the widget get cut off; as long as you allocate your widgets early, your widget address is probably less than 2^32, and you are okay. But if your plugin loads later, your widget IDs (which are secretly ptrs to structs) will be > 2^32 and converting them to long changes the address, crashing the sim.
This is exactly why Gizmo appeared to be “crashing” XSquawkBox: XSquawkBox was using ‘long’; if Gizmo ran first and allocated memory (which Gizmo is well within its rights to do!) then XSquawkBox’s widget IDs would be greater than 2^32 and the ‘long’ bug would kick in.
(I don’t know when Wade will release an updated XSquawkBox, and I do not plan to discuss XSquawkBox any more on this blog. You can follow XSB here.)
Whose Bug Is It Anyway?
The XSquawkBox + Gizmo crash illustrates an important point: if two add-ons work by themselves but crash when used together, we can’t know which add-on is at fault without more investigation.
In this case, the bug was in XSquawkBox. But before I investigated on my computer, Ben Russell reported to me that removing some initialization code from Gizmo also “fixed” the problem (in that it made the symptoms disappear). Yet we know from investigation in the code that XSquawkBox had the bug (using long for pointers on Windows).
The moral of the story is: if two add-ons crash together, we can’t know which add-on is fault by which add-on changes to “fix” the problem. It is very common in the OpenGL world for the driver team to change the driver to work around buggy apps, and for apps to work around problems in buggy drivers. A change to code might be a fix for a bug, but it might be a work-around, avoiding a bug that still exists.
Here’s my take-away point: identifying a conflict between two programs is a way to narrow down a bug, but it is not a fix. We (Laminar Research) almost always ask you to remove add-ons when you see a crash. This is not a fix! We want you to remove add-ons to identify the conflict between X-Plane and a particular add-on (or between two add-ons). The next step is for us to figure out why the add-on might crash X-Plane or vice versa. Typically we prefer to contact the add-on maker directly to get technical information. What we are looking for is an identified conflict with the minimum number of variables.
