EXPERIMENTAL USE ONLY!!! -- [What?] -- Login to add public annotations [How? >>YouTube].

I received an email yesterday that I found a little bit concerning; an author wrote this:

I hope 10.30 is not going to change any of the datarefs we as a community have identified and are using to make the sim better and if it does I will work to modify the scripts to make them work again.

Here’s the problem: his add-on is one of the new Lua scripts that changes the internal art controls for atmospheric scattering – it is almost certainly not built on public x-Plane SDK datarefs, it is built upon finding, locating, and hacking the internal art controls in the sim.

So I hope the author isn’t going to mind working to modify his scripts again, because those art controls will change.

Simply put, building an add-on based on art controls is like building a house next to an active volcano – it might be a nice place to live, but you’re going to have to rebuild your house from time to time.  Be prepared to do your work over, because art controls will change – it’s a matter of when, not if.  The rest of this post is some warnings, scare tactics, and an explanation of why the art controls change.

Scary Warnings, Etc.

In my past comments, both public and private, I have tried to be clear that the art controls are not a stable interface, they are subject to change, and that you can’t build an add-on that will keep working while we patch if you use them.  (I’d also hope that the fact that every art control starts with sim/private would get people’s attention.)  But I apparently never wrote one big public global “no warranty” message*, so…um…here we go.

The art controls are not a public interface to make X-Plane add-ons.  They are an internal development tool.  They are unsupported, undocumented, unsafe, and most importantly subject to change with every patch of X-Plane.  If you create an add-on that requires reading or writing the art controls, you can expect that your add-on will stop working when X-Plane is updated.  When your add-on breaks, please do not complain or file a bug.

So hopefully that makes things clear.  Art controls will change,you’ll have to do your work over, and we’re telling (and have been telling) you that since people first figured out how to access them.  If you learned about art controls from someone other than me and weren’t aware of this, the person who taught you forgot to tell you the most important thing.

What Is An Art Control

An art control is an internal hook into our engine that we use to tune and calibrate algorithms, measure performance, and debug problems.  I started adding art controls so that:

  • Our art team could “tune” X-Plane without having to have the source code or recompile the sim and
  • So I could see the results of changes to the engine without having to quit and restart X-Plane.

They are a critical part of X-Plane development.

They are not, however, some kind of public interface for third party development.  We have public datarefs, listed on the X-Plane SDK website, that let you replace the flight model, customize the control surfaces, etc.  Those datarefs are a real “interface” to the sim – even when we modify X-Plane, we try to keep that interface working.  Many of the public datarefs are over ten years old now.

By comparison, the art controls aren’t an interface – they are deeply tied to the how of the engine, e.g. the specific algorithm we are using today.  If we come up with a better algorithm, those tuning controls lose relevance and the art control goes away.

Why Don’t You Make the Art Controls Public

The immediate question that most developers have when they hear my rantexplanation of art controls is: why don’t you make the art controls public so we can use them?

It’s a good question, particularly when the art controls are useful.  The answer is that the art controls don’t represent a specification for what the sim does, they represent side-effects of the algorithm we are currently using.  Let me use atmospheric scattering as an example.

Atmospheric scattering is essentially a mathematical formula (run on the GPU) that makes the color of pixels “more blue” when they are farther away.  The art controls for scattering become constants in that formula – thus you can change the constants (but not the overall shape) of the formula via art controls.

But here’s the problem: the formula I use isn’t very good.**  The result is that when the constants are high (for “heavy” scattering) the far view turns a really rich, electric blue that just doesn’t look anything like the planet we actually live on.

As long as we keep this formula, we’re stuck with two options, neither of which are much fun:

  1. Keep scattering turned way down.  This is what we ship with; clearly users want more scattering – you can see that by all of the positive reactions to add-ons that crank the art controls up.
  2. Turn up the scattering and try to ignore the blue.  This is what most of the scripts that hack the scattering art controls do, and I would say it’s clearly a trade-off.

So I am working now on a better formula, one that won’t produce incorrect colors at high scattering levels.  But when that formula is done, it will be a different formula, with different constants in different places in the equations.

And that is why art controls change. The very code that gave birth to them changes, and they get deleted and replaced with new art controls.  The only way to keep the scattering controls forever is to never fix the flaws in the current formulas.

What Art Controls Will Change, and When

First, I hope that if I’ve made one thing clear, it’s this: if you want your add-on to keep working with future X-Plane patches, you need to avoid the art controls.

Realistically, the art controls tend to change when there is major development in the sub-system that they are part of.  So the shadow art controls will change any time we try to make the shadows better; the scattering art controls will change any time we try to make scattering better.

So if you are just tinkering with X-Plane, the good news is: if your art controls have changed, there’s a chance that you may be able to achieve a better effect using the new code.

One final note: if you want to build an add-on that needs to work past version updates (e.g. a payware add-on where having to follow our patch schedule is not a good idea) and the only way you can accomplish your goals is with art controls, send me an email.  We may be able to add a real interface to replace art control hacking.  This won’t always be possible, but sometimes it is, and the results will be a lot less work for your add-on in the long term and happier customers.

* See Resources/settings.txt for the warranty on hacking settings.txt.

** In my defense, the real look of the sky comes from a huge number of photons crashing into a huge number of molecules a huge number of times – we’re not even remotely close to having enough computing power to solve that honestly, so all scattering code is based on gross simplifications of gross simplifications.

About Ben Supnik

Ben is a software engineer who works on X-Plane; he spends most of his days drinking coffee and swearing at the computer -- sometimes at the same time.

29 comments on “Art Controls Are An Active Volcano”

  1. Ben, apologies if this is off topic – in reference to the super cyan effect with scattering, is this due to the scattering being applied on top of the sky colors?

    Any thoughts about having scattering itself render the sky like the nvidia demo? (Not attached to any version, won’t interpret thoughts as a promise)

    I am not asking to annoy you, I am asking because atmospheric rendering is interesting. Thanks!

    1. No – it is not due to additive lighting between the sky and fog. It’s because the mathematical function that generates the in-scatter colors (the light added due to fog) looks silly at very high values. It’s simply the wrong math.

      I’m not sure what NVidia demo you’re referring to – they’ve produced a ton of demos. Do you mean the O’Neil scattering from GPU gems 2 or something more recent?

  2. Ben, GPU Gems 2

    Please correct me if I am wrong, but it sounds that what you are describing is that the atmospheric scattering effect is applied to the terrain, rather than to the air. (so pixels above terrain use the skycolor algorithm, rather than scattering)

    I think what I was getting at is that I would imagine, given very little knowledge and no expectation that this is feasible within the GPU budget you guys are targeting, that the atmospheric scattering can just be applied to the atmosphere (basically a volumetric sphere around the earth) and based on your current altitude, angle of the sun and temp/humidity (for the mie scattering), you could basically get the skycolors/scattering effect on the terrain all from the same process.

    It might also simplify HDR (by not requiring tone mapping different layers of a render), and fog could be a special case of this effect (really, inside fog/cloud is just mie scattering like crazy)

    And again, you guys are awesome. I know this is a flight sim, not a atmospheric sim. I just find when I use some of those scattering scripts (knowing the rug can be pulled out at any moment) i find i get a way better sense of scale and can turn down the autogen without losing the “realness” feeling 🙂 – with good terrain and good atmosphere effects, i’d almost be fine with ortho scenery 🙂

    1. Hi John,

      Ah – okay … GPU Gems 2 – also sometimes referred to as the O’Neil paper…I actually contacted the author way back when and he was very helpful – a great guy!

      We started with O’Neil’s work – what we have now is sort of plan B code. The problem with O’Neil’s design is that, by simulating only one scattering interaction, it creates sunsets that are not even remotely correct color-wise. His algorithm is also strictly procedural, so if the colors produced by the equations are ‘off’, an artist can’t just go in and color correct them – there are a few starting values, each of which has a wide range of effects.

      (In O’Neil’s defense, he was dealing with the GeForce 6800 – it’s amazing that he was able to create the effects he wanted at all.)

      To answer your question about in-scatter and out-scatter, basically the terrain must have both in-scatter and out-scatter to look right – in real life the mountains don’t look blue because they are blue – they look blue because blue light has been scattered INTO the line between you and the mountains. This is why things look ‘so’ blue – even an object with no blue albedo (e.g. its blue is 0% up-close) gets tint via in-scatter. The out-scatter removes light that is lost via the scattering process.

      The sky is really a special case – maximum in-scatter, nothing to out scatter (since space is already black). This is why when you see the moon during the day, the dark side of the moon is pure sky-blue – behind the in-scatter from the sky, it’s a totally unlit (and thus black) dark side of the moon and totally black space. The sky blue is added in.

      Re: your comments on parameterization, there was a French paper I once read that derived from O’Neil and then did more or less what you describe: they attempted to parameterize on more parameters to get a more accurate effect. They ended up with much better results, but with four input parameters (sorry, I don’t remember what they were) which makes lookup real hard.

      One last comment on this: the thing that makes it hard to adapt O’Neil’s idea to a real game is the line integral. One of the really clever things O’Neil’s paper gives you is the notion that the infinite line integral of atmospheric transference to space can be subtracted from another infinite integral (they’re both rays from a start point to infinity) to get a definite line integral. This is what makes it possible to get correct scattering everywhere in the scene at once. But it only works if you have a ‘quantity’ like transference – once you map to color space, you can’t just go around subtracting one thing from another.

      So it’s hard to put lookup tables in place for the artists to tune things…they can only get in and tune things late in the game, and by that time too much information might be lost.

      cheers
      Ben

      1. Ben,

        Awesome reply. I wasn’t so much looking for the ability to tweak the scattering, so much as it’d be cool if x-plane did the inscatter/outscatter automatically, and the sky and the scatter effect just emerged from that. (and omg, it’d scatter the streetlights too…)

        I get that this is complex though, because you are basically faking bidirectional path tracing with billions of scatters with a good-enough shader.

        1. Exactly – if we could actually run a much larger number of calculations, things would start to be emergent. But we’re still way in the domain of ‘clever trickery’ because the amount of scattering interactions is so much larger than today’s hardware can handle.

  4. I really am not a fan of Real Terra Haze and much prefer the standard “shipped” look. Many users seem to like the look as did I for a short while but the novelty soon wears off.

    Looking forward to 10.30 so much. As someone who has a stable system and fly’s heavies regularly the only think I need is the extended DSF coverage. This will be a huge thing for me 🙂

  5. SO SO SO looking forward to the “view in the distance” being improved.

    Will this help the Takeoff view at KBFI (Rwy 13) where Mt Rainier, on a sunny day, should basically devastate you with beauty in the distance directly ahead?

    1. I think you’ll get a better view of Rainier.

      Please note that at this time our plan is to go public beta WITHOUT the view distance increase – the view distance increase will be added to a later public beta.

        1. Yes. I have a second fog feature that doesn’t play nicely with the first one that I need to sort out. We don’t need to make people wait for the G430 because I’m still trying to get fog right. Plus aircraft developers may want to try the G430 with their plane and see what happens.

      3. Yes, Mt Rainier can be seen now (just quickly tested it in the internal beta) from KBFI if one dials up the visibility to around 90nm …

        1. AWESOME!

          Whenever that feature comes along, I’ll be excited!

          Also very excited about the G430 though as well.

          Great to get some updates!

  6. Pingback: Art Controls Are An Active Volcano | Aerosoft Sim News XP

  7. Re: 430

    Is this going to upgrade the texture quality on the screen of the 430 itself?

    It seems that on even new Carenado aircraft, with settings dialed very high, the screen of the GPS is still blurry and blocky and just “gross” looking.

    It looked like those new 430 screenshots showed a beautiful sharp display, correct?
    If so, that is going to be really really nice.

    1. Here’s the short of it:
      – The existing 10.25 old GPS was very small – 62 pixels tall. Therefore it tends to look blurry in a 3-d pit, because there just isn’t that much detail in the base instrument. Due to weird code, the base instrument cannot be up-sized and retextured – a known limitation of 10.25.

      – In 10.30 there is the option of the new GPS – it really wants 240 pixels tall – 4x the res. So here’s what the NEW instrument does, more or less, last I checked, Philipp may have to set me straight here:

      1. If the new instrument is given a REALLY SMALL spot (e.g. 62 pixels tall) it draws a compact display with just frequencies – not realistic, but at least readable.

      2. If the new instrument is given a BIGGER spot (I think the cutover is around 100 pixels, but authors SHOULD give it the full 240) you get the full instrument layout, and how sharp it is will be a function of the spot size – at 240 pixels tall you get the ideal look.

      3. At all times, at any size, you can click on the NEW instrument’s screen in 3-d or 2-d and get a POPUP that runs at full size and full sharpness.

      So…until people remap their panel texture UV maps to give the instrument more size, it will look blurry or even reduce complexity. But the popup will always be available.

      We can’t really do better than this – we (X-PLane) can’t change the amount of UV space for just one instrument on the panel.

      1. Sounds fantastic.
        In something like the Carenado A36, I’m looking forward (once they update to the new GPS) to popping out the Aspen PFD and the G430/530 right next to it and then just flying with the forward view and no 3d Cockpit.

        Max performance and hugely readable gauges that are interactive and all you need to fly around and do approaches. 🙂

        Sounds like a dream come true!

  8. I for one welcome the new art controls and the new algorythems to change the athmospheric scattering so they get more blue and have even greater view (as with the indicated 10.30 fix).

    But instead of community hacking the art controls every time in order to create fantastisc looking enhancements of this great platform, why not follow Lockhead Martin community engagement with Prepar3D. Create a dedicated forum and invite the community to give feedbacks, suggestions, creative tips and solutions Laminar can implement if so decided?

    I would be happy to accomodate such a collobration and re-build the forum to fit any suggestion by LM dev team.

    1. I support the idea!
      I suspect that you guys at LR are tired of answering the same questions/complains/feature request even though you are trying your best to answer them and keep us updated. I think it could be worth to establish more structured/centralized way of communication. Maybe forums, each one per feature request/issue, where X-Plane users could follow the current status of development and exchange the information between themselves and LR? As of today the information IS there but it is scattered around many places (this blog, x-plane.org forums and so on). E.g. questions & replies regarding visibility range have been posted numerous times on this blog but only fraction of X-Plane users follows it thoroughly and reads all entries/comments.
      Also how about setting up some kind of voting system where X-Plane users could cast votes for their favorite new feature? Just to let you guys know what is in demand and what features are not so important to us.
      just my 2 cents..

      1. My view is that “more structured” wouldn’t be a win. The reason is that what we (the small number of LR employees trying to take in a lot of information) need most is information _quality_, but I’ve yet to see a formal system that produces consistently high quality information without also filtering out some important stuff. The result is that, instead, we all have our ears in different places, trying to keep an eye out for “hrm – that post looks pretty sane.” A few random notes:

        – We _do_ have a formal system for bug reporting (the bug reporter) and every bug report gets read by someone. The system suffers from a few problems though:

        1. It is overrun with user tech support complaints – that disclaimer to NOT post tech support q’s is universally ignored. We forward those requests to tech support. But if you’re wondering why you can’t file an INTERNAL bug in our bug database, with a real bug #, that’s why. A very very small number of users who have consistently reported solid bugs get a direct link to the bug report. But truth is, if your bug reports are good and you file with the bug report forum, we’ll find them. The easiest way to get your bugs heard is to file better reports.

        2. The most important bugs we get are regression bugs from third party authors – and unfortunately authors tend to think that other authors will file for them – everyone has incentive to let someone else file the bug. If I had an nickel for every time an author reports a bug that’s been in the sim for a YEAR and (when I say “You are the first to report this”) says “oh, we’ve been discussing this on xyz forums since 1979” I’d be able to buy an island.

        – I consider forums to be a PITA…I don’t have time to go reading forums to search out information, and they tend to be badly curated – they display information chronologically, and (this is just me being grumpy) it’s a lot of signatures and avatars relative to information content. So when I get a bug report that links to a forum, that’s (to me) a total fail, because the forum post is pretty much never a clean, well written bug report.

        With that in mind, if you write up your bug report cleanly somewhere, I’m happy to click the link, if the info is all there on the other side, and it’s correct and has the info I need without stuff I don’t need.

        – We don’t take public polls re features too seriously because of selection bias: we hear directly from categories of users who buy X-Plane whose needs are very different than the sub-section of the community that is _most_ active on the forums. (You only have to look at the size of VATSIM’s population that flies on a regular basis with XSquawkBox vs. the number of X-Plane units sold to understand that small groups of users can be vocal.) It is interesting to see what people want, but polls tend to go toward the “neat toys” category of things (e.g. just about any special FX could get some poll votes) while totally ignoring things that actually keep the lights on.

        (For example: installation really, really matters — if getting the demo works badly, ours odds of selling you the sim are slim. If you buy the sim and your installation process sucks, you’re in a bad mood about x-plane from day 1. The installer is not why I got into writing flight simulator code, but it is the “first date” of X-Plane. And yet…you’re not going to find any poll that puts installation in the top todo items. And if I post about improvements in the installer to this blog, the comments will be totally off-topic – even more so than normally.)

        – There’s mismatch between what users are interested in telling us and what we have time to investigate deeply. The sim is heavily road-mapped and a lot of the next year of development is already planned. We may change the plan (a lot) but in the end if you come to me and say “I really want to talk to you about snow shaders” – well, this is hard. We do have snow shaders on our road map, but we’re not there yet. I sometimes ask users to write up a ‘1 page’ – that is, a short but detailed summary of their point that I can then file in my notes by topic, so that when I do come back to the subject, I have the info archived.

        You can see this mismatch in this blog when the topic post is completely unrelated to the comments, which go off in another direction.

        Finally, I think that any user serious enough about X-Plane to be reading, well, down here, can help disseminate information. All the way back in X-Plane 6, Andy Goldstein had the “Unofficial X-Plane FAQ”. Andy’s a smart guy who has been using X-Plane since 1637, and he put down answers to the most common questions in a clean, well-curated format. I think this did contribute positively to the amount of info about X-Plane that was out there.

    2. I think you mean suggestion by the “Laminar” team, Tom.

      A key difference to remember is that LM has the manpower resources to interact with a community of suggestions. Laminar is orders of magnitude smaller, and we already bug Ben too much. 🙂

      What we really need to do is to start a coffee fund for the guy so that he can work 24×7 on X-Plane.

      1. I think Ben and company have a pretty good idea of where they what to take X-Plane, and I think they’ll get there alot faster without “we the community” bugging them with what we think is reality….kinda like all the first graders picking what the teacher,, teaches….cheers

        1. Completely agree, Thomas. And Ben just took a bunch of space to give all the background info anyone with a crowd sourced input stream idea might want. Everyone has a voice and many want to be heard, but flight sim development is not a democracy. 🙂

          Ben, many thanks for a good deal of thoughtful and gentle explanation. Really looking forward to seeing in the distance….I want to take a U-2 up and see for miles and miles and miles…… 😀

          1. It should be noted that, while flight sim development is not a democracy, sometimes we (LR) do not know best…it’s tricky getting user feedback when internal business concerns aren’t public, etc.

Comments are closed.