TL;DR version: my iMac’s fusion drive “lost its marbles” right before I went on vacation. This has delayed cutting an 11.05 release candidate 2 with a few scenery fixes, but we should get to it next week. In parallel, we’re working furiously to get all of the code locked down for 11.10.
Everything else that follows is really, really, really, really boring. I’m writing it only because some of my co-workers watched this slow motion car crash and tightened up their backup game a bit. If my drive fail can shake you out of complacency, read on.
Basically: my iMac is my main development machine, and the data is backed up and/or duplicated in a bunch of different places: a USB time machine archive, a Backblaze cloud backup (both are “full machine”), DropBox for virtually all of my documents, and my work for Laminar is kept on Laminar’s source control servers. Data loss was never a huge risk here.
Time loss, however, is a real risk! My goal was to lose as little work time to fixing my machines as possible. So my plan was: restore from time machine disk backup, request a cloud backup restore via hard drive, return the hard drive. The total cost would be a few hours of disk copying and less than an hour of my time. My development machine would be usable for new work while waiting for the cloud backup to arrive.
This has not gone as well as I had hoped! You can learn from my fail here — a few notes.
Your backup might as well not be a backup if you have not checked that the backup contains the data you think it contains. It turns out that both the cloud backup and time machine backup were missing files! I’m very lucky that they weren’t missing the same files.
Time machine sometimes decides not to back stuff up. OS X has a hidden per-file/directory attribute that can exclude a file from backup without showing it in the Time Machine UI! Once you check your time machine backup and find a folder is missing, from terminal you can do tmutil isexcluded <file path> to see if the file has been explicitly excluded. If it is, tmutil removeexclusion <file path> fixes this.
Backblaze ships with a bunch of file exclusions too – mostly designed to not archive stuff that isn’t your data. But beware – stuff you care about might not be on the list. (For example, virtual disks in a virtual machine are excluded by default.) I had to add back .iso files to the backup list. Backblaze backups are also not bootable. This is something I can live with, but always read the fine print about what’s in the backup.
The Backblaze data restore has been very slow – over ten days for less than half a terabyte and it’s still “in progress”.* While they haven’t exceeded the maximum restore time they advertise, it’s slow enough that the delay matters.
One other note on Backblaze: I saw major performance problems on my iMac while Backblaze was running, even when a backup was not running (since they were scheduled for overnight). I do not think this is necessarily Backblaze’s fault – it may be a problem with CoreStorage (which “runs” the fusion drive) or even a fault with my drives. From what I can tell, cloud backup exacerbated it by putting a lot more file traffic on my system.
A possible danger if (like me) you keep documents on DropBox to have them everywhere: when I restored my iMac from Time Machine, I was exposing DropBox to my data from a week ago. I didn’t wait to see if DropBox would figure out what happened; I unlinked my iMac while it was offline after the restore, then re-established DropBox and let it download my data. Better safe than sorry.
I have been backing up to portable 2.5″ USB drives because they’re cheap and really convenient, but they have a down-side: the mechanisms can easily fail and take your whole backup down. I have five of these drives and one has failed in a three year period.
I’m really unhappy with CoreStorage, to the point where I would not recommend a fusion drive anymore. CoreStorage is an Apple virtual-volume technology (similar to soft-RAID) that makes one small SSD and one large HDD look like a single unified volume, with some of the data “cached” on the SSD for performance. CoreStorage is a lot newer than HFS, so when things go wrong, most disk utilities you would go to just don’t work.
I actually ended up in a state where (after wasting almost an entire day) I could see my data, but only in single-user mode with a read-only file system. I might have been able to directly copy the data, but I picked to format the drive and restore from the backup to save more of my time and get back to coding X-Plane. My suggestion for developers getting iMacs: get an internal SSD (whatever storage size you can afford) and supplement with a fast external hard drive over Thunderbolt.
Going forward, I am replacing the portable backup drives with a Synology NAS RAID device – this gets me high performance, high capacity backup (about 10 TB) with redundant drives. I picked HGST drives because they’ve had a good track record for reliability. With a large network attached storage server, I can have all of my machines backing up in the house all of the time, and have that be the primary way of getting my data back. I’m keeping cloud backup as a last-resort-the-house-burned-down kind of thing.
If my cloud backup hasn’t shipped Monday, I will rebuild the setup I use to cut builds by hand (it’ll take a few hours but it’s doable) and we’ll cut 11.05r2 that way. If the drive comes, I can get the last of my data back and we’ll get to 11.05r2 the easy way. Either way, we’ll get things moving again.
* I opted for a hard drive restore, which should have one day of shipping time, instead of a download; a smaller restore based on download made clear that the transfer speeds would be slower than FedEx for that quantity of data.
If you have been doing work with manipulators during beta.1, please download this new one and review the notes! Download the latest here: io_xplane2blender_3_4_0_b3.zip
Fixes
#ATTR_layer_group_draped causing KeyErrors
Reverts bad change to manipulators properties*
Adds
“.beta.3” will be printed on all OBJs. This is not permanent, expect something better in the future
What happened to the manipulators?
A poor decision to change the Manipulator Type (Drag XY, Push, Command Knob, Toggle, Delta, etc) means that if you changed the type of manipulator between beta.1 (or 7fe534ad1f906b7853827 if you constantly check out the latest cutting edge commits [which I do not recommend for this exact reason]), this beta will make those changes irrelevant.
Example, suppose on 8/1/2017 you create a switch and set the manipulator type to Toggle, then with beta.1 you change the type to Push. Beta.3 will show the type as Toggle. All you need to do is go back to the switch and change the type back to Push. The rest of the values you set will still be there. If you created manipulators during the beta, they will be set back to the default “Drag XY” type and will need to be fixed.
This is an unfortunate lesson to be learned, and if you are facing a lot of tediousness (30 manipulators to hunt down and change) or potential errors (you can’t remember how many or which ones you changed) please contact me!
I will personally help you recover the changes you made during beta.1 – present.
The next version of XPlane2Blender is right around the corner, come test it! Highlights of this release are Optimized Animations, Increased Usability, and X-Plane 11 OBJ8 features (mainly Blend Glass mode and Normal Metalness). Read more about what has been changed on the release page and download it!
(Link to beta removed until major breaking bug has been fixed. Make backups of files before using any beta product.)
As with any beta, make backups before using a partially tested product. We don’t predict there should be anything breaking in it, but its never a bad idea to be safe.
Bugs and feed back on Github is preferred over this comment section, but most of all I want to hear your feedback. To stay focused, only comments related to XPlane2Blender, the beta, and Blender will be responded to for this section. The status of other 3D Modeling Plugins/VR/Weather Systems/etc is off topic.
Please tell me what confuses you about XPlane2Blender on this bug, or here!
We are going to be releasing the XPlane2Blender 3.4 beta soon, and with it, a refresh of the UI and documentation. Thanks to a great e-mail about a lack of documentation, it was put as an important part of 3.4 release roadmap. It goes to show… we can’t fix it if we don’t know what’s wrong, even if its not a code problem. And we do want to fix it, I swear!
In addition, I want to remind everyone a core part of the Laminar Research philosophy, identity, and business plan is a thriving modding and third-party plugin ecosystem. Aside from build scripts and the like, Laminar Research employees use the same scenery development tools that are available to all. This is was a deliberate choice that elevates everyone to the same level – except when there is a gap of knowledge. This is never intentional, and never benefits anyone in the long run, especially third-party-devs. If your work is suffering because we forgot that not everyone knows what every little checkbox means, tell us! We’ll put it in the bug queue like everything else, and try to get back to you, personally, quickly.
We just posted X-Plane 11.05 today – to try it, click “get betas” and run the updater. Steam users: we’ll try to get a Steam version up in a few days if no massive bugs are found.
This update is really a release of airports from the X-Plane Scenery Gateway – see the release notes here for a complete list of airports included.
This release also includes our first land-marks: custom buildings from the strip near Julian’s KLAS, which is also included.
Ted and I have been working on a new version of XPlane2Blender for Blender 2.5. The new version mostly focuses on bug fixes and optimizations to get perfect WYSIWYG output from Blender to X-Plane with optimal OBJ code.
If you use the version 3.3.x of this exporter and would like to try test builds, please email me. We have a suite of test cases that we run the exporter through to confirm that it is operating properly, but it’s also useful to run it on real-world examples to see if there are cases we missed.
That’s a new dataref that (finally) lets plugin authors tilt the pilot’s head.
Previously this had to be done by manipulating the multi-monitor view offsets. That method worked (barely) but doesn’t work with multi-monitor in v11. The new dataref is multi-monitor friendly.
(The new dataref is also FMOD-sound-friendly — that is, when you roll your head over you’ll hear sounds above you out your left ear, as expected. Monitor offsets do not do that – only the pilot’s true virtual head position affects sound.)
To get this beta you’ll need to run the updater and click “Check for New Betas” – we won’t ever prompt you to install a beta when you have a final release of X-Plane. Here’s the Release Notes. Please file bugs using the Bug Reporter!
11.02 is a small maintenance patch. Our main focuses were:
Performance tuning we could do without too much technical risk. (We’ll do the more adventurous stuff in 11.10.)
Fine tuning the various SDKs.
Bug Fixes.
As a small beta, I’m hoping the entire beta period will be less than two weeks.
X-Plane 11.02 should be out this week; we’re down to one bug, whose fix I am verifying now. There have been a number of questions about performance, so to start, here is some info on three things we’ve done to make 11.02 faster than 11.01.
8-bit Water. The dynamic FFT-based ocean wave textures we stream in X-Plane 11 are floating point textures in 11.01 (F32 on the CPU, F16 on the GPU). This was an easy decision for getting the tech working, but as it turns out, transferring the textures to the GPU is slow, particularly on the NVidia drivers.*
For 11.02, Sidney has rewritten the shaders to cope with 8-bit waves. The results look almost the same, but the amount of data transferred is 4x smaller, and more importantly, 8-bit RGBA is the path most likely to be handled well by the driver, so this should be a win.
Sidney also wrote some code to transfer the textures asynchronously, but we’re holding off until 11.10 for that, as it may require debugging or behave weirdly on some drivers.
Faster Car Bucketing. The cars have always cost more CPU than they should, and profiling indicated that 90% of the work was in moving the cars around in our scenery system as they drove. The code to “rebucket” them has been modified and is now significantly cheaper. We are not turning the car density up yet (it’s not that fast), but at this point with the cars at the highest setting we ship, they now take 2-3 ms total to compute, which means they have no frame rate impact. I’d like to bring the density up in the future if we can get further performance wins, which I think we can.
Better Core Scheduling. If you’ve been reading carefully, you should be shouting at the screen about now about how the hell something that takes 2-3 ms total is “free” – 3 ms means that if you were running at 60 fps you’re down to 50. That’s not free?
I am declaring the cars free because they now run in parallel to the flight model, and it’s very likely that the total flight model work takes at least 2-3 ms, even without AI planes. The third optimization is a big cleanup of the multi-core scheduling that we do within a frame.
X-Plane uses multiple cores both to load background scenery as you fly and to speed up some calculations within a frame. As of now, the major “per frame” multi-core computations are:
The flight model (if you have more than one aircraft – we can’t multi-core a single plane).
DSF scenery maintenance (not super expensive, but does get multi-core acceleration).
Car computation (typically uses 1-2 cores at most).
FFT water calculations for the next frame (uses up to four cores).
X-Plane 11.01 was not scheduling these particularly well – here’s a picture.
What you’re seeing is X-Plane kicking off the FFT water too early, and that work blocks X-plane from completing AI aircraft calculations. The big red bar up top is the sim waiting (and FPS dying) because the AI planes weren’t done in time.
(The bottom ‘track’ with nothing on it is an IO thread that’s waiting in case we need to do UDP I/O. Since I had IO off, it is efficiently sleeping. This profile is on a 4-core machine so we couldn’t have stuck work down there.)
Here’s 11.02:
We start the (newly optimized) cars as early as possible so they complete at about the same time as the flight model; we get all DSF work done immediately, and we don’t start water until the very end. In the meantime, the main thread is free to go do the actual frame rendering.
This is just an incremental step for multi-core use; we have been steadily adding more multi-core work for the last few years, and we’ll be adding more in future X-Plane 11 updates. For example, X-Plane 10.50 re-structured the renderer, separating the work of discovering what to draw (“culling”) from the work of actually drawing. In X-Plane 11, we can do that culling on multiple cores, improving total framerate.
I don’t have great numbers on what kind of performance change you’ll see in 11.02; it’s actually hard to measure the improvements here with the FPS test because the FPS test runs a replay (and not the actual flight model) and doesn’t run long enough to generate car traffic. But we think it should be a good incremental improvement.
* It is not a bug that this case was slow for the NVidia driver; no OpenGL driver is contractually obligated to do anything in a particular time frame. It was slightly surprising in that NVidia seems to go farther than other GL vendors to optimize less common and less efficient code paths.
NVidia does normally allow for complete threading of CPU-side driver work, so it’s possible they thought there was no need to optimize this case directly since it would be on a worker thread; by comparison, Apple does not use a general worker thread for their driver but does use a worker thread for all CPU-based texture transfers, at least as far as we can tell by profiling X-Plane.
