DSF Art Asset Storage RFC

This RFC proposes the ability to embed small directly accessed art assets inside DSFs.

A DSF_File_Specification file contains geographic/graphic data, but contains file-based references to the external art assets used to visualize the data. Every art asset in X-Plane is an external file, and those art assets are never image files. Image files are always further linked from a primary art asset. (Thus for a 2500 orthophoto scenery pack, there must be 2500 .ter “art asset” files that describe the meta data for the orthophotos.)

X-Plane can typically defer the loading of the image files associated with a DSF, but it must load all of the art asset meta data when a scenery pack is loaded, in order to configure the rendering engine properly. (Indeed one motivation for putting data in external meta-data files was to allow X-Plane to fully set up the rendering engine without having to touch image files, which might be coming in on another thread at a much slower rate, while the user flies.)

The problem is that as scenery packs have started to use large numbers of individual art asset files, disk seek time has become a bottleneck during initial scenery load. Multiple-minute load times are not uncommon.

This RFC aims to streamline this process by allowing a DSF to directly contain the contents of some primary art asset files. This is particularly a win for very small art asset files (e.g. .ter and .pol files which are often much smaller than a single disk block).

Conceptual Setup

Conceptually, a DSF would become both a DSF file and a container file system for some number of primary art asset files. Art assets would simply be referenced by ‘file name’ inside the DSF.

X-Plane would then attempt a direct match between referenced art asset name and a file ‘contained’ in the DSF. For example, if a polygon definition in the DSF polygon definition header atom contained the string ‘lakes/big_lake.pol’ and there was an entry in the DSF file table labeled ‘lakes/big_lake.pol’ then the .pol file would be loaded from the corresponding DSF data, without ever looking for a file called big_lake.pol in the lakes directory of the scenery pack.

Note that the ‘directory’ character in the DSF has no meaning – it is simply a convenience to be able to create ‘directory-like’ hierarchies; this is provided to make it simple to convert an existing scenery pack to use the DSF as ca container – see “Authoring Use” below.

Also note that this system cannot be combined with the library; library items must come from external disk storage.

DSF Syntax

A new files atom would contain three sub-atom:

• A string table atom would contain the names of all contained ‘files’.
• An offsets atom would contain a beginning and ending offset length pair (8 bytes per file) into the data atom. The order matches the files atom.
• The data atom contains the back-to-back file data.

The string table must be sorted by file name (using a simple byte-sequence comparison like memcmp); the offsets atom follows this order. The data atom does not have to follow the same order as the offsets/names; the offsets allow files to be reordered from the sorted order.

Authoring Use

The most likely scenario for authoring use would be a pair of authoring tools to ‘pack’ and ‘unpack’ a DSF. Since the DSF file data is atomic, the DSF could have files added and removed ‘losslessly’ – that is, without reducing the quality of encoded mesh data. (By comparison, running a file through DSF2Text can reduce mesh precision.)

This would allow authors to work with the existing format, and then ‘pack’ the DSF at the end of development to speed up load time.