Comments on: Why Is Installer Scanning Slow? https:/2008/08/why-is-installer-scanning-slow/ Developer resources for the X-Plane flight simulator Tue, 01 Feb 2011 18:58:41 +0000 hourly 1 https://wordpress.org/?v=6.6.1 By: Matt Schifter https:/2008/08/why-is-installer-scanning-slow/#comment-967 Sun, 07 Feb 2010 16:35:47 +0000 http://www.x-plane.com/dev_blog/?p=319#comment-967 Hi,

Actually, since .zip files provide internal integrity checks, you could create a very large zip file, copy it over, verify it using standard zip file integrity testing, use an MD5 on that, and then unzip all the files on the destination at once. Using the largest file size for the copy from the CD will multiply your installation performance by orders of magnitude. I'm at about 170 K per second installation rate. Part of the performance lag must be that I'm having to use an external drive for storage, however, it is not a slow drive.

"Real-time" Regards,

Matt

]]>
By: Anonymous https:/2008/08/why-is-installer-scanning-slow/#comment-968 Wed, 06 Aug 2008 11:23:00 +0000 http://www.x-plane.com/dev_blog/?p=319#comment-968 Well, you’re certainly right about the support nightmare in the case
of locally cached hash values. (I suppose one could provide an option in the installer that says
‘yes, I know what I’m doing, just use the local cache instead of recomputing everything’, but I guess this would bring support problems of its own….)

]]>
By: Benjamin Supnik https:/2008/08/why-is-installer-scanning-slow/#comment-969 Wed, 06 Aug 2008 10:51:00 +0000 http://www.x-plane.com/dev_blog/?p=319#comment-969 A hash is EXACTLY how it works now! The manifest list for an install is a set of MD5 sums of each file.

The speed question is: when do we compute the LOCAL hash for the LOCAL install?

Right now, we scan that local file and hash it when you update, e.g. we say “scanning.” That’s the slow part…it requires us to scan every byte of the install.

So the optimization would be to cache our local hashes…but this is the step I do not want to do – because if a file was changed, the local cache would be out of date and we could miss a file that needs updating.

Basically what I’m saying is: I don’t want to get clever and cache the hashing work on an update because the scan isn’t THAT slow and the consequences of getting a file wrong are unbounded tech support nightmares.

By comparison, I’m willing to play fast and loose with the hash on global scenery because file mismatch is a lot more rare in this case, and the hash is horribly slow for that much data.

]]>
By: Anonymous https:/2008/08/why-is-installer-scanning-slow/#comment-970 Wed, 06 Aug 2008 10:45:00 +0000 http://www.x-plane.com/dev_blog/?p=319#comment-970 Couldn’t a hash be used here? Something like the following scheme perhaps…

The current X-Plane install would have a manifest file containing a tuple
(path, hash) for each known file in the install. The hash would
be the output of some hash function
(MD5, SHA-1,…) run on the file contents. The new installer would then just download the new manifest file and compare it with the one in the current installation. If the hash value differs for the same path, then that file needs to be downloaded.
This way you never need to read individual files or even traverse directory hierarchy. You only need to keep the manifest file up to date.

]]>