Partial Transfer Fails

[Kevin Reed]

If a project compresses files and relies on libCurl to decompress the file during transfer, then in the event that a transfer is interrupted, BOINC relies on the decompressed file size to set the Request Range header. This will be invalid and therefore fail with:

416 Requested Range Not Satisfiable

this will repeat until the timeout limit for the download.

It would be great if in the event of the failure of a partial file download, the client would then attempt a full download.

[Nicolas]

Is the server sending Transfer-Encoding: gzip, Content-Encoding: gzip, or is the file type actually a .gz, thus sending neither encoding header?

[Kevin Reed]

I am using the suffix gzb and setting the following header:

Content-Encoding: gzip

Ex: ​https://grid.worldcommunitygrid.org/boinc/download/3b6/faah7186_ZINC00054326_xmdEq_1MSN_01_ZINC00054326.pdbqt.gzb

Note that this example will be deleted eventually.

[Nicolas]

Who is supposed to decompress that file, the science app, or are you relying on the BOINC client doing it?

[Kevin Reed]

The boinc client decompresses the file.

[Kevin Reed]

We were using mod_deflate as outlined here: http://boinc.berkeley.edu/trac/wiki/FileCompression

Most of that page reflects what we had been doing.

We need to conserve storage on the server side, so we have started to 'pre-compress' the files for download and we are using the suffix .gzb to indicate this. We have added the following to our httpd.conf in order to add the header:

AddEncoding? gzip gzb

[Nicolas]

Older clients that don't support compression will fail; the server should give a compressed file only if the client says Accept-Encoding: gzip; otherwise it should give the decompressed file. I just tested your URL, and I'm sure it's sending a gzipped response always.

[Kevin Reed]

Clients older than 5.4 no longer work with World Community Grid. We specify that in config.xml using the tags:

<min_core_client_version>

504

</min_core_client_version>

The 5.4 and more recent clients work with this correctly with the exception of partial file transfers. We have been using this for about 3 months.

[Nicolas]

That's not the point. Do servers have a giant database saying what versions of what web browsers support compression?