Data server protocol

Core client communicate with data servers using HTTP. A data server is typically implemented using a web server together with a BOINC-supplied CGI program, file_upload_handler. This protocol supports "partial transfers". In particular, if a transfer fails while in progress (network connection broken, server out of space, etc) after N bytes has been transfered, then the next transfer attempt will start at the correct N-byte offset, so that the existing data which has already been transfered does not need to be moved again.

Download

File download is done by a normal GET request to a URL from the FILE_INFO element. The file offset, if any, is given in Range: attribute of the HTTP header.

Upload

File upload is done using POST operations to a CGI program. A security mechanism prevents unauthorized upload of large amounts of data to the server. Two RPC operations are used.

1) Get file size

The request message has the form:

<data_server_request>
    <core_client_major_version>1</core_client_major_version>
    <core_client_minor_version>1</core_client_minor_version>
    <core_client_release>1</core_client_release>
    <get_file_size>filename</get_file_size>
</data_server_request>

The reply message has the form:

<data_server_reply>
    <status>x</status>
    [ <message>text<message>
    | <file_size>nbytes</file_size> ]
</data_server_reply>

Status is:

0

Success. Nbytes is 0 if the file doesn't exist.

1

Transient error, for example the server is out of disk space. The client should try another data server, or try this one later (starting the transfer at the end of the file already on the server).

-1

Permanent error. The client should give up on the result.

In the error cases, the <file_size> element is omitted and the <message> element gives an explanation.

2) Upload file

Request message format:

<data_server_request>
<core_client_major_version>1</core_client_major_version>
<core_client_minor_version>1</core_client_minor_version>
    <core_client_release>1</core_client_release>
<file_upload>
<file_info>
   ...
<xml_signature>
   ...
</xml_signature>
</file_info>
<nbytes>x</nbytes>
<md5_cksum>x</md5_cksum>
<offset>x</offset>
<data>
... (nbytes bytes of data; may include non-ASCII data)
(no closing tags)

The <file_info> element is the exact text sent from the scheduling server to the client. It includes a signature based on the project's file upload authentication key pair. <nbytes> is the size of the file. <md5_cksum> is MD5 of the entire file. <offset> is the offset within the file.

Reply message format:

<data_server_reply>
    <status>x</status>
    <message>text</message>
</data_server_reply>

Status is:

0

success

1

transient error; The client should try another data server, or try this one later.

-1

Permanent error. The client should give up on the result.

In the error cases, the <message> element gives an explanation.

TODO: if there's an error in the header (bad signature, or file is too large) the client should learn this without actually uploading the file.