Client data model

This document describes proposed changes to the client to support distributed storage.

Current

FILE_INFO elements

• status (present, not present, error)
• urls
• bool generated_locally
• bool upload_when_present
• bool uploaded
• bool sticky

Problems

Many. Example: suppose the server asks the client to upload a file that the client doesn't have. Since generated_locally is false and the file is not present, the client will try to download it (from the upload URL!).

Proposed

FILE_INFO elements:

• status
• upload_urls
• download_urls
• bool uploaded
• bool sticky

Policy:

• If a file has a download URL and is not present, download it
• If a file has an upload URL, is present, and uploaded is false, upload it
• start a job if its input files are either present or optional_input

Handling <file_info> elements in scheduler replies:

• if referenced from an app version or workunit, store URLs in download_urls
• if referenced from a result, store URL in upload_urls.

Deprecated fields in scheduler replies

• <generated_locally>
• <upload_when_present>

Handling upload requests:

• Clear "uploaded" flag
• If the file isn't present, mark result as error and put appropriate text in stderr_out.

Compatibility: we'll change <file_info> to <file> in client_state.xml. We'll parse <file_info> elements (for upward compatibility). We won't provide backward compatibility (i.e., if you upgrade to 6.13.x, then downgrade to 6.12, all tasks and app versions will disappear).

Locally-generated input files

One (hypothetical) class of files: input files which, if not present, are generated computationally by the app. Such files should be listed (in sched reply) as sticky optional input files with no download URL, and as optional output files (this causes them to be marked as present).

The app must use file locking to ensure that two jobs don't try to generate the file at the same time.

Upload certificates

Upload certificates are a mechanism to prevent upload server DoS attacks, and also to prevent an attack in which bad guys upload result files for other users' jobs. Currently no projects that I know of use this mechanism, but it is worth keeping available.

Current: the upload certificate is a signature of the rest of the <file_info> element.

Problem: this makes <file_info> elements sensitive to white space changes, and it makes it difficult to parse <file_info> using XML_PARSER.

Solution: the upload certificate is a signature of

<name>x</name><max_nbytes>N</max_nbytes>

(no white space, N written with %.0f).

Server changes

• file_upload_handler: parse the request with XML_PARSER, then convert to above form
• process_result_template.cpp: generate signatures as above