Changes between Version 7 and Version 8 of RemoteInputFiles

Timestamp:

Oct 18, 2016, 9:28:30 PM (8 years ago)

Author:

davea

Comment:

--

RemoteInputFiles

@@ -2 +2 @@
 = Remote management of input files =
 
-For a file to be used as an input file of a BOINC job,
-it must be available to BOINC clients via HTTP.
-The standard way to do this is put the file
-in the project's "download directory" on the project server.
+Input files of BOINC jobs must be available on a public web server.
+Usually this is the project's BOINC server.
 
 For projects that use [RemoteJobs remote job submission],
-job submitters don't have login access to the server,
+job submitters don't have login access to the BOINC server,
 so they can't store files there directly.
 Instead, BOINC provides two mechanisms that allow
-job submitters to store and manage files on the BOINC server.
+them to store and manage files on the BOINC server.
 
-Each of these mechanisms deals with several issues:
+'''Job-based file management''': files are automatically transferred
+as part of [RemoteJob remote job submission].
+It's intended to be integrated with such a system;
+job submitters are not aware of it.
+
+'''Per-user file sandbox''': job submitters explicitly maintain,
+via a web interface, a set of files on the server.
+
+Each mechanism deals with several issues:
 
  * '''File immutability''': BOINC requires that a file
@@ -20 +26 @@
   they must be able to submit one job with an input file of a given name,
   and a second job with an input file of the same name but different contents.
- * File cleanup: There must be some way to clean up
-  files on the server when they are no longer needed.
- * Authorization: only users authorized to submit jobs
+ * '''File cleanup''': files on the server must be deleted when they are no longer needed.
+ * '''Authorization''': only users authorized to submit jobs
   should be able to move files to the server.
 
-Note: both mechanisms upload files via PHP.
+'''Note''': both mechanisms upload files via a PHP script.
 PHP's default max file upload size is 2MB.
 To increase this, edit /etc/php.ini, and change, e.g.
@@ -33 +38 @@
 }}}
 
-== Content-based file management ==
+== Job-based file management ==
 
-In this system, the BOINC name of a file
-(i.e. its name on the BOINC server)
-is based on its hash; thus file immutability is automatic.
+In this system, the BOINC name of a file (i.e. its name on the BOINC server)
+includes its hash; thus file immutability is automatic.
 
 File cleanup is based on file/batch associations.
@@ -48 +52 @@
 the RPC handler is html/user/job_files.php.
 
-The following C++ interfaces are provided
-(in lib/remote_submit.cpp).
-This is to be called on the job submission host;
+=== C++ interface ===
+
+The following C++ functions are provided (in lib/remote_submit.cpp).
+They are to be called on the job submission host;
 the files must exist on that host,
 and their MD5s must have already been computed.
@@ -67 +72 @@
  * '''project_url''': the project's master URL
  * '''authenticator''': the job submitter's authenticator.
- * '''boinc_names''': a list of the "BOINC names" of the files.  Must include a hash; can include a suffix if needed.
+ * '''boinc_names''': a list of the "BOINC names" of the files.
+    Must include the MD5 hash; can have a prefix or suffix if needed.
  * '''batch_id''': the ID of a batch whose jobs will reference the files
   (these jobs need not exist yet).
@@ -110 +116 @@
 This will delete files that are no longer associated with an active batch.
 
+=== Python interface ===
+
+The Python interface does both RPCs in one function:
+{{{
+import submit_api
+
+req = UPLOAD_FILES_REQ()
+req.project = project_url
+req.authenticator = get_auth()
+req.batch_id = 271
+req.local_names = ('updater.cpp', 'kill_wu.cpp')
+req.boinc_names = ('xxx_updater.cpp', 'xxx_kill_wu.cpp')
+r = upload_files(req)
+if r[0].tag == 'error':
+        print 'error: ', r[0].find('error_msg').text
+        return
+print 'success'
+}}}
+
 == Per-user file sandbox ==