Changes between Version 11 and Version 12 of RemoteInputFiles
- Timestamp:
- Jan 30, 2017, 4:06:58 PM (8 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
RemoteInputFiles
v11 v12 1 1 [[PageOutline]] 2 = Remote management of input files=2 = Job-based input file management = 3 3 4 4 Input files of BOINC jobs must be available on a public web server. 5 Usually this is the project's BOINC server.6 7 5 For projects that use [RemoteJobs remote job submission], 8 6 job submitters don't have login access to the BOINC server, 9 7 so they can't store files there directly. 10 Instead, BOINC provides two mechanisms that allow11 them to store and manage files on the BOINC server.12 8 13 '''Job-based file management''': files are automatically transferred 14 as part of [RemoteJob remote job submission]. 15 It's intended to be integrated with such a system; 16 job submitters are not aware of it. 9 There are several options: 17 10 18 '''Per-user file sandbox''': job submitters explicitly maintain, 19 via a web interface, a set of files on the server. 11 * Files are served from a publicly-accessible server, 12 possibly other than the BOINC server. 13 They must be managed, and file immutability enforced, 14 by a mechanism outside BOINC. 15 * [FileSandbox Per-user file sandbox]: job submitters explicitly maintain, 16 via a web interface, a set of files on the server. 17 * '''Job-based file management''': files are automatically transferred 18 from the submission machine to the BOINC server via Web RPCs. 20 19 21 Each mechanism deals with several issues: 20 This document describes the latter mechanism. 22 21 23 * '''File immutability''': BOINC requires that a file24 of a given name can never be changed.25 * '''File cleanup''': files on the server must be deleted when they are no longer needed.26 * '''Authorization''': only users authorized to submit jobs27 should be able to move files to the server.28 29 '''Note''': both mechanisms upload files via a PHP script.30 PHP's default max file upload size is 2MB.31 To increase this, edit /etc/php.ini, and change, e.g.32 {{{33 upload_max_filesize = 64M34 post_max_size = 64M35 }}}36 37 == Job-based file management ==38 22 39 23 In this system, you must supply physical names of files … … 42 26 43 27 File cleanup is based on file/batch associations. 28 You must create a batch (with create_batch()) 29 before querying or uploading files. 44 30 Each file can be associated with one or more batches. 45 31 Files that are no longer associated with an active batch are … … 50 36 the RPC handler is html/user/job_files.php. 51 37 52 This mechanism requires you to create a batch (with create_batch()) 53 before querying or uploading files. 54 55 === C++ interface === 38 == C++ interface == 56 39 57 40 The following C++ functions are provided (in lib/remote_submit.cpp). 58 41 They are to be called on the job submission host; 59 the files must exist on that host, 60 and their MD5s must have already been computed. 42 the files must exist on that host. 61 43 {{{ 62 44 extern int query_files( 63 45 const char* project_url, 64 46 const char* authenticator, 65 std::vector<string> &boinc_names, 47 std::vector<string> &boinc_names, // must be unique, e.g. by including content hash 66 48 int batch_id, 67 49 std::vector<int> &absent_files, // output … … 117 99 This will delete files that are no longer associated with an active batch. 118 100 119 == = Python interface ===101 == Python interface == 120 102 121 103 The Python interface does both RPCs in one function (in lib/submit_api.py): … … 136 118 }}} 137 119 138 == Per-user file sandbox==120 == File size limits == 139 121 140 This mechanism allows job submitters to explicitly 141 upload files via a web interface: PROJECT_URL/sandbox.php. 142 143 Links to the files are stored in a "sandbox directory" 144 PROJECT_ROOT/sandbox/USERID/. 145 The entries in this directory have contents 122 '''Note''': This mechanism upload files via a PHP script. 123 PHP's default max file upload size is 2MB. 124 To increase this, edit /etc/php.ini, and change, e.g. 146 125 {{{ 147 size MD5 126 upload_max_filesize = 64M 127 post_max_size = 64M 148 128 }}} 149 129 150 The actual files are stored in the download directory,151 under the name '''sb_userid_MD5'''.152 153 Currently, files in the sandbox are not cleanup up automatically.154 The web interface allows users to delete their files.
