Changes between Version 9 and Version 10 of CondorBoinc
- Timestamp:
- Nov 24, 2012, 12:09:10 AM (11 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
CondorBoinc
v9 v10 6 6 so that a BOINC-based volunteer computing project can provide computing resources to a Condor pool. 7 7 8 A n importantdesign goal is transparency:8 A central design goal is transparency: 9 9 from the job submitter's viewpoint, 10 things should look as much like Condor as possible:10 things should look exactly like Condor: 11 11 i.e. they prepare Condor submit files and use condor_submit. 12 12 … … 20 20 In Condor, a file is associated with a job, and has a single name. 21 21 * BOINC is designed for apps for which the number and names of output files 22 is fixed at the time of job submission ;22 is fixed at the time of job submission. 23 23 Condor doesn't have this restriction. 24 24 25 25 * Application concept: 26 26 In Condor, a job is associated with a single executable, and can run 27 only on hosts of the appropriate platform (and possibly other attributes,28 as specified by the job's !ClassAd).27 only on hosts of the appropriate platform 28 (and possibly other attributes, as specified by the job's !ClassAd). 29 29 In BOINC, there may be many app versions for a single application: 30 30 e.g. versions for different platforms, GPU types, etc. 31 31 A job is associated with an application, not an app version. 32 32 33 == Applications ==34 35 Applications could run on BOINC in several ways:36 * As native BOINC applications.37 This would requiremaking source-code modifications and recompiling38 for different platforms, linking with the BOINC API library. Too complex.39 * In virtual machines.33 == BOINC environment choices == 34 35 BOINC offers three "environments" in which applications can be deployed: 36 * '''Native''': 37 This requires making source-code modifications and recompiling 38 for different platforms, linking with the BOINC API library. 39 * '''Virtual machine-based''': 40 40 This would eliminate multi-platform issues 41 41 but would require volunteer hosts to have VirtualBox installed. 42 Maybe someday. 43 * Using the BOINC wrapper. 42 * '''BOINC wrapper''': 44 43 Requires apps to be built for different platforms, but no source code mods. 45 Let's use this to start. 44 45 Using the BOINC wrapper is the path of least resistance at this point. 46 46 47 47 The Condor pool admins will select a set of applications to run under BOINC. … … 56 56 57 57 Goal: minimize data transfer and storage on the BOINC server. 58 To do this, we'll add the following to BOINC: 59 60 * DB tables for files, and for batch/file associations 61 * daemon for deleting files and DB records of files with 62 no associations, or past all lease ends 58 To do this, we'll add the following mechanism to BOINC: 59 60 * DB tables for files, and for batch/file associations (with lease ends). 61 File names will be based on MD5s. 62 * Web RPCs for querying and uploading files. 63 * Daemon for deleting files and DB records of files with 64 no associations, or past all lease ends. 63 65 64 66 For output files, we'll take the approach that each job has (from BOINC's viewpoint) 65 a single output file, which is a zipped archive of its actual output s.67 a single output file, which is a zipped archive of its actual output files. 66 68 This will get copied to the submitter host, unzipped, 67 69 and its components moved to the appropriate directory. … … 69 71 == Job submission mechanism == 70 72 71 We'll use Condor's existing mechanism for sending jobs to 72 non-Condor back ends. 73 We'll use Condor's existing mechanism for sending jobs to non-Condor back ends. 73 74 This will involve 2 components: 74 75 … … 79 80 * Periodically poll the BOINC server for completed jobs; 80 81 when a job is newly completed, 81 82 82 download its output from the BOINC server, 83 and store it into the appropriate directories on the submit node. 83 84 * A new class in Condor's job_router for managing communication 84 85 with the BOINC GAHP. … … 98 99 job name 99 100 cmdline 100 for each input file 101 path on submit node 102 name by which app will open file 103 bool return_all_output_files 104 if the above not set: for each output file 105 open name (what the app will create) 106 final name (e.g. may have process appended) 107 directory where output file(s) go 101 for each input file 102 path on submit node 103 name by which app will open file 104 bool return_all_output_files (or regular expression) 105 if the above not set 106 for each output file 107 open name (what the app will create) 108 final name (e.g. may have process appended) 109 directory where output file(s) go 108 110 output: 109 111 error code 110 112 }}} 111 113 112 What the BOINC GAHP does:114 The BOINC GAHP handles this as follows: 113 115 114 116 * Make list of all input files … … 186 188 * All jobs belong to a single BOINC account. 187 189 188 189 190 == Changes to BOINC == 190 191 … … 194 195 * Add lease_end field to batch 195 196 196 == Implementation notes ==197 198 The BOINC GAHP could be implemented in P HP, Python,or C++.197 == BOINC GAHP implementation notes == 198 199 The BOINC GAHP could be implemented in Python or C++. 199 200 My inclination is to use Python; we can assume it's available on the submit node.
