wiki:JobSubmission

Version 4 (modified by davea, 15 years ago) (diff)

--

  1. Submitting jobs
    1. Input template files
    2. Output template files
    3. Putting input files in the download directory
  2. Submitting jobs
    1. Input and output template files
    2. Staging input files
    3. Submitting a job manually
    4. Submitting jobs from a C++ program

Submitting jobs

To submit a job you must

  1. Write XML 'template files' that describe the job's input and outputs (typically the same template files can be used for many jobs).
  2. Create the job's input file(s), and put them in the right place (as determined by the file name) in the download directory hierarchy.
  3. Invoke a BOINC function or script that submits the job.

Once this is done, BOINC takes over: it creates one or more instances of the job, distributes them to client hosts, collects the output files. It validates? and processes the results, and deletes the input and output files.

Typically, steps 2) and 3) are done by a work generator program that creates lots of jobs.

Input template files

An input template file has the form 

<file_info>
    <number>0</number>
    [ <sticky/> ]
    [ <nodelete/> ]
    [ <report_on_rpc/> ]
</file_info>
[ ... other files  ]
<workunit>
    <file_ref>
        <file_number>0</file_number>
        <open_name>NAME</open_name>
        [ <copy_file/> ]
    </file_ref>
    [ ... other files ]
    [ <command_line>-flags xyz</command_line> ]
    [ <rsc_fpops_est>x</rsc_fpops_est> ]
    [ <rsc_fpops_bound>x</rsc_fpops_bound> ]
    [ <rsc_memory_bound>x</rsc_memory_bound> ]
    [ <rsc_disk_bound>x</rsc_disk_bound> ]
    [ <delay_bound>x</delay_bound> ]
    [ <min_quorum>x</min_quorum> ]
    [ <target_nresults>x</target_nresults> ]
    [ <max_error_results>x</max_error_results> ]
    [ <max_total_results>x</max_total_results> ]
    [ <max_success_results>x</max_success_results> ]
    [ <credit>X</credit> ]
</workunit>

Elements and tags must be on separate lines as shown. The components are:

<file_info>
describes an input file.
<open_name>
the physical name of the file.
<sticky/>
if present, the file remains on the client after job is finished.
<nodelete/>
if present, the file is not deleted from the server after job is completed.
<report_on_rpc/>
if present, report file in each scheduler request (sticky files)
<file_ref>
describes the way the file is referenced.
<open_name>
the logical name of the file
<copy_file>
if present, the file is copied into the job's slot directory
<command_line>
The command-line arguments to be passed to the main program.
<credit>
The amount of credit to be granted for successful completion of this workunit. Use this only if you know in advance how many FLOPs it will take. Your validator must use get_credit_from_wu() as its compute_granted_credit() function.
<rsc_fpops_est> etc.
Job attributes such has how much disk space will be used. BOINC will supply reasonable defaults for these, but you should supply the correct values; otherwise, for example, BOINC might try to run the job on a host with insufficient disk space.

Output template files

An output template file has the form 

<file_info>
    <name><OUTFILE_0/></name>
    <generated_locally/>
    <upload_when_present/>
    <max_nbytes>32768</max_nbytes>
    <url><UPLOAD_URL/></url>
</file_info>
<result>
    <file_ref>
        <file_name><OUTFILE_0/></file_name>
        <open_name>result.sah</open_name>
        [ <optional>0|1</optional> ]
        [ <no_validate>0|1</no_validate> ]
    </file_ref>
</result>

Elements and tags must be on separate lines as shown. The elements include:

<file_info>
describes an output file.
<name>
the physical file name. Typically use <OUTFILE_0>, <OUTFILE_1> etc.; BOINC will replace this with a generated name based on the job name.
<file_ref>
describes how an output file will be referenced by the application.
<open_name>
the "logical name" by which the application will reference the file.
<copy_file/>
if present, the file will be generated in the slot directory, and copied to the project directory after the job has finished. Use this for legacy applications.
<generated_locally/>
always include this for output files.
<max_nbytes>
maximum file size. If the actual size exceeds this, the file will not be uploaded, and the job will be marked as an error.
<url>
the URL of the file upload handler. You may include this explicitly, or use <UPLOAD_URL/> to use the URL in your project's config.xml file.
<optional>
if 0 or absent, your application must create the file, otherwise the job will be marked as an error.
<no_validate>
if true, don't include this file in the result validation process (relevant only if you are using the sample bitwise validator).
<no_delete/>
if present, the file will not be deleted on the server even after the job is finished.

Note: when a job is created, the name of its output template file is stored in the database. The file is read when instances of the job are created, which may happen days or weeks later. Thus, editing an output template file can affect existing jobs. If this is not desired, you must create a new output template file.

You can safely remove a workunit template file after creating your last WU with it. However, you can't delete a result template file until any WU that refers to it is completed (i.e. no more replicas will be created).

Putting input files in the download directory

To move an input file to the download directory, use 

dir_hier_path filename

which prints the full pathname and creates the directory if needed (run this in the project's root directory). For example: 

cp test_files/12ja04aa `bin/dir_hier_path 12ja04aa`

copies an input file from the test_files/ directory to the download directory hierarchy.

  1. Submitting jobs
    1. Input template files
    2. Output template files
    3. Putting input files in the download directory
  2. Submitting jobs
    1. Input and output template files
    2. Staging input files
    3. Submitting a job manually
    4. Submitting jobs from a C++ program

Submitting jobs

To submit a job you must

  1. Write XML 'template files' that describe the job's input and outputs (typically the same template files can be used for many jobs).
  2. Create the job's input file(s), and put them in the right place (as determined by the file name) in the download directory hierarchy.
  3. Invoke a BOINC function or script that submits the job.

Once this is done, BOINC takes over: it creates one or more instances of the job, distributes them to client hosts, collects the output files. It validates? and processes the results, and deletes the input and output files.

Typically, steps 2) and 3) are done by a work generator program that creates lots of jobs.

Input and output template files

An input template file has the form 

<file_info>
    <number>0</number>
    [ <sticky/> ]
    [ <nodelete/> ]
    [ <report_on_rpc/> ]
</file_info>
[ ... other files  ]
<workunit>
    <file_ref>
        <file_number>0</file_number>
        <open_name>NAME</open_name>
        [ <copy_file/> ]
    </file_ref>
    [ ... other files ]
    [ <command_line>-flags xyz</command_line> ]
    [ <rsc_fpops_est>x</rsc_fpops_est> ]
    [ <rsc_fpops_bound>x</rsc_fpops_bound> ]
    [ <rsc_memory_bound>x</rsc_memory_bound> ]
    [ <rsc_disk_bound>x</rsc_disk_bound> ]
    [ <delay_bound>x</delay_bound> ]
    [ <min_quorum>x</min_quorum> ]
    [ <target_nresults>x</target_nresults> ]
    [ <max_error_results>x</max_error_results> ]
    [ <max_total_results>x</max_total_results> ]
    [ <max_success_results>x</max_success_results> ]
    [ <credit>X</credit> ]
</workunit>

Elements and tags must be on separate lines as shown. The components are:

<file_info>
describes an input file.
<open_name>
the physical name of the file.
<sticky/>
if present, the file remains on the client after job is finished.
<nodelete/>
if present, the file is not deleted from the server after job is completed.
<report_on_rpc/>
if present, report file in each scheduler request (sticky files)
<file_ref>
describes the way the file is referenced.
<open_name>
the logical name of the file
<copy_file>
if present, the file is copied into the job's slot directory
<command_line>
The command-line arguments to be passed to the main program.
<credit>
The amount of credit to be granted for successful completion of this workunit. Use this only if you know in advance how many FLOPs it will take. Your validator must use get_credit_from_wu() as its compute_granted_credit() function.
<rsc_fpops_est> etc.
Work unit attributes

An output template file has the form 

<file_info>
    <name><OUTFILE_0/></name>
    <generated_locally/>
    <upload_when_present/>
    <max_nbytes>32768</max_nbytes>
    <url><UPLOAD_URL/></url>
</file_info>
<result>
    <file_ref>
        <file_name><OUTFILE_0/></file_name>
        <open_name>result.sah</open_name>
        [ <optional>0|1</optional> ]
        [ <no_validate>0|1</no_validate> ]
    </file_ref>
</result>

Elements and tags must be on separate lines as shown. The elements include:

<file_info>
describes an output file.
<name>
the physical file name. Typically use <OUTFILE_0>, <OUTFILE_1> etc.; this will be replaced with a generated name based on the job name.

<file_ref> describes an the output file will be referenced by the application.

<open_name>
the "logical name" by which your application will reference the file.
<generated_locally/>
always include this for output files.
<max_nbytes>
maximum file size. If the actual size exceeds this, the file will not be uploaded, and the job will be marked as an error.
<url>
the URL of the file upload handler. You may include this explicitly, or use <UPLOAD_URL/> to use the URL in your project's config.xml file.
<optional>
if 0 or absent, your application must create the file, otherwise the job will be marked as an error.
<no_validate>
if true, don't include this file in the result validation process (relevant only if you are using the sample bitwise validator).

Note: when a job is created, the name of its output template file is stored in the database. The file is read when instances of the job are created, which may happen days or weeks later. Thus, editing an output template file can affect existing jobs. If this is not desired, you must create a new output template file.

You can safely remove a workunit template file after creating your last WU with it. However, you can't delete a result template file until any WU that refers to it is completed (i.e. no more replicas will be created).

Staging input files

Before submitting a job, you must put its input files into the proper place in the download directory hierarchy. BOINC provides both a command-line tool and a function for this. The command-line tool is: 

dir_hier_path filename

which prints the full pathname and creates the directory if needed (run this in the project's root directory). For example: 

cp test_files/12ja04aa `bin/dir_hier_path 12ja04aa`

copies an input file from the test_files directory to the download directory hierarchy.

The function (a member function of the SCHED_CONFIG class) is 

// convert filename to path in download hierarchy
//
SCHED_CONFIG::download_path(
    const char* filename,    // name of input file
    char* path               // path in download hierarchy where it should go
);

Submitting a job manually

create_work is a command-line tool for submitting jobs. Run it from the project root directory. 

create_work [ arguments ] infile_1 ... infile_n

Mandatory arguments are:

--appname name
application name
--wu_name name
workunit name
--wu_template filename
WU template filename relative to project root; usually in templates/
--result_template filename
result template filename, relative to project root; usually in templates/

Optional arguments are:

--batch n
--priority n

The following job parameters may be passed in the input template, or as command-line arguments to create_work, or not passed at all, in which case defaults will be used.

--command_line "-flags foo"
--rsc_fpops_est x
FLOPs estimate
--rsc_fpops_bound x
--rsc_memory_bound x
--rsc_disk_bound x
--delay_bound x
--min_quorum x
--target_nresults x
--max_error_results x
--max_total_results x
--max_success_results x
--opaque N
--additional_xml 'x'
This can be used to supply, for example, <credit>12.4</credit>.

Submitting jobs from a C++ program

BOINC's library provides a function for submitting jobs: 

int create_work(
    DB_WORKUNIT& wu,
    const char* wu_template,                  // contents, not path
    const char* result_template_filename,     // relative to project root
    const char* result_template_filepath,     // absolute or relative to current dir
    const char** infiles,                     // array of input file names
    int ninfiles
    SCHED_CONFIG&,
    const char* command_line = NULL,
    const char* additional_xml = NULL
);

The name and appid fields of the DB_WORKUNIT structure must always be initialized. Other job parameters may be passed either in the DB_WORKUNIT structure or in the input template file (the latter has priority). On a successful return, wu.id contains the database ID of the workunit.