Changes between Version 3 and Version 4 of VolunteerStorage


Ignore:
Timestamp:
Jul 15, 2011, 12:01:45 AM (13 years ago)
Author:
davea
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • VolunteerStorage

    v3 v4  
    11[[PageOutline]]
    22
    3 = Distributed file management =
     3= Distributed data management =
    44
    5 BOINC allows you to explicitly manage files on client hosts.
    6 This might be used, e.g., to implement a high-latency distributed file system.
     5BOINC provides features for implementing distributed data management systems.
    76
    8 == Retrieving file lists ==
     7Such a system is implemented as several components:
    98
    10 To instruct a host to send a list of all persistent files, use the function
     9 * A "plug-in" to the BOINC scheduler, which is called on each scheduler RPC;
     10 * A daemon program that handles timeouts;
     11 * Interface programs, for example, programs allowing users to submit and retrieve files.
     12
     13Note: to use these features, you must include
    1114{{{
    12 request_file_list(int host_id)
     15<msg_to_host/>
    1316}}}
    14 or the command line program (run from the project root directory)
     17in your config.xml.
     18
     19== Sending files to hosts ==
     20
     21From an interface program or daemon, call
     22
    1523{{{
    16 request_file_list [ -host_id X ]
     24int send_file(int host_id, const char* file_name)
    1725}}}
    18 If -host_id is absent, get file lists for all hosts.  A message is created for the specific host (or all hosts) and added to the  msg_to_host table in the database. The upload message included in the next RPC reply to the host.
    1926
    20 The file list will be included in the next scheduler RPC request. You must modify the scheduler to parse and store it.
     27or use the command line program
    2128
     29{{{
     30send_file --host_id X --file_name Y
     31}}}
    2232
     33From the scheduler plugin, call
     34
     35{{{
     36int send_file_sched(const char* filename);
     37}}}
    2338
    2439== Retrieving files ==
    2540
    26 A persistent file can be retrieved from a specific host. This can be done using the function
     41From an interface program or daemon, call
    2742
    2843{{{
    29 get_file(int host_id, const char* file_name)
     44int get_file(int host_id, const char* file_name)
    3045}}}
    3146
    32 or the command line program (run in the project root dir)
     47or use the command line program
    3348
    3449{{{
    35 get_file -host_id X -file_name Y
     50get_file --host_id X --file_name Y
    3651}}}
    3752
    38 This program must be run in the project's root directory. get_file() creates a result with a name of the form:
     53
     54From the scheduler plugin, call
    3955
    4056{{{
    41 get_FILENAME_HOSTID_TIME
     57int get_file_sched(const char* filename);
    4258}}}
    43 Example: get_test.mpg_34_123456789 is a result representing the  upload of test.mpg from host number 34 at time 1234567891.
    44 
    45 An upload message is created for the specific host and added to the msg_to_host table in the database.  This message instructs the client to upload the file and acknowledge a successful upload. The upload message included in the next RPC reply to the host. The message has the form:
    46 
    47 {{{
    48 <app>
    49     <name>FILE_MOVER</name>
    50 </app>
    51 <app_version>
    52     <app_name>FILE_MOVER</app_name>
    53     <version_num>BOINC_MAJOR_VERSION</version_num>
    54 </app_version>
    55 <file_info>
    56     <name>file_name</name>
    57     <url>upload_dir</url>
    58     <max_nbytes>1e10</max_nbytes>
    59     <upload_when_present/>
    60 </file_info>
    61     RESULT_XML
    62 <workunit>
    63     <name>result.name</name>
    64     <app_name>FILE_MOVER</app_name>
    65 </workunit>
    66 }}}
    67 Include
    68 {{{<msg_to_host/>}}}
    69 in config.xml. Currently
    70 {{{<ignore_upload_certificates/>}}}
    71 needs to be included as there is no way to send upload certificates with these files.
    72 
    73 
    74 == Sending files ==
    75 
    76 To send a file to a specific host, use the function
    77 
    78 {{{
    79 send_file(int host_id, const char* file_name)
    80 }}}
    81 
    82 or the command line program (run from the project root directory)
    83 
    84 {{{
    85 send_file -host_id X -file_name Y
    86 }}}
    87 
    88 send_file creates a result and initializes it with a name of the form send_FILENAME_HOSTID_TIME.
    89 
    90 A message is created for the host and added to the msg_to_host table.  This message instructs the client to download the file and acknowledge a successful download. The download message included in the next RPC reply to the host. The message has the form:
    91 
    92 {{{
    93 <app>
    94         <name>FILE_MOVER</name>
    95 </app>
    96 <app_version>
    97         <app_name>FILE_MOVER</app_name>
    98         <version_num>n</version_num>
    99 </app_version>
    100 <result>
    101     <wu_name>x</wu_name>
    102     <name>y</name>
    103 </result>
    104 <file_info>
    105         <name>file_name</name>
    106         <url>download_dir/file_name</url>
    107         <md5_cksum>md5</md5_cksum>
    108         <nbytes>file->nbytes</nbytes>
    109         <sticky/>
    110 </file_info>
    111 <workunit>
    112         <name>result.name</name>
    113         <app_name>FILE_MOVER</app_name>
    114         <file_ref>
    115                 <file_name>file_name</file_name>
    116         </file_ref>
    117 </workunit>
    118 }}}
    119 
    120 For this to work, you need to include {{{<msg_to_host/>}}} in [ProjectOptions#client-control config.xml].
    121 
    12259
    12360== Deleting files ==
    12461
    125 To delete a file from a host, use the function
     62From an interface program or daemon, call
    12663
    12764{{{
    128 delete_file(int host_id, const char* file_name)
     65int delete_file(int host_id, const char* file_name)
    12966}}}
    13067
    131 or the command line program (run from the project root dir)
     68or use the command line program
    13269
    13370{{{
    … …  
    13572}}}
    13673
    137 `delete_file()` creates a message for the specific host and adds it to the `msg_to_host` table. This message instructs the client to delete the file. The message is included in the next scheduler reply to the host. The message XML has the form
     74From the scheduler plugin, call
    13875
    13976{{{
    140 <delete_file_info>file_name</delete_file_info>
     77int delete_file_sched(const char* filename);
    14178}}}
    14279
    143 For this to work, you need to include {{{<msg_to_host/>}}} in [ProjectOptions#client-control config.xml].
     80== Implementation ==
    14481
     82From interface programs,
     83'''put_file''' and '''get_file''' create a message to the host (in the msg_to_host table).
     84The message is a chunk of XML that describes a "virtual" app and app version
     85(with no associate executable),
     86and a workunit and result that contain a single input or output file.
     87The result has a name of the form '''file_xfer_*''',
     88which tells the scheduler to treat it specially.