Applications that use coprocessors

BOINC supports applications that use coprocessors. The supported coprocessor types (as of [24404])are NVIDIA and ATI GPUs.

The BOINC client probes for coprocessors and reports them in scheduler requests. The client keeps track of coprocessor allocation, i.e. how many instances of each are free. It only runs an app if enough instances are available.

You can develop your application using any programming system, e.g. CUDA (for NVIDIA), CAL (for ATI) or OpenCL.

Dealing with GPU memory allocation failures

GPUs don't have virtual memory. GPU memory allocations may fail because other applications are using the GPU. This is typically a temporary condition. Rather than exiting with an error in this case, call

boinc_temporary_exit(60);

This will exit the application, and will tell the BOINC client to restart it again in at least 60 seconds, at which point memory may be available.

Device selection

Some hosts have multiple GPUs. When your application is run by BOINC, it receives information about which GPU instance to use. This is passed as a command-line argument

--device N

where N is the device number of the GPU that is to be used. If your application uses multiple GPUs, it will be passed multiple --device arguments, e.g.

--device 0 --device 3

Some OpenCL apps can use either NVIDIA or ATI GPUs, so they must also be told which type of GPU to use. This is passed in the APP_INIT_DATA structure returned by boinc_get_init_data().

char gpu_type[64];     // "nvidia" or "ati"
int gpu_device_num;

Cleanup on premature exit

The BOINC client may kill your application during execution. This may leave the GPU in a bad state. To prevent this, call

boinc_begin_critical_section();

before using the GPU, and between GPU kernels do

if (boinc_status.quit_request || boinc_status.abort_request) {
    // cudaThreadSynchronize(); or whatever is needed
    boinc_end_critical_section();
    exit(0);
}

Plan classes

Each coprocessor application has an associated plan class which determines the hardware and software resources that are needed to run the application.

NOTE: plan classes for AMD GPUs must contain the substring 'ati'; plan classes for NVIDIA GPUs much contain either 'nvidia' or 'cuda' as a substring.

The following plan classes for NVIDIA are pre-defined:

cuda

NVIDIA GPU, compute capability 1.0+, driver version 177.00+, 254+ MB RAM.

cuda23

Requires driver version 190.38+, 384+ MB RAM.

cuda_fermi

Requires compute capability 2.0+ and CUDA version 3.0+

opencl_nvidia_101

Requires OpenCL 1.1+ support

For ATI the situation is more complex because AMD changed the DLL names from amd* to ati* midstream; applications are linked against a particular name and will fail if it's not present.

ati

CAL version 1.0.0+, amd* DLLs

ati13amd

CAL version 1.3+, amd* DLLs

ati13ati

CAL version 1.3+, ati* DLLs

ati14

CAL version 1.4+, ati* DLLs

opencl_ati_101

OpenCL 1.1+

You can verify which DLLs your application is linked against by using ​Dependency Walker against your application. If your executable contains DLL names prefixed with 'amd' then your plan class will be ati or ati13amd depending on which version of the CAL SDK you are using. If the DLL names are prefixed with 'ati' then use the ati13ati or ati14 plan classes.

In all cases (NVIDIA and ATI), the application is assumed to use 1 GPU, and the CPU usage is assumed to be 0.5% the FLOPS of the GPU. If there is a choice, the scheduler will give preference to later classes, i.e. it will pick cuda23 over cuda.

Once you have chosen a plan class for your executable, create an app version, specifying its plan class.

Defining a custom plan class

If your application has properties that differ from any of the pre-defined classes, you can modify them, or better yet define your own.

To define a new NVIDIA/CUDA plan class, add a new clause to app_plan_cuda() in sched/sched_customize.cpp. For example, the plan class cuda23 is defined by:

    ...
    if (!strcmp(plan_class, "cuda23")) {
        if (!cuda_check(c, hu,
            100,        // minimum compute capability (1.0)
            200,        // max compute capability (2.0)
            2030,       // min CUDA version (2.3)
            19500,      // min display driver version (195.00)
            384*MEGA,   // min video RAM
            1.,         // # of GPUs used (may be fractional, or an integer > 1)
            .01,        // fraction of FLOPS done by the CPU
            .21            // estimated GPU efficiency (actual/peak FLOPS)
        )) {
            return false;
        }
    }

To define a new ATI/CAL plan class, add a new clause to app_plan_ati(). For example:

    if (!strcmp(plan_class, "ati14")) {
        if (!ati_check(c, hu,
            1004000,    // min display driver version (10.4)
            false,      // require libraries named "ati", not "amd"
            384*MEGA,   // min video RAM
            1.,         // # of GPUs used (may be fractional, or an integer > 1)
            .01,        // fraction of FLOPS done by the CPU
            .21         // estimated GPU efficiency (actual/peak FLOPS)
        )) {
            return false;
        }
    }

To define a new OpenCL plan class, add a new clause to app_plan_opencl(). For example:

    if (!strcmp(plan_class, "opencl_nvidia_101")) {
        return opencl_check(
            c, hu,
            101,        // OpenCL version (1.1)
            256*MEGA,   // min video RAM
            1,          // # of GPUs used
            .1,         // fraction of FLOPS done by the CPU
            .21         // estimated GPU efficiency (actual/peak FLOPS)
        );
    }