| 7 | | |
| 8 | | Starting with version 5.4, |
| 9 | | the BOINC client is able to handle HTTP `Content-Encoding` types |
| 10 | | 'deflate' (zlib algorithm) and 'gzip' (gzip algorithm). |
| 11 | | The client decompresses these files 'on the fly' |
| 12 | | and stores them on disk in uncompressed form. |
| 13 | | This can be used in the following two ways. |
| 14 | | |
| 15 | | Both methods store files uncompressed on the client. |
| 16 | | If you need compression on the client, |
| 17 | | you must do it at the application level (see below). |
| | 6 | Starting with version 5.4, the BOINC client is able to handle HTTP `Content-Encoding` types 'deflate' (zlib algorithm) and 'gzip' (gzip algorithm). The client decompresses these files 'on the fly' and stores them on disk in uncompressed form. This can be used in the following two ways. |
| | 7 | |
| | 8 | Both methods store files uncompressed on the client. If you need compression on the client, you must do it at the application level (see below). |
| 20 | | |
| 21 | | To use this method, gzip your downloadable files, |
| 22 | | giving them a filename suffix such as '.gz'. |
| 23 | | (The name used in your `<file_info>` elements, |
| 24 | | however, is the original filename without '.gz'). |
| 25 | | |
| 26 | | Include the following line in `httpd.conf`: |
| 27 | | {{{ |
| 28 | | AddEncoding x-gzip .gz |
| 29 | | }}} |
| 30 | | and restart apache. |
| 31 | | |
| 32 | | This method has the advantage of reducing server disk usage and server CPU load, |
| 33 | | but it will only work with 5.4+ clients. |
| 34 | | BOINC clients older than 5.4 won't be able to download files. |
| 35 | | Use the 'min_core_client_version' entry in config.xml to enforce this. |
| | 11 | To use this method, gzip your downloadable files, giving them a filename suffix such as '.gz'. (The name used in your `<file_info>` elements, however, is the original filename without '.gz'). |
| | 12 | |
| | 13 | This method has the advantage of reducing server disk usage and server CPU load, but it will only work with 5.4+ clients. BOINC clients older than 5.4 won't be able to download files. Use the 'min_core_client_version' entry in config.xml to enforce this. |
| 38 | | |
| 39 | | You can use the Apache 2.0 mod_deflate module to automatically compress files on the fly. |
| 40 | | See http://httpd.apache.org/docs/2.0/mod/mod_deflate.html. |
| 41 | | This method will work with all BOINC clients, |
| 42 | | but it will do compression only for 5.4+ clients. |
| 43 | | |
| 44 | | You can use this in conjunction with gzip encoding because the mod_deflate module |
| 45 | | allows you to exempt certain filetypes from on-the-fly compression. |
| 46 | | |
| 47 | | This method increases CPU load on the web server, |
| 48 | | but this is typically not significant. |
| | 16 | You can use the Apache 2.0 mod_deflate module to automatically compress files on the fly. See http://httpd.apache.org/docs/2.0/mod/mod_deflate.html. This method will work with all BOINC clients, but it will do compression only for 5.4+ clients. |
| | 17 | |
| | 18 | You can use this in conjunction with gzip encoding because the mod_deflate module allows you to exempt certain filetypes from on-the-fly compression. |
| | 19 | |
| | 20 | This method increases CPU load on the web server, but this is typically not significant. |
| | 21 | |
| | 22 | ==== Configuration File ==== |
| 89 | | |
| 90 | | If you include the `<gzip_when_done>` tag in an [XmlFormat#Files output file description], |
| 91 | | the file will be gzip-compressed after it has been generated. |
| 92 | | |
| 93 | | The gzip_when_done is only supported in client version 5.8+. |
| 94 | | If you receive files from clients that do not support the gzip_when_done flag, |
| 95 | | then you should open the files with a function similar |
| 96 | | to this to your validator/assimilator: |
| | 81 | If you include the `<gzip_when_done>` tag in an [wiki:XmlFormat#Files output file description], the file will be gzip-compressed after it has been generated. |
| | 82 | |
| | 83 | The gzip_when_done is only supported in client version 5.8+. If you receive files from clients that do not support the gzip_when_done flag, then you should open the files with a function similar to this to your validator/assimilator: |
| | 84 | |
| 129 | | |
| 130 | | You can also do compression in your application. |
| 131 | | To assist this, BOINC provides a library |
| 132 | | boinc_zip, based on the [http://www.info-zip.org Info-Zip] libraries, |
| 133 | | but combines both zip & unzip |
| 134 | | functionality in one library. |
| 135 | | Any questions/comments please email Carl Christensen (carlgt1 at yahoo dot com) |
| | 114 | You can also do compression in your application. To assist this, BOINC provides a library boinc_zip, based on the [http://www.info-zip.org Info-Zip] libraries, but combines both zip & unzip functionality in one library. Any questions/comments please email Carl Christensen (carlgt1 at yahoo dot com) |
| 145 | | The "unzip" functionality is there, that is you can unzip |
| 146 | | a file and it will create all directories & files in the zip file. |
| 147 | | The "zip" functionality has some limitations due to the cross-platform |
| 148 | | nature: mainly it doesn't provide zipping recursively (i.e. |
| 149 | | subdirectories); and wildcard handling is done using the "boinc_filelist" |
| 150 | | function which will be explained below. |
| | 121 | The "unzip" functionality is there, that is you can unzip a file and it will create all directories & files in the zip file. The "zip" functionality has some limitations due to the cross-platform nature: mainly it doesn't provide zipping recursively (i.e. subdirectories); and wildcard handling is done using the "boinc_filelist" function which will be explained below. |
| 153 | | |
| 154 | | For Windows, you can just add the project "boinc_zip" to your |
| 155 | | Visual Studio "Solution" or "Workspace." Basically just "Insert Existing |
| 156 | | Project" from the Visual Studio IDE, navigate over to the boinc/zip |
| 157 | | directory, and it should load the appropriate files. You can then build |
| 158 | | "Debug" and "Release" versions of the library. Then just add the |
| 159 | | appropriate reference to "boinc_zip.lib" (Release build) or "boinc_zipd.lib" |
| 160 | | (Debug build) in your app. |
| 161 | | |
| 162 | | For Linux & Mac, you should be able to run "./configure" and then do a "make" |
| 163 | | to build the "libboinc_zip.a" lib that you will link against. In extreme |
| 164 | | cases, you may need to do an "aclocal && autoconf && automake" first, |
| 165 | | to build properly for your platform. |
| 166 | | |
| 167 | | Also, please note that boinc_zip relies on some BOINC functions that you will need |
| 168 | | (and will most likely be in your app already since they are handy) -- |
| 169 | | namely `boinc/lib/filesys.C` and `boinc/lib/util.C`. |
| | 124 | For Windows, you can just add the project "boinc_zip" to your Visual Studio "Solution" or "Workspace." Basically just "Insert Existing Project" from the Visual Studio IDE, navigate over to the boinc/zip directory, and it should load the appropriate files. You can then build "Debug" and "Release" versions of the library. Then just add the appropriate reference to "boinc_zip.lib" (Release build) or "boinc_zipd.lib" (Debug build) in your app. |
| | 125 | |
| | 126 | For Linux & Mac, you should be able to run "./configure" and then do a "make" to build the "libboinc_zip.a" lib that you will link against. In extreme cases, you may need to do an "aclocal && autoconf && automake" first, to build properly for your platform. |
| | 127 | |
| | 128 | Also, please note that boinc_zip relies on some BOINC functions that you will need (and will most likely be in your app already since they are handy) -- namely `boinc/lib/filesys.C` and `boinc/lib/util.C`. |
| 172 | | Basically, you will need to `#include "boinc_zip.h"` in your app (of course |
| 173 | | your compiler will need to know where it is, i.e. -I../boinc/zip). |
| 174 | | |
| 175 | | Then you can just call the function `boinc_zip` with the appropriate arguments |
| 176 | | to zip or unzip. There are three overloaded boinc_zip's provided: |
| | 131 | Basically, you will need to `#include "boinc_zip.h"` in your app (of course your compiler will need to know where it is, i.e. -I../boinc/zip). |
| | 132 | |
| | 133 | Then you can just call the function `boinc_zip` with the appropriate arguments to zip or unzip. There are three overloaded boinc_zip's provided: |
| | 134 | |
| 186 | | `szFileZip` is the name of the zip file to create or extract |
| 187 | | (I assume the user will provide it with the .zip extension) |
| 188 | | |
| 189 | | The main differences are in the file parameter. The zip library used was |
| 190 | | exhibiting odd behavior when "coexisting" with unzip, particularly in the |
| 191 | | wildcard handling. So a function was made that creates a `ZipFileList` class, |
| 192 | | which is basically a vector of filenames. If you are just compressing a |
| 193 | | single file, you can use either the `std::string` or `const char* szFileIn` overrides. |
| | 144 | `szFileZip` is the name of the zip file to create or extract (I assume the user will provide it with the .zip extension) |
| | 145 | |
| | 146 | The main differences are in the file parameter. The zip library used was exhibiting odd behavior when "coexisting" with unzip, particularly in the wildcard handling. So a function was made that creates a `ZipFileList` class, which is basically a vector of filenames. If you are just compressing a single file, you can use either the `std::string` or `const char* szFileIn` overrides. |
| 197 | | To zip multiple files in a "mix & match" fashion, you can use the `boinc_filelist` |
| 198 | | function provided. Basically, it's a crude pattern matching of files in a |
| 199 | | directory, but it has been useful for us on the CPDN project. Just create a |
| 200 | | `ZipFileList` instance, and then pass this into `boinc_filelist` as follows: |
| | 150 | To zip multiple files in a "mix & match" fashion, you can use the `boinc_filelist` function provided. Basically, it's a crude pattern matching of files in a directory, but it has been useful for us on the CPDN project. Just create a `ZipFileList` instance, and then pass this into `boinc_filelist` as follows: |
| | 151 | |
| 210 | | if you want to zip up all text (.txt) files in a directory, just pass in: |
| 211 | | the directory as a `std::string`, the pattern, i.e. ".txt", `&yourZipList` |
| 212 | | |
| 213 | | The last two flags are the sort order of the file list (CPDN files need to be |
| 214 | | in a certain order -- descending filenames, which is why that's the default). |
| 215 | | The default is to "clear" your list, you can set that to `false` to keep adding |
| 216 | | files to your `ZipFileList`. |
| 217 | | |
| 218 | | When you have created your `ZipFileList` just pass that pointer to `boinc_zip`. |
| 219 | | You will be able to add files in other directories this way. |
| 220 | | |
| 221 | | There is a `ziptest` Project for Windows provided to experiment, which can |
| 222 | | also be run (the "ziptest.cpp") on Unix & Mac to experiment |
| 223 | | with how `boinc_zip` work (just g++ with the `boinc/lib/filesys.C` & `util.C` as |
| 224 | | described above). |
| | 161 | if you want to zip up all text (.txt) files in a directory, just pass in: the directory as a `std::string`, the pattern, i.e. ".txt", `&yourZipList` |
| | 162 | |
| | 163 | The last two flags are the sort order of the file list (CPDN files need to be in a certain order -- descending filenames, which is why that's the default). The default is to "clear" your list, you can set that to `false` to keep adding files to your `ZipFileList`. |
| | 164 | |
| | 165 | When you have created your `ZipFileList` just pass that pointer to `boinc_zip`. You will be able to add files in other directories this way. |
| | 166 | |
| | 167 | There is a `ziptest` Project for Windows provided to experiment, which can also be run (the "ziptest.cpp") on Unix & Mac to experiment with how `boinc_zip` work (just g++ with the `boinc/lib/filesys.C` & `util.C` as described above). |
| 255 | | |
| 256 | | These basic routines may be useful if you want to compress/decompress a file |
| 257 | | using the zlib library (usually called "libz.a" and available for most platforms). |
| 258 | | Include the header file below (qcn_gzip.h) in your program, and link against libz, |
| 259 | | and you will gain two simple to use functions for gzip'ing or gunzip'ing a file. |
| 260 | | This is for simple single file or file-by-file compression or decompression |
| 261 | | (i.e. one file that is to be compressed into a .gz or decompressed back to |
| 262 | | it's original uncompressed state). |
| 263 | | You can check for boinc client status if you want the ability to quit |
| 264 | | inside an operation etc. |
| | 189 | These basic routines may be useful if you want to compress/decompress a file using the zlib library (usually called "libz.a" and available for most platforms). Include the header file below (qcn_gzip.h) in your program, and link against libz, and you will gain two simple to use functions for gzip'ing or gunzip'ing a file. This is for simple single file or file-by-file compression or decompression (i.e. one file that is to be compressed into a .gz or decompressed back to it's original uncompressed state). You can check for boinc client status if you want the ability to quit inside an operation etc. |
