Changes between Version 1 and Version 2 of FileCompression
- Timestamp:
- May 21, 2007, 3:10:17 PM (18 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
FileCompression
v1 v2 3 3 == Compression of output files == 4 4 5 If you include the <gzip_when_done> tag in an [http://boinc.berkeley.edu/xml.php#fileoutput file description], the file will be gzip-compressed after it has been generated.5 If you include the `<gzip_when_done>` tag in an [XmlFormat#Files output file description], the file will be gzip-compressed after it has been generated. 6 6 7 7 == Compression of input files == 8 8 9 Starting with version 5.4, the BOINC client is able to handle HTTP Content-Encodingtypes 'deflate' (zlib algorithm) and 'gzip' (gzip algorithm). The client decompresses these files 'on the fly' and stores them on disk in uncompressed form.9 Starting with version 5.4, the BOINC client is able to handle HTTP `Content-Encoding` types 'deflate' (zlib algorithm) and 'gzip' (gzip algorithm). The client decompresses these files 'on the fly' and stores them on disk in uncompressed form. 10 10 11 11 You can use this in two ways: 12 12 13 13 * Use the Apache 2.0 mod_deflate module to automatically compress files on the fly. This method will work with all BOINC clients, but it will do compression only for 5.4+ clients. Info on how to configure this is below. 14 * Compress files and give them a filename suffix such as '.gz'. The name used in your <file_info>elements, however, is the original filename without '.gz'.14 * Compress files and give them a filename suffix such as '.gz'. The name used in your `<file_info>` elements, however, is the original filename without '.gz'. 15 15 16 Include the following line in httpd.conf:16 Include the following line in `httpd.conf`: 17 17 18 18 {{{ … … 21 21 22 22 23 23 This will add the content encoding to the header so that the client will decompress the file automatically. This method has the advantage of reducing server disk usage and server CPU load, but it will only work with 5.4+ clients. Use the 'min_core_version' field of the app_version table to enforce this. You can use this in conjunction because the mod_deflate module allows you to exempt certain filetypes from on-the-fly compression. 24 24 25 25 Both methods store files uncompressed on the client. If you need compression on the client, you must do it at the application level. The BOINC source distribution includes a version of the zip library designed for use by BOINC applications on any platform (see below). … … 52 52 You need to read the Apache 2.0 documentation about this 53 53 module to make sure you understand it. 54 However, our httpd.conffile for these changes includes the following:54 However, our `httpd.conf` file for these changes includes the following: 55 55 {{{ 56 56 # Enable module … … 78 78 79 79 This configuration tells Apache to compress all files served from 80 the download direction except for files that end with gz,gif,jpg, 81 jpeg and png. 80 the download direction except for files that end with `gz`,`gif`,`jpg`,`jpeg` and `png`. 82 81 An alternate way to specify the files is the following: 83 82 {{{ … … 88 87 </Directory> 89 88 }}} 90 This configuration tells apache to compress only the file types91 .faa and .mask served from the download director.89 This configuration tells Apache to compress only the file types 90 `.faa` and `.mask` served from the download directory. 92 91 93 92 == Using boinc_zip == … … 104 103 against to provide basic zip/unzip compression functionality. It 105 104 should only add a few hundred KB to your app (basically like 106 distributing zip & unzip exe's for different platforms).105 distributing `zip` & `unzip` executable binaries for different platforms). 107 106 108 Limitations: the "unzip" functionality is there, that is you can unzip 107 === Limitations === 108 The "unzip" functionality is there, that is you can unzip 109 109 a file and it will create all directories & files in the zip file. 110 110 The "zip" functionality has some limitations due to the cross-platform … … 113 113 function which will be explained below. 114 114 115 Building: For Windows, you can just add the project "boinc_zip" to your 115 === Building === 116 117 For Windows, you can just add the project "boinc_zip" to your 116 118 Visual Studio "Solution" or "Workspace." Basically just "Insert Existing 117 119 Project" from the Visual Studio IDE, navigate over to the boinc/zip … … 126 128 to build properly for your platform. 127 129 128 Also, please note that boinc_zip relies on some BOINC functions that you 129 will need (and will most likely be in your app already since they are handy) 130 -- namely boinc/lib/filesys.C and boinc/lib/util.C 130 Also, please note that boinc_zip relies on some BOINC functions that you will need (and will most likely be in your app already since they are handy) -- namely `boinc/lib/filesys.C` and `boinc/lib/util.C`. 131 131 132 Using: 133 Basically, you will need to #include "boinc_zip.h"in your app (of course132 === Using === 133 Basically, you will need to `#include "boinc_zip.h"` in your app (of course 134 134 your compiler will need to know where it is, i.e. -I../boinc/zip). 135 135 136 Then you can just call the function "boinc_zip"with the appropriate arguments137 to zip or unzip. There are three over riddenboinc_zip's provided:136 Then you can just call the function `boinc_zip` with the appropriate arguments 137 to zip or unzip. There are three overloaded boinc_zip's provided: 138 138 {{{ 139 139 int boinc_zip(int bZipType, const std::string szFileZip, … … 143 143 int boinc_zip(int bZipType, const char* szFileZip, const char* szFileIn); 144 144 }}} 145 bZipType is ZIP_IT or UNZIP_IT(self-explanatory)145 `bZipType` is `ZIP_IT` or `UNZIP_IT` (self-explanatory) 146 146 147 szFileZipis the name of the zip file to create or extract147 `szFileZip` is the name of the zip file to create or extract 148 148 (I assume the user will provide it with the .zip extension) 149 149 150 150 The main differences are in the file parameter. The zip library used was 151 151 exhibiting odd behavior when "coexisting" with unzip, particularly in the 152 wildcard handling. So a function was made that creates a "ZipFileList"class,152 wildcard handling. So a function was made that creates a `ZipFileList` class, 153 153 which is basically a vector of filenames. If you are just compressing a 154 single file, you can use either the std::string or const char* szFileInoverrides.154 single file, you can use either the `std::string` or `const char* szFileIn` overrides. 155 155 156 You can also just pass in a "*" or a "*.*"to zip up all files in a directory.156 You can also just pass in a `*` or a `*.*` to zip up all files in a directory. 157 157 158 To zip multiple files in a "mix & match" fashion, you can use the boinc_filelist158 To zip multiple files in a "mix & match" fashion, you can use the `boinc_filelist` 159 159 function provided. Basically, it's a crude pattern matching of files in a 160 160 directory, but it has been useful for us on the CPDN project. Just create a 161 ZipFileList instance, and then pass this into boinc_filelistas follows:161 `ZipFileList` instance, and then pass this into `boinc_filelist` as follows: 162 162 {{{ 163 163 bool boinc_filelist(const std::string directory, … … 168 168 }}} 169 169 if you want to zip up all text (.txt) files in a directory, just pass in: 170 the directory as a std::string, the pattern, i.e. ".txt", &yourZipList170 the directory as a `std::string`, the pattern, i.e. ".txt", `&yourZipList` 171 171 172 172 The last two flags are the sort order of the file list (CPDN files need to be 173 173 in a certain order -- descending filenames, which is why that's the default). 174 The default is to "clear" your list, you can set that to "false"to keep adding175 files to your "ZipFileList".174 The default is to "clear" your list, you can set that to `false` to keep adding 175 files to your `ZipFileList`. 176 176 177 When you have created your "ZipFileList" just pass that pointer to boinc_zip.177 When you have created your `ZipFileList` just pass that pointer to `boinc_zip`. 178 178 You will be able to add files in other directories this way. 179 179 180 There is a "ziptest"Project for Windows provided to experiment, which can180 There is a `ziptest` Project for Windows provided to experiment, which can 181 181 also be run (the "ziptest.cpp") on Unix & Mac to experiment 182 with how boinc_zip work (just g++ with the boinc/lib/filesys.C & util.Cas182 with how `boinc_zip` work (just g++ with the `boinc/lib/filesys.C` & `util.C` as 183 183 described above).
