Requesting 7zip support in BOINC

[Ageless]

For Hydrogen@Home we're busy trying out several ways to get 7zip archives to decompress through BOINC. So far without success.

Asking through email, Rom answered we'd best make use of the wrapper application. That's all fine, but doesn't the wrapper decompress natively to a slot? And then when the wrapper app ends, doesn't BOINC delete the contents of the slot? We're trying to keep the decompressed files in the project folder.

I managed to get an answer from Carl Christensen (he's still alive :-)).
self-extracting execs across platforms isn't possible with the way BOINC is setup. If you like the 7zip algorithm (which I've never heard of) I suggest making a "boinc_7zip" library similar to how I did 'boinc_zip' -- I'm sure David Anderson would welcome this addition to the BOINC CVS tree. My other suggestion is to use the built-in web compression (I think it's mod_deflate in apache) to do "transparent" gzip compression (the details are on the wiki -- basically when I redid the BOINC networking layer two years ago using libcurl this featre was made possible). I'm afraid I am all tied up to do any BOINC programming now.

So the question is, can someone make a boinc_7zip library for integration into BOINC? Would other projects be interested in this?

If you're unfamiliar with 7zip's capabilities, read ​its site or ​Wikipedia on 7zip. It does compress better than zip or gzip.

[Nicolas]

Something like boinc_7zip would also decompress to the slot... I'm not sure how things work if you add files to the project directory without "letting BOINC know" the file is there.

[Nicolas]

ballen? Whoops.

[Ageless]

Replying to Nicolas:

Something like boinc_7zip would also decompress to the slot... I'm not sure how things work if you add files to the project directory without "letting BOINC know" the file is there.

It's not tasks we're trying to zip up, just static library files and probably the executables. The difference between users having to download 49MB worth of stuff or 5MB worth of archive is huge.

But does boinc_zip decompress to a slot? I thought CPDN used it to decompress to their project folder, then copied to the slot. I asked for a boinc_7zip, but perhaps that the present boinc_zip can be adapted so 7zip is also supported. Although the neatest would be if it could be done through Apache, like mod_deflate. Oh wishes.

[Ageless]

Hopefully no commas are needed between CC's?

[Nicolas]

Replying to Ageless:

It's not tasks we're trying to zip up, just static library files and probably the executables. The difference between users having to download 49MB worth of stuff or 5MB worth of archive is huge.

I don't think CPDN compresses the apps.

But does boinc_zip decompress to a slot? I thought CPDN used it to decompress to their project folder, then copied to the slot. I asked for a boinc_7zip, but perhaps that the present boinc_zip can be adapted so 7zip is also supported. Although the neatest would be if it could be done through Apache, like mod_deflate. Oh wishes.

Remember that technically you can decompress anywhere on the user's hard disk.

Is it so much of a problem to bundle 7z.exe with the Windows application, p7zip with the Linux app, etc. and use that to decompress? The "application" would consist of the extracting tool (7z or p7zip), the .7z archive itself, and a tiny wrapper that would start the extracting tool to decompress to slot and run the extracted main app. BURP does that, but with a .zip archive (bundling unzip.exe).

[jackygrahamez]

Here is my failed one-eyed attempt to work arround the problem with a dirty solution. I used different decompressions for different platforms and I used a batch file in the case of windows to decompress to project directory if it was not decomressed already, then copy to slot. But that caused errors about unable to handle processes.

[Ananas]

7-Zip is freesource and not only an executable binary. They have an SDK here : ​http://www.7-zip.org/sdk.html so there's no need for a separate .exe file as soon as the format is recognized by the core client.

On some file types (e.g. EPS) it is up to 2 times more efficient than any other compressor, usually it's only a few % though.

[Nicolas]

Replying to Ananas:

7-Zip is freesource and not only an executable binary. They have an SDK here : ​http://www.7-zip.org/sdk.html so there's no need for a separate .exe file as soon as the format is recognized by the core client.

Adding support on the core client is a bad idea. Means all previous versions that don't support it won't be able to run that project. It's easy enough to make the science app extract the files on startup.

[Ananas]

Replying to Nicolas:

... Means all previous versions that don't support it won't be able to run that project ...

Not necessarily. If the scheduler knows (by the reported version) that the client supports 7z it can send 7z compressed files in 1st place.

[Nicolas]

Replying to Ananas:

Replying to Nicolas:

... Means all previous versions that don't support it won't be able to run that project ...

Not necessarily. If the scheduler knows (by the reported version) that the client supports 7z it can send 7z compressed files in 1st place.

And any older version gets bigger-than-acceptable files. Instead, if the science app does the extraction, any BOINC version would be able to work fine with 7zip, since it's not the core client the one extracting anything, it's the science app, project-provided.

Also, making the scheduler send a different file depending on the version would mean quite a bit of hacking on the scheduler code.

[Ageless]

Replying to Nicolas:

Replying to Ananas:

Replying to Nicolas:

... Means all previous versions that don't support it won't be able to run that project ...

Not necessarily. If the scheduler knows (by the reported version) that the client supports 7z it can send 7z compressed files in 1st place.

Also, making the scheduler send a different file depending on the version would mean quite a bit of hacking on the scheduler code.

It would also mean that projects need to have two zips of each file available, going against what I asked for in the first place.

(I see we're still without assignment. No one wanting to take it? ;))

[davea]

7zip compression/decompression will have to be done in the application. You can extract files into the project directory if you want.

[Nicolas]

avoiding skew of ticket-fixing stats for 6.8