Allow splitting of result zip files into smaller segments to allow for network upload restrictions

[simalam]

Due to the increasing size of some of the project files, especially the climateprediction.net Project, users behind proxy servers are unable to finalise the upload of completed projects as zip file is too large. This is seen as "HTTP errors" in the messages log as per below:

16/04/2009 11:34:21 AM climateprediction.net Started upload of hadam3p_mabq_1971_2_006045576_1_1.zip
16/04/2009 11:34:24 AM climateprediction.net Temporarily failed upload of hadam3p_mabq_1971_2_006045576_1_1.zip: HTTP error
16/04/2009 11:34:24 AM climateprediction.net Backing off 2 hr 22 min 36 sec on upload of hadam3p_mabq_1971_2_006045576_1_1.zip

What is happening is that this zip file is 25.21MB in size and the HTTP upload request (not sure if it is a HTTP PUT or POST) is blocked by a proxy server as it is larger than the proxy server max HTTP upload size. See my post in this ​thread at climateprediction.net for more info.

Could a network preference be created that splits the upload zip files into definable segments, e.g. per MB? Maybe similar to the trickle upload process?

This would allow users who are restricted in their internet usage, especially in a work or academic environment to continue to participate in large projects such as the climateprediction.net.

Cheers,
Sim Alam

[simalam]

If this could be applied to existing/pending transfers that would be even better.

[Ageless]

Adjusted title.

[simalam]

Any updates on this? It has been nearly 18 months since I logged it.

Cheers, Sim

[davea]

Please set <http_debug>, so I can see the HTTP error return. Maybe there's some way to detect that the request was too large

[simalam]

Ok, from the below log file the HTTP error return appears to be:

20-Sep-2010 14:11:06 [---] [http] [ID#29] Received header from server: HTTP/1.0 413 Request Entity Too Large

Extract of stdoutdae.txt with server names etc changed:

20-Sep-2010 14:11:03 [---] [http] HTTP_OP::libcurl_exec(): ca-bundle 'D:\Program Files\BOINC\ca-bundle.crt' 20-Sep-2010 14:11:03 [---] [http] HTTP_OP::libcurl_exec(): ca-bundle set 20-Sep-2010 14:11:03 [---] [proxy] HTTP_OP::no_proxy_for_url(): http://uploader.oerc.ox.ac.uk/cpdn_cgi/file_upload_handler 20-Sep-2010 14:11:03 [---] [proxy] returning false 20-Sep-2010 14:11:03 [---] [proxy]: setting up proxy my.proxy.com:8080 20-Sep-2010 14:11:03 [climateprediction.net] Started upload of hadam3p_ndh0_1968_2_006165950_2_1.zip 20-Sep-2010 14:11:04 [---] [http] [ID#29] Info: timeout on name lookup is not supported 20-Sep-2010 14:11:04 [---] [http] [ID#29] Info: About to connect() to proxy my.proxy.com port 8080 (#0) 20-Sep-2010 14:11:04 [---] [http] [ID#29] Info: Trying 172.26.254.49... 20-Sep-2010 14:11:04 [---] [http] [ID#29] Info: Connected to my.proxy.com (172.26.254.49) port 8080 (#0) 20-Sep-2010 14:11:04 [---] [http] [ID#29] Sent header to server: POST http://uploader.oerc.ox.ac.uk/cpdn_cgi/file_upload_handler HTTP/1.1 20-Sep-2010 14:11:04 [---] [http] [ID#29] Sent header to server: User-Agent: BOINC client (windows_x86_64 6.11.7) 20-Sep-2010 14:11:04 [---] [http] [ID#29] Sent header to server: Host: uploader.oerc.ox.ac.uk 20-Sep-2010 14:11:04 [---] [http] [ID#29] Sent header to server: Accept: */* 20-Sep-2010 14:11:04 [---] [http] [ID#29] Sent header to server: Accept-Encoding: deflate, gzip 20-Sep-2010 14:11:04 [---] [http] [ID#29] Sent header to server: Proxy-Connection: Keep-Alive 20-Sep-2010 14:11:04 [---] [http] [ID#29] Sent header to server: Content-Type: application/x-www-form-urlencoded 20-Sep-2010 14:11:04 [---] [http] [ID#29] Sent header to server: Content-Length: 290 20-Sep-2010 14:11:04 [---] [http] [ID#29] Sent header to server: 20-Sep-2010 14:11:05 [---] [http] [ID#29] Info: HTTP 1.0, assume close after body 20-Sep-2010 14:11:05 [---] [http] [ID#29] Received header from server: HTTP/1.0 200 OK 20-Sep-2010 14:11:05 [---] [http] [ID#29] Received header from server: Date: Mon, 20 Sep 2010 05:11:05 GMT 20-Sep-2010 14:11:05 [---] [http] [ID#29] Received header from server: Server: Apache/2.2.13 (Linux/SUSE) 20-Sep-2010 14:11:05 [---] [http] [ID#29] Received header from server: Content-Type: text/plain 20-Sep-2010 14:11:05 [---] [http] [ID#29] Received header from server: X-Cache: MISS from l2-13.proxy.com 20-Sep-2010 14:11:05 [---] [http] [ID#29] Received header from server: X-Cache-Lookup: MISS from l2-13.proxy.com:8079 20-Sep-2010 14:11:05 [---] [http] [ID#29] Received header from server: X-Cache: MISS from proxy02.my.proxy.com 20-Sep-2010 14:11:05 [---] [http] [ID#29] Received header from server: X-Cache-Lookup: MISS from proxy02.my.proxy.com:8080 20-Sep-2010 14:11:05 [---] [http] [ID#29] Received header from server: Via: 1.0 l2-13.proxy.com:8079 (squid/2.6.STABLE20), 1.0 proxy02.my.proxy.com:8080 (squid/2.6.STABLE20) 20-Sep-2010 14:11:05 [---] [http] [ID#29] Received header from server: Proxy-Connection: close 20-Sep-2010 14:11:05 [---] [http] [ID#29] Received header from server: 20-Sep-2010 14:11:05 [---] [http_xfer] [ID#29] HTTP: wrote 93 bytes 20-Sep-2010 14:11:05 [---] [http] [ID#29] Info: Expire cleared 20-Sep-2010 14:11:05 [---] [http] [ID#29] Info: Closing connection #0 20-Sep-2010 14:11:06 [---] [http] HTTP_OP::libcurl_exec(): ca-bundle set 20-Sep-2010 14:11:06 [---] [proxy] HTTP_OP::no_proxy_for_url(): http://uploader.oerc.ox.ac.uk/cpdn_cgi/file_upload_handler 20-Sep-2010 14:11:06 [---] [proxy] returning false 20-Sep-2010 14:11:06 [---] [proxy]: setting up proxy my.proxy.com:8080 20-Sep-2010 14:11:06 [---] [http] [ID#29] Info: timeout on name lookup is not supported 20-Sep-2010 14:11:06 [---] [http] [ID#29] Info: About to connect() to proxy my.proxy.com port 8080 (#0) 20-Sep-2010 14:11:06 [---] [http] [ID#29] Info: Trying 172.26.254.49... 20-Sep-2010 14:11:06 [---] [http] [ID#29] Info: Connected to my.proxy.com (172.26.254.49) port 8080 (#0) 20-Sep-2010 14:11:06 [---] [http] [ID#29] Sent header to server: POST http://uploader.oerc.ox.ac.uk/cpdn_cgi/file_upload_handler HTTP/1.1 20-Sep-2010 14:11:06 [---] [http] [ID#29] Sent header to server: User-Agent: BOINC client (windows_x86_64 6.11.7) 20-Sep-2010 14:11:06 [---] [http] [ID#29] Sent header to server: Host: uploader.oerc.ox.ac.uk 20-Sep-2010 14:11:06 [---] [http] [ID#29] Sent header to server: Accept: */* 20-Sep-2010 14:11:06 [---] [http] [ID#29] Sent header to server: Accept-Encoding: deflate, gzip 20-Sep-2010 14:11:06 [---] [http] [ID#29] Sent header to server: Proxy-Connection: Keep-Alive 20-Sep-2010 14:11:06 [---] [http] [ID#29] Sent header to server: Content-Type: application/x-www-form-urlencoded 20-Sep-2010 14:11:06 [---] [http] [ID#29] Sent header to server: Content-Length: 26416862 20-Sep-2010 14:11:06 [---] [http] [ID#29] Sent header to server: Expect: 100-continue 20-Sep-2010 14:11:06 [---] [http] [ID#29] Sent header to server: {{{20-Sep-2010 14:11:06 [---] [http] [ID#29] Info: Expire cleared 20-Sep-2010 14:11:06 [---] [http] [ID#29] Info: HTTP 1.0, assume close after body 20-Sep-2010 14:11:06 [---] [http] [ID#29] Received header from server: HTTP/1.0 413 Request Entity Too Large 20-Sep-2010 14:11:06 [---] [http] [ID#29] Received header from server: Server: squid/2.6.STABLE20 20-Sep-2010 14:11:06 [---] [http] [ID#29] Received header from server: Date: Mon, 20 Sep 2010 04:11:06 GMT 20-Sep-2010 14:11:06 [---] [http] [ID#29] Received header from server: Content-Type: text/html 20-Sep-2010 14:11:06 [---] [http] [ID#29] Received header from server: Content-Length: 1379 20-Sep-2010 14:11:06 [---] [http] [ID#29] Received header from server: Expires: Mon, 20 Sep 2010 04:11:06 GMT 20-Sep-2010 14:11:06 [---] [http] [ID#29] Received header from server: X-Squid-Error: ERR_TOO_BIG 0 20-Sep-2010 14:11:06 [---] [http] [ID#29] Received header from server: X-Cache: MISS from proxy02.my.proxy.com 20-Sep-2010 14:11:06 [---] [http] [ID#29] Received header from server: X-Cache-Lookup: NONE from proxy02.my.proxy.com:8080 20-Sep-2010 14:11:06 [---] [http] [ID#29] Received header from server: Via: 1.0 proxy02.my.proxy.com:8080 (squid/2.6.STABLE20) 20-Sep-2010 14:11:06 [---] [http] [ID#29] Received header from server: Proxy-Connection: close 20-Sep-2010 14:11:06 [---] [http] [ID#29] Received header from server: 20-Sep-2010 14:11:06 [---] [http_xfer] [ID#29] HTTP: wrote 1379 bytes 20-Sep-2010 14:11:06 [---] [http] [ID#29] Info: Expire cleared 20-Sep-2010 14:11:06 [---] [http] [ID#29] Info: Closing connection #0 20-Sep-2010 14:11:07 [climateprediction.net] Temporarily failed upload of hadam3p_ndh0_1968_2_006165950_2_1.zip: HTTP error 20-Sep-2010 14:11:07 [climateprediction.net] Backing off 9 hr 18 min 38 sec on upload of hadam3p_ndh0_1968_2_006165950_2_1.zip

[davea]

Apache has configuration parameters for max request size:

LimitXMLRequestBody 134217728 LimitRequestBody? 134217728

These apparently default to 1MB; the above lines set them to 128MB

[simalam]

Replying to davea:

Apache has configuration parameters for max request size:

LimitXMLRequestBody 134217728 LimitRequestBody? 134217728

These apparently default to 1MB; the above lines set them to 128MB

I don't understand how that helps.

The problem I have is that the BOINC client has to connect to the internet via a proxy server. It is the proxy server that is generating the HTTP 413 errors and I can't get the upload size limit increased on it.

[davea]

Yes, it won't help that. I'll keep this ticket open but won't have time to work on it for a while.