Large numbers of file transfers freeze the client

[JKeck]

If the client has large numbers of pending file transfers, greater than 1000 to 2000, the client becomes unresponsive and boinc.exe will use all available CPU cycles. On a single CPU host this makes the computer completely unusable, dual and hyperthreaded hosts will still be usable but little or no BOINC work will be done.

quote from Krunchin-Keith:

BOINC manager was usually un-responsive. In windowsXP any time I clicked on anything in manager it said (Not responding). BOINCview also would not communicate with host via RPC's. I did get one host sort of working and it took me 1/2 hour to click-wait-click-wait-click-wait to be able to suspend 1 project at a time, only about 16 projects. Once I got all suspended I only ran SciLINK with no new work to clear the backlog. They had I think 20 files per download (not sure on this number). and 50 upload files. All were small and the processing of a task took 0-5 seconds. So a request for work, usually returned 100-200 tasks, once 20 files were downloaded, 2 at a time, then within 5 seconds you now have 50 upload files, and the next set of 20 has not completed download. This spirals uphill quickly. 5600 files was less than 9M on disc. Small files, less than 1k each, but the overhead for each transfer slowed things down. To clear the backlog I found it best to set max transfers and max transfers per project to 8/8 or 16/16, not run any work and only run the core_client. running the manager made things worse. Once I shut down manager I was able to use BOINCview to control and monitor the core_client. It took some wheres about 6 hours to clear all those files. boinc,exe was running pretty much at 50% CPU usage (max on a HT cpu for 1 of the 2 processes) Once the transfers got to less than about 1600 I was able to slowly restart running other projects. Anything above about 2000 transfers is hard for it to handle and sluggish or un-responsive to RPC commands.

[Nicolas]

Duplicate of #282. However, the description here seems to be more complete.

[MikeMarsUK]

See also Trac/113, and Trac/171 which address the Boinc manager's half of the problem (the other half being the behaviour of the daemon with excess transfers in Trac/282 mentioned above).