Upload project backoff needs to take account of multiple servers

[Thyme Lawn]

CPDN has a number of upload servers (currently 9) which accept specific upload files from different application types. If one of the upload servers is down (currently the case) the project back off mechanism can result in no uploads being possible to the other servers. It also appears that once a host has a large number of pending uploads it stops requesting new work, which is perfectly reasonable when it is genuinely impossible to upload any of the files.

When an upload fails BOINC backs off on that file for a semi-random period of time. After up to 3 consecutive files have failed it also backs off on the project for a different semi-random period of time. The interaction of these timers ensures that as the number of files waiting for the problem server builds up you eventually hit a point when the files waiting to be uploaded to the other (functional) servers are never tried.

It is possible to circumvent this by selecting any pending file for the project on the Transfers tab and clicking the "Retry now" button. That starts a cycle of up to 3 file retries from the first file with no individual back off timer running, so the button might have to be clicked again after the project enters another back off to get down to a file which can be uploaded.

This would be resolved if the core client grouped the pending uploads for a project by destination server and stopped trying the pending files for a server after one failure if there are files queued for another server.

[Pepo]

Maybe the client could simply drop any notion of a Project, when it handles file uploads, and just take the files' file upload handler URLs (and similarly for downloads) into account?