Download/Upload KBytes/sec Preferences not working as expected - Burst behavior

[JacobKlein]

When testing Download and Upload preferences, on my Windows 7 x64 machine, it does not look like speeds are being limited correctly. Windows Task Manager shows limiting, but the speeds burst above the limit, then go to 0, repeatedly.

If I limit to 40 KB/s, I'd expect a smooth and steady 40 KB/s, not 0 for a few seconds, then 85 briefly, then to 0, then to 75, etc. Yet, that's the behavior I'm observing in Task Manager, with an update speed of 1 second.

Below are some emails where I originally reported this on the Alpha Mailing list.

===================
Jacob Klein jacob_w_klein at msn.com
Wed Mar 16 !22:16:14 PDT 2011

I've (finally) finished my 6.12.18 alpha preferences testing.
I reported the results in the test results, but figured I'd mention the major bug here also.

Essentially, whenever I have a KBytes/sec limit, the download rate sometimes just stops completely, and sometimes starts/stops.
It certainly isn't smooth like I would expect.
My internet normally downloads at ~330KB/sec, and my testing was a setting of 40 KBytes/sec as the limit.
I was resetting projects to get transfers to kick in.

My PC has 2 network adapters as seen in Task Manager.
- "Ethernet 2" is what is used for Internet and BOINC.
- "Bluetooth Network Connection" is never used.
... but I get the same results when I disable the Bluetooth and restart BOINC, so I don't think that adapter is interfering.

It's kind of weird, because, even though the GUI shows 4 downloads each of 10 KBytes/sec..
Windows Task Manager (which shows the actual data, and is updating every second), shows sporadic jumps above 40, and then 0, and isn't smooth.
It's almost as if the GUI "Speed" calculation is wrong, but BOINC uses that wrong calculation to limit the speed.
It could even be a timing problem or race condition, not sure.

Can anyone replicate the odd behavior?

Other smaller preference bugs that I found:
- "Confirm before connecting to internet" did not do anything for me (on Ethernet), but tool tip does not say dial-up only.
- If Network is suspended because of "Suspend Network" activity, it should say "Suspended - user request", even if a time-of-day preference is set.
- Day-of-week override does not have tool tips on day hover or check box hover

Several BOINC Manager inputs needs better sanitization
- "Idle minutes" preference input should not allow negative numbers
- "Hours" inputs should not allows entries like !13:239
- "% of the processors" should not allow negative numbers or numbers >100
- "% CPU Time" should not allow negative numbers or numbers >100
- "While processor usage is less than" should not allow negative numbers or numbers >100
- "Maximum download rate" should not allow negative numbers
- "Maximum upload rate" should not allow negative numbers

A few web-based inputs need better sanitization
- "Maximum download rate" should not allow negative numbers
- "Maximum upload rate" should not allow negative numbers

Kind regards,
Jacob Klein

===================
Rom Walton romw at romwnet.org
Thu Mar 17 !11:44:53 PDT 2011

I'll take a look this weekend.

Our app only knows what it sends out over the outgoing socket and what
is read from the incoming socket, things like TCP/IP overhead are not
included in the BOINC calculations.

Resource Manager from the Task Manager is displaying overall throughput
including TCP/IP's overhead. Resource Manager also includes the sum
total of all traffic across all processes.

As far as the input validation in the Manager goes, I'll get that
cleaned up for 7.0. I'm not going to hold up the 6.12 release for it.

----- Rom

===================
Jacob Klein jacob_w_klein at msn.com
Tue Mar 22 !06:00:10 PDT 2011

Did your investigation find anything?

===================
Rom Walton romw at romwnet.org
Tue Mar 22 !08:38:09 PDT 2011

BOINC believes it is only sending out what it is told to. The odd stuff
is happening at a lower level of the software stack.

Resource Manager is including the TCP/IP overhead while what BOINC
reports does not. I don't know if the burst type behavior is Windows or
Libcurl. If I had to take a guess I would say Windows. Without knowing
how your network is configured, Windows does try to make use of things
like Jumbo Frames and TCP checksum offloading to increase network
throughput.

At this point I'm happy as long as BOINC doesn't saturate the WAN
(Internet) connection.

----- Rom

[davea]

BOINC doesn't enforce bandwidth limits itself; it relies on the HTTP library (libCURL) to do so. I guess libCURL does this in a coarse-grained way, e.g. by turning transmission on and off at 1-second scale. As long as the average is enforced correctly on the 10+ sec time scale, I don't think it's worth filing a bug to libCURL about this.

[JacobKlein]

Can we upgrade to the latest version of libCurl?  From the best I can tell, we last dropped 7.19.7, but since then, there have been 2 years worth of bugfixes (including ones related to speeds/rates/timers), including 2 vulnerability plugs.

Could we upgrade to 7.21.4 or better, to see if it fixes the problems in this ticket?

[Nicolas]

BOINC on Windows is dynamically linked to libcurl; so in theory you could replace your C:\Program Files\BOINC\libcurl.dll with a newer one, and test yourself if it fixes the problem.

However, it seems the curl website doesn't have Windows 64-bit downloads for libcurl, only 32-bit... :/

[JacobKlein]

Even though I have a 64-bit machine, I did some testing with the 32-bit install of BOINC.

I compared the stock BOINC install (which uses libcurl 7.19.7) against running BOINC with the newest libcurl dll files.

It looks like the 7.21.4 versions of the libcurl dll files do not really help to alleviate this particular problem.  I do, however, think it is worth upgrading to the latest version, especially because of the vulnerabilities that have been patched.

[JacobKlein]

And I did some further research.  I found a man page about CURL located here: ​http://curl.haxx.se/docs/manpage.html ... and, although there are a bunch of command line options, I'm pretty sure the dll calls are equivalent in design.

My point is that, for the "--limit-rate <speed>" command line switch, the man page states: "The given rate is the average speed counted during the entire transfer. It means that curl might use higher transfer speeds in short bursts, but over time it uses no more than the given rate."

So, I suppose that this ticket is a limitation with libcurl, and I'll remark it as wontfix.  But I still think we should consider upgrading our libcurl libraries, and possibly try to keep them up to date.