large amount of tasks = boinc and boincmgr hogging all CPU time

[Aaron Finney]

For some reason, the more tasks you have in your queue, the more CPU time that BOINC.exe and boincmgr.exe are needing, I'm assuming to update the grid in boinc manager. At some point, the CPU time required to parse the list of tasks in the queue reaches a 'critical mass' and boinc.exe and boincmgr.exe just use the CPU time up to 100% updating themselves. No scientific work can get through, because boinc.exe and boincmgr.exe have higher priority levels.

[Aaron Finney]

boinc taking up 50% of resource share

[Aaron Finney]

Boinc manager with list of large tasks taking up 50% of CPU time.

[Aaron Finney]

Task manager after removing 90% of tasks.

[Aaron Finney]

This also can cause BOINC and the BOINC manager to become completely unresponsive.

[Nicolas]

Well, that's a huge list of tasks! It would need some profiling to know what piece of code is being the bottleneck.

[Nicolas]

An idea:

• Add a GUI RPC to get details of a single result (or better: a specific list of results).
• Make the manager update every second only the results in "Running" state, and refresh the whole list using get_results on longer periods, like every 10 seconds.

If a new manager connects to an old client not supporting the RPC, it should notice the RPC is missing and avoid it. After connection, the first time the manager refreshes the result list, try the new RPC. If it fails with "RPC unrecognized", fallback to the current method of getting all results every second, and don't retry the new RPC during the rest of that connection.

[Nicolas]

An addition to my previous suggestion: because of the way the <get_results> tag is parsed, we don't really need a new RPC. Just more parameters to the existing RPC. It would be even better from a backwards compatibility standpoint. Sending an RPC like the following:

<boinc_gui_rpc_request>
<get_results>
<filter>scheduler_state eq 2</filter>
</get_results>
</boinc_gui_rpc_request>

currently sends back the whole list of results without any error. A future version of the client could actually parse that tag to send only results that match the expression, and a future manager could take advantage of it, without any compat. problem. If the manager sends a filter, but gets the full list anyway, no problem, just update whole list on the GUI as it does now.

Probably such a generic <filter> expression is too complex to implement. That's just an example; any filter syntax would work.

[Der Meister]

I just did some profiling of the manager. Unfortunately I don't have access to a client with more than 5 results at the moment, therefore I did not profile in the same situation described here.

However, my first results show that a lot of CPU time is used for memory management and string manipulation. Most of this seems to come from internal wxWidgets functions. A particular function of the manager than seems to be expensive is RESULT::parse in gui_rpc_client_ops.C. This would be consistent with the observation described in this ticket. Unfortunately I don't see any easy way to optimise this function, except Nicolas'. But this seems to be a quite big change to the manager and the core client.

I think I will run another profiling session as soon as I am able to persuade my core client to fetch a larger amount of work.

[Der Meister]

Another solution came to my mind: What about turning the static refresh frequency into a dynamic one? The delay between two refresh-cycles could be proportional to the number of results (or more abstract: proportional to the number of elements in the current view). That should help to keep the CPU usage of the core client and the manager on a constant low level.

If there are no objections against this proposal I would like to write a patch for it.

[Der Meister]

Patch implementing the dynamic refresh rate

[Didactylos]

This is a blocker originally for 5.10! There is a patch provided.

Resetting milestone to 6.2. Major bugs should not be version-bumped without explanation.

[romw]

Bumping to 6.6

[romw]

Moving this to 6.10.

The patch cannot be applied as we are now using the Async RPC mechinism.

The decided solution to the problem is too spin up a new thread to monitor the async thread and throttle it back so that it cannot use more than 5% of the CPU. The async thread may need to be broken up into two different threads so that on-demand RPCs are not subject to being throttled.

[davea]

No more threads.

1) show only active tasks; button to show all

2) reduce refresh rate if CPU usage is above threshold

[Nicolas]

So it wasn't much of a "decided solution"...

[Nicolas]

If you plan to add a new RPC to get only active tasks, remember what I mentioned before in this ticket: adding tags inside <get_results> is fully backwards compatible. Older clients will return the full result list with no problems. So don't make a new RPC, use something like:

<get_results><only_active/></get_results>

[romw]

This issue should be resolved in the latest beta releases.