BOINC Workshop: BOF on VM applications
Kings College London, September 1, 2010
Attendees:
- Ben Segal (chair) <b.segal AT cern.ch>
- Chris Reynolds (secretary) <C.Reynolds AT westminster.ac.uk>
- Wenjing Wu <wuwj AT ihep.ac.cn>
- Uwe Beckert <yoyo AT mailueberfall.de>
- Diogo Ferreira <defer AT student.dei.uc.pt>
- Thiago Presa <thiago.addevico AT gmail.com>
- Yuting Chen <yuting.chen AT twgrid.org>
- Daniel Lombrana <teleyinex AT gmail.com>
Opening Comments
Ben referred to the work at CERN developing a "general solution" for BOINC apps using VM's. Consists of an extended BOINC Wrapper (VMWrapper) and a general-purpose VMController package which provides all needed host<->guest-VM functionality. All this code is written in Python for cross-platform support, but hit problems with packaging and installing the system cross-platform, especially on Windows. But this Python "general solution" is being completed, on a timescale of a few months.
References:
CERN is currently using a minimal-functionality VM-wrapper in C to prototype BOINC applications for LHC physics computing, with an approach requiring very few host<->guest functions. This approach profits from the use of the CernVM system, which not only provides optimal image size and update management, but also interfaces to existing LHC job scheduler systems, avoiding the use of BOINC scheduling (see CoPilot? reference).
References:
Question: Should we be backwards compatible with existing BOINC Wrapper system?
(CERN's Python VMWrapper is, but not the minimum functionality C VM-wrapper).
Answer: Agreed that in terms of functionality it would be preferable.
Question: How best to provide Guest<->Host File I/O?
Answer: Shared directories probably preferable in comparison to ssh.
Note: The CERN VMController system uses Chirp - see: http://boinc.berkeley.edu/trac/wiki/VirtualBox
Agreement: Everyone agreed that cross platform support is key.
Standardization Issues
How to standardize an API for interactions with hypervisors of different VM technologies?
- Diogo's work seems nice in this respect due to his focus on future proofing (via plugins).
ACTION: Diogo to forward paper on his work and to update the BOINC Wiki site "../wiki/VmApps" accordingly.
- Another alternative is to use LibVirt?, providing we can generate a Windows version...
- We could really do with talking to Libvirt developers, since they may well have faced similar problems to the ones we face. How much work to replace Linux specific functions of LibVirt? with Win32 calls?
Scientist Adoption
We discussed some factors which could help scientists adopt a BOINC-VM solution, either (1) as a simpler alternative to classic BOINC application porting, or (2) as a method of running their applications in "cloud mode", on BOINC as well as on other clouds. CERN is trying to enable both methods: (1) with the full-function VMWrapper/VMController code, and (2) via CernVM/CoPilot.
Questions discussed:
- Promote Rpath-Rbuilder for automating the VM image stripping/compacting? This is the CernVM approach.
- Method of image and data distribution: Bittorrent or the CernVM method (or both)? Image download and update efficiency is important.
- How do we provide "standard BOINC versions" of the Virtualization platform - hypervisors, VM image formats, etc?
Note that BOINC itself is installed with admin privileges, so we could theoretically install VirtualBox or other hypervisors at BOINC install time. (Preferably we attach to a VM specific project, and it notifies that a VM technology is required and optionally sets it up).
Note that such a solution requires some BOINC client updates. Automation could be problematic when we consider existing VirtualBox installs.
* David Anderson is now in communication with Oracle on the topic of bundling VirtualBox with BOINC.
