VirtualBox looks certainly promising.

Logistic advantages

1) One order of magnitude lighter, both its installation package (~35 MB) and its installed size (~60 MB). Compare with the 500+ MB of VMWare Server 2.0, that increase in some 150 extra MB when installed.

2) License. Its OSE (Open Source Edition) is published under the GPL v.2, but even the non-libre version -PUEL, Personal Use and Evaluation License, ​http://www.virtualbox.org/wiki/VirtualBox_PUEL- could be used for our purposes, but that's something to be checked by someone who actually knows something about licensing, unlike myself.

3) Faster and "less painful" installation process, partly due to its lighter weight. No license number required, hence less hassle for the user.

Technical points

The interaction with the VM is made possible even from the command line, in particular from the single command !VBoxManage (extensive doc available at ​http://download.virtualbox.org/virtualbox/2.1.4/UserManual.pdf ). Of particular interest for us are the following VBoxManager's arguments:

• startvm
• controlvm pause|resume|reset|poweroff|savestate ...
• snapshot
• vmstatistics
• createvm
• registervm

All the functionalities exposed by this command are also available throughout a C++ COM/XPCOM based API, as well as Python bindings. More on this later.

Following the capabilities enumeration introduced by Kevin, VirtualBox would compare to his analysis based on VMWare Server as follows:

1) Manage the Image. Covered by the "snapshot" command

2) Boot the virtual machine. Covered by "startvm"

3) Copy files host -> guest *Not* directly supported by the VirtualBox API. We'd need to resource to external solutions such as a properly configured SSH server on the Appliance.

4) Run a program on the guest. Same as 3)

5) Pause and unpause the guest. Covered by "controlvm pause/resume"

6) Retrieve files from the guest. See 3) and 4), same situation.

7) Shutdown the guest Covered by "controlvm poweroff"

Tackling the interaction with the appliance

A straightforward solution could be the configure the appliance to have a running ssh server, setup for public-key authentication such that communication with the host system is seamless. Moreover, this approach is complementary to whatever interaction support there might already be, such as the one provided by VIX: shell access to the appliance could provide us with certain information that would be impossible or just inconvenient to get ahold otherwise. Anything having to do with the running environment (ulimits, environment variables, etc) come to mind.

Bindings

Both VMWare Server and VirtualBox make available C/C++ APIs, as well as Python, with different levels of support -in case of VMWare, it's an unsupported project-. VirtualBox's API is based on COM/XPCOM, and it's possible to implement a unified windows/linux approach based on the former technology. The actual code implementing the VBoxManage command is a very good reference ( ​http://www.virtualbox.org/browser/trunk/src/VBox/Frontends/VBoxManage ) Therefore, implementing a "hypervisor abstraction layer" is in principle feasible, with a common win/linux codebase both for VIX and VirtualBox API. I'll be providing code snippets towards this goal in the following days. Interestingly enough, a working wrapper prototype could be implemented without much effort by taking advantage of the aforementioned VBoxManage command. That of course is somewhat "hackish", but nevertheless a convenient tool to have.

Interacting with the VM Appliance

Another very nice feature of VirtualBox is the possibility to interact with the running appliance through a Remote Desktop connection, which can be properly secured both in term of authentication and encrypted traffic (that is to say, these features are already supported by VirtualBox).

Conclussions

VirtualBox provides several appealing features, as powerful as those provided by VMWare at a lower cost -both in terms of inconveniences for the user and licensing-. However, it lacks support for direct interacting with the guest appliance: there are no equivalents to VIX's CopyFileFromGuestToHost?, RunProgramInGuest?, etc. related to the seven points summarizing the requirements. This inconvenience can nevertheless be addressed as mentioned with certain additional benefits and no apparent drawbacks.

To-Do

• Compare performance
• Implement a working prototype based on the VirtualBox API
• ... such that it works with no of minimal code changes both in windows and linux
• Demostrate it with a custom appliance implementing the ssh based communication mechanism

Overcoming VirtualBox API Limitations

Introduction

In previous sections, two limitations of the API offered by VirtualBox where pointed out. Namely, the inability to directly support the execution of command and file copying between the host and the guest. While relatively straightforward solutions exist, notably the usage of SSH, they raise issues of their own: the guest needs to (properly) configure this SSH server. For this to be effective, we obviously need the guest to be accessible "from the outside world". This is not always the case, most notably if the guest's networking is based on NAT, as it usually -and conveniently- is.

Thus, the requirements for a satisfactory solution would include:

• Minimal or no configuration required on the guest side.
• No assumptions on the network reachability of the guest. That is to say, guest should always act as a client.

Additional features to keep in mind:

• Scalability. The solution should account for the execution of an arbitrary number of guests on a given host.
• Technology agnostic: dependencies on any platform/programming language/hypervisor should be kept to a minimum or avoided altogether.

Proposed Solution

Following Predrag Buncic's advice, I began looking into such a solution based on asynchronous message passing. In order to keep the footprint, both on the host and the guest sides, the ​STOMP protocol came to mind. The protocol is simple enough as to have implementations in a large number of programming languages, while fulfilling all flexibility needs. Despite its simplicity and being relatively unheard of, ActiveMQ supports it out-of-the-box (even though it'd be advisable to use something lighter for a broker).

Focusing on the problem at hand, we need to tackle the following problems:

• Command execution on the guest
• File transfer from the host to the guest
• File transfer from the guest to the host

The main reason for differentiating between the last two points has to do with the no-servers-at-the-guest-side restriction. The file transfer mechanism will not be "symmetric".

The following diagram depicts a bird's eye view of the system's architecture:

Command Execution

Requesting the execution of a program contained in the guest fit nicely into an async. message passing infrastructure: a tailored message addressed to the guest we want to run the command on is published, processed by this guest and eventually answered back with some sort of status (maybe even periodically in order to feedback about progress).

Given the subscription-based nature of the system, several guests can be addressed at once by a single host, triggering the execution of commands (or any other action covered by this mechanism) in a single go. Note that neither the hosts nor the (arbitrary number of) guests need to know how many of the latter conform the system: new guest instances need only subscribe to these "broadcasted" messages on their own to become part of the overall system. This contributes to the *scalability* of the system.

File Transfers

This is a trickier feature: transfers must be bidirectional, yet the guests can only act as clients, with -a priori- no knowledge of the hosts' location (remember that the only link between host and guest(s) is the broker). The proposed solution consists of a simple HTTP server on the host side. The reasons to consider HTTP are twofold:

• Bypassing firewalls. Even if this wouldn't usually be an issue -host and guests running on the same machine-, HTTP is the less likely protocol to be filtered out by firewalls.
• Simplicity. It's possible to implement -or reuse- such a server with a small footprint. Likewise for the guests' client side.

The only requirement on this HTTP server is that is must accept PUT requests, in order for the guests to upload files.

To enable the guests to access this HTTP server, its host and port must be handed to them. This is accomplished, how else, by message passing: because host and guests are all subscribed to a common broker on a common topic, such information can be published (ie, broadcasted) in a way very much similar to the one described in the previous point for commands. Once guests have taken notice of the file host (which does *not* have to be the same as the BOINC host), file transmission is handled in the same message-oriented way, once the guests are aware of the file host details.

The Devil in the Details

In the previous argumentations, it was implied that both host and guests were already connected to a common broker. This is clearly not the case upon startup. Elaborate mechanism might include the use of ​Zeroconf methods such as ​avahi or ​bonjour, but in principle and for the time being, it can be assumed that the broker will be accesible to the guest on the IP acting as its default gateway. In the common case of a NAT setup for the guests, this IP is configurable at the hypervisor level, corresponding to the BOINC host machine. Even if we wanted to balance the load, placing the broker somewhere else, an adequate port forwarding mechanism could be setup at the BOINC host side.

As a particularity of VirtualBox NAT system, there is no out-of-the-box connectivity between the hypervisor host and the VM (for details, see the VirtualBox Manual, section 6.4). Fortunately, VirtualBox includes a port forwarding mechanism that makes it possible to map ports on the VM to the hypervisor's host, circumventing this problem. On the other hand, there should be no need to forward any port from the VM, as it's supposed to perform only client-side operations. On the other hand, if other -unrelated- services, such as the CernVM's web-based administration mechanism, need be accessed, it's convenient to keep in mind this feature. Details are available at the aforementioned VirtualBox Manual, section 6.4.1.

Open Questions

• Where should the broker live? Conveniently on the same machine as the hypervisor or on a third host? Maybe even a centralized and widely known (ie, standard) one? This last option might face congestion problems, though.
• Broker choice. Full-fledged (​ActiveMQ) or more limited but lighter? (ie, ​Gozirra). On this question, unless a centralized broker is universally used, the lighter version largely suffices. Otherwise, given the high load expected, a more careful choice should be made.

Implementation

​CernVM has been taken as the base guest system. In order to run the developed prototype, the following packages need to be installed (by running conary update <package>):

• subversion
• gcc

The following python projects (in the following order)

have to be downloaded and installed using the provided setup.py:

• Zope interfaces (​http://www.zope.org/Products/ZopeInterface)
• twisted (​http://twistedmatrix.com)
• stomper (subversion repository @ ​http://stomper.googlecode.com/svn )

This prototype has been implemented in Python, given its cross-platform nature and the suitability of the tools/libraries it provides.

Overview

Upon initialization, guests connect to the broker, that's expected to listen on the default STOMP port 61613 at the guest's gateway IP. Once connected, it "shouts out" he's joined the party, providing a its unique id (see following section for details). Upon reception, the BOINC host notes down this unique id for further unicast communication (in principle, other guests don't need this information). The host acknowledges the new guest (using the STOMP-provided ack mechanisms).

Unique Identification of Guests

The preferred way to identify guests is based on ​UUID. Python includes a uuid module since version 2.5. Unfortunately, CernVM ships with version 2.4, so the uuid.py file define this module had beed to include "by hand".

Tailor-made STOMP Messages

For command execution:

    HEADERS:
      cmd: cmd-to-execute
      stdout: stdout-file
      stderr: stderr-file
      [shell: shell-to-use] (def: /bin/sh)
      cwd: cwd-to-use
      env: mapping-defining-exec-env

    BODY: 
      VM-RUN-CMD

For file transfers:

The negotiation for the direct connection between the file host and the guest(s) takes place using the following message:

Once the guests have this information, the following message format deals with the actual file transfer requests:

    HEADERS:
      host: file-host-address
      port: file-host-port
      path: path-to-files
      files: list-of-files

    BODY:
      UP[-TO-HOST] | DOWN[-TO-HOST]

Conclusions

(TODO)

Note that the described approach is equally valid for any hypervisor, rendering it as a candidate for an independent implementation for the aforementioned features of command execution and file transference.

Alternatives

XMPP?