Changes between Version 3 and Version 4 of VirtualBox

Timestamp:

Apr 19, 2009, 3:38:30 PM (15 years ago)

Author:

dgquintas

Comment:

--

VirtualBox

@@ -107 +107 @@
  - Demostrate it with a custom appliance implementing the ssh based  communication mechanism
 
+
+= Overcoming VirtualBox API Limitations =
+
+== Introduction ==
+In previous sections, two limitations of the API offered by VirtualBox
+where pointed out. Namely, the inability to directly support the 
+execution of command and file copying between the host and the guest. 
+While relatively straightforward solutions exist, notably the usage of SSH,
+they raise issues of their own: the guest needs to (properly) configure this
+SSH server. For this to be effective, we obviously need the guest to
+be accessible "from the outside world". This is not always the case, most
+notably if the guest's networking is based on NAT, as it usually -and 
+conveniently- is. 
+
+Thus, the requirements for a satisfactory solution would include:
+
+  * Minimal or no configuration required on the guest side. 
+  * No assumptions on the network reachability of the guest. That is to say,
+    guest should always act as a client.
+
+Additional features to keep in mind:
+
+  * Scalability. The solution should account for the execution of an arbitrary
+    number of guests on a given host. 
+  * Technology agnostic: dependencies on any platform/programming
+    language/hypervisor should be kept to a minimum or avoided altogether.
+ 
+
+== Proposed Solution ==
+Following Predrag Buncic's advice, I began looking into such a solution based on
+asynchronous message passing. In order to keep the footprint, both on the host and the guest sides,
+the [http://stomp.codehaus.org/Protocol STOMP protocol] 
+came to mind. The protocol is simple enough as to have implementations in a
+large number of programming languages, while fulfilling all flexibility needs. Despite its 
+simplicity and being relatively unheard of, ActiveMQ supports it out-of-the-box (even though
+it'd be advisable to use something lighter for a broker).
+
+Focusing on the problem at hand, we need to tackle the following problems:
+
+  * Command execution on the guest
+  * File transfer from the host to the guest
+  * File transfer from the guest to the host
+
+The main reason for differentiating between the last two points has to do 
+with the no-servers-at-the-guest-side restriction. The file transfer mechanism will not 
+be "symmetric". 
+
+The following diagram depicts a bird's eye view of the system's architecture:
+  [[Image(arch.png)]]
+
+
+  === Command Execution ===
+  Requesting the execution of a program contained in the guest fit nicely into an async.
+  message passing infrastructure: a tailored message addressed to the guest we want to
+  run the command on is published, processed by this guest and eventually answered back
+  with some sort of status (maybe even periodically in order to feedback about progress). 
+
+  Given the subscription-based nature of the system, several guests can be addressed at
+  once by a single host, triggering the execution of commands (or any other action
+  covered by this mechanism) in a single go. Note that neither the hosts nor the 
+  (arbitrary number of) guests need to know how many of the latter conform the system:
+  new guest instances need only subscribe to these "broadcasted" messages on their own
+  to become part of the overall system. This contributes to the *scalability* of the system.
+
+
+  === File Transfers ===
+  This is a trickier feature: transfers must be bidirectional, yet the guests can only
+  act as clients, with -a priori- no knowledge of the hosts' location (remember that the
+  only link between host and guest(s) is the broker). 
+  The proposed solution consists of a simple HTTP server on the host side. The reasons 
+  to consider HTTP are twofold:
+
+  * Bypassing firewalls. Even if this wouldn't usually be an issue -host and guests running
+    on the same machine-, HTTP is the less likely protocol to be filtered out by firewalls.
+  * Simplicity. It's possible to implement -or reuse- such a server with a small footprint.
+    Likewise for the guests' client side.
+
+  The only requirement on this HTTP server is that is must accept PUT requests, in order
+  for the guests to upload files. 
+
+  To enable the guests to access this HTTP server, its host and port must be handed to them.
+  This is accomplished, how else, by message passing: because host and guests are all subscribed
+  to a common broker on a common topic, such information can be published (ie, broadcasted) in
+  a way very much similar to the one described in the previous point for commands. Once guests
+  have taken notice of the file host (which does *not* have to be the same as the BOINC host),
+  file transmission is handled in the same message-oriented way, once the guests
+  are aware of the file host details.
+
+
+  === The Devil in the Details ===
+  In the previous argumentations, it was implied that both host and guests were
+  already connected to a common broker. This is clearly not the case upon startup. 
+  Elaborate mechanism might include the use of [http://www.zeroconf.org/ Zeroconf] methods 
+  such as [http://avahi.org/ avahi] or 
+  [http://bonjour.macosforge.org/ bonjour], but in principle and for the 
+  time being, it can be assumed that the broker
+  will be accesible to the guest on the IP acting as its default gateway. In the common case
+  of a NAT setup for the guests, this IP is configurable at the hypervisor level, corresponding
+  to the BOINC host machine. Even if we wanted to balance the load, placing the broker somewhere
+  else, an adequate port forwarding mechanism could be setup at the BOINC host side.
+
+  As a particularity of VirtualBox NAT system, there is no out-of-the-box connectivity between
+  the hypervisor host and the VM (for details, see the VirtualBox Manual, section 6.4). 
+  Fortunately, VirtualBox includes a port forwarding mechanism that makes it possible to map
+  ports on the VM to the hypervisor's host, circumventing this problem. 
+  On the other hand, there should be no need to forward any port from the VM, as it's 
+  supposed to perform only client-side operations. 
+  On the other hand, if other -unrelated- services, such as the CernVM's web-based administration
+  mechanism, need be accessed, it's convenient to keep in mind this feature. Details are available
+  at the aforementioned VirtualBox Manual, section 6.4.1.
+
+
+  === Open Questions ===
+  * Where should the broker live? Conveniently on the same machine as the hypervisor or on
+    a third host? Maybe even a centralized and widely known (ie, standard) one? This last option
+    might face congestion problems, though. 
+  * Broker choice. Full-fledged ([http://activemq.apache.org/ ActiveMQ]) or more limited but lighter? 
+    (ie, [http://www.germane-software.com/software/Java/Gozirra/ Gozirra]). On this 
+    question, unless a centralized broker is universally used, the lighter version largely suffices.
+    Otherwise, given the high load expected, a more careful choice should be made.
+
+
+== Implementation ==
+[http://cernvm.cern.ch/cernvm/ CernVM] has been taken as the base guest system.
+In order to run the developed prototype, the
+following packages need to be installed (by running conary update <package>):
+  - subversion
+  - gcc
+
+The following python projects (in the following order) 
+  have to be downloaded and installed using the provided setup.py:
+
+  - Zope interfaces (http://www.zope.org/Products/ZopeInterface)
+  - twisted (http://twistedmatrix.com)
+  - stomper (subversion repository @ http://stomper.googlecode.com/svn )
+
+This prototype has been implemented in Python, given its cross-platform nature and the suitability
+of the tools/libraries it provides.
+
+  === Overview ===
+  Upon initialization, guests connect to the broker, that's expected to listen on the 
+  default STOMP port 61613 at the guest's gateway IP. 
+  Once connected, it "shouts out" he's joined the party, providing a its unique id (see
+  following section for details). Upon reception, the BOINC host notes down this unique id for
+  further unicast communication (in principle, other guests don't need this information). The 
+  host acknowledges the new guest (using the STOMP-provided ack mechanisms).
+
+
+  === Unique Identification of Guests ===
+  The preferred way to identify guests is based on [http://tools.ietf.org/html/rfc4122 UUID]. 
+  Python includes a uuid module since version 2.5. Unfortunately, CernVM ships with version 2.4, 
+  so the uuid.py file define this module had beed to include "by hand".
+
+  === Tailor-made STOMP Messages ===
+  For command execution:
+{{{
+    HEADERS:
+      cmd: cmd-to-execute
+      stdout: stdout-file
+      stderr: stderr-file
+      [shell: shell-to-use] (def: /bin/sh)
+      cwd: cwd-to-use
+      env: mapping-defining-exec-env
+}}}
+
+{{{
+    BODY: 
+      VM-RUN-CMD
+}}}
+
+  For file transfers:
+    The negotiation for the direct connection between the file host and the guest(s) takes
+    place using the following message:
+
+    Once the guests have this information, the following message format deals with the actual
+    file transfer requests:
+
+{{{
+    HEADERS:
+      host: file-host-address
+      port: file-host-port
+      path: path-to-files
+      files: list-of-files
+}}}
+
+{{{
+    BODY:
+      UP[-TO-HOST] | DOWN[-TO-HOST]
+}}}
+    
+    
+== Conclusions ==
+(TODO)
+
+Note that the described approach is equally valid for any hypervisor, rendering
+it as a candidate for an independent implementation for the aforementioned features
+of command execution and file transference.
+
+
+== Alternatives ==
+XMPP?
+
+