Libvirt: Metering Domain Network Traffic

We can use the libvirt API to get network traffic at the interface level. So to meter the network traffic at the domain level, we have to find all interfaces the domain is using, and aggregate the traffic stats. This isn't too difficult — we just have to query the running domains, and query their XML descriptions individually in order to find the interfaces we need. One challenge we might face, using this approach, is the fact that any two interfaces could be using different networks. For example, let's say that we only want network traffic on the default network? Here is an example of how to generate this network traffic in Python. We're using the collections and the etree modules.

Hot-Plugging Data With Python And Libvirt

The CD-ROM drive revolutionized the way we're able to move data around on personal computers.  Never mind the internet, the notion of seamlessly transporting information over the wire between two machines was unthinkable fifteen years ago.  But besides transferring data back and forth, we've also come to depend on portable storage devices for practical reasons.

Sometimes the network isn't available to us.  Or, maybe there is a network, but we simply don't want our data traversing it.  We like to carry data around with us — on USB drives.

The portable storage abstraction persists in virtual infrastructures.  The same practical limitations we experience with physical machines still affect those that run on a hypervisor — machines aren't going to come pre-loaded with all the data they'll ever need.  This is where portable storage — hot-pluggable storage — becomes advantageous.

Data and devices
Plugging data into a virtual machine means setting up a virtual device and attaching it to the domain.  With Libvirt, this is easy to do because everything is defined using an XML description.  This means that we can create a device on the fly and plug it in — even after the domain is running.  Hot-plugging devices like this is important because we can't afford to shut-down the machine, attach a device, and start it up.  This incurs too much overhead — not to mention the inflexibility.  Imagine you had to reboot your laptop in order for your operating system to recognize a USB drive — you might as well install a new hard drive in the mean time.

Here is an example of how you can pass some arbitrary data to a running virtual machine.  There are three phases required here to take a string and transform it into a device the machine can read from:

1. Write the string to a file.
2. Create an ISO file.
3. Build the device XML

The Python code that builds the device an attaches it to a running domain...

import shutil
import os.path
import tempfile
import subprocess

import libvirt

def mkdata(data):
    tmpdir = tempfile.mkdtemp()
    tmpfile = tempfile.mkstemp(dir=tmpdir)[1]
    with open(tmpfile, 'w') as ofile:
        ofile.write(data)
    return tmpdir

def mkiso(indir):
    tmpiso = tempfile.mkstemp(suffix='.iso')[1]
    subprocess.call(['mkisofs', '-o', tmpiso, indir])
    shutil.rmtree(os.path.abspath(indir))
    return tmpiso
    
def mkdevice(iso):
    return """<disk type='file' device='disk'>
                <driver name='qemu' type='raw'/>
                <source file='%s'/>
                <target dev='sdb' bus='usb'/>
              </disk>""" % iso

if __name__ == '__main__':
    
    conn = libvirt.open('qemu:///system')
    domain = conn.lookupByName('MyVM')
    
    tmpdir = mkdata('Some virtual machine data...')
    tmpiso = mkiso(tmpdir)
    device = mkdevice(tmpiso)
    
    domain.attachDevice(device)

Let's take a closer look at what we're doing here.

The first function, mkdata(), takes an input value and writes it to a temporary file.  But before doing so, it creates a temporary directory for the file.  The reason being, we need a directory to create the ISO format for the device.

The second function, mkiso(), takes an input directory and passes it to the mkisofs command.  Once the ISO is built, we can remove the temporary directory and return the ISO file location.

The third function, mkdevice(), builds the Libvirt device we're going to attach to the domain.  The iso parameter is the ISO file we want to pass to the virtual machine, generated by mkiso().  You'll notice that this is a disk device, as that is what an ISO image is — a disk.  The driver element specifies that this disk is in raw format.  There are other options available such as qcow, but with this approach, the format is almost always raw.  The source element in the device description points to the ISO file we've just generated.

The target element tells the domain what the device label should be.  As the Libvirt documentation spells-out, there is never any guarantee that the device label you give it here will be retained by the guest operating system.  For example, in our case, we're telling the guest domain to label the new device as sdb — sda is probably taken.  However, if sdb is already taken, then we won't know for sure what the device will be called.  This is one area that this method is lacking in — ensuring that the device will be named appropriately so it may be referenced later on.

Finally, with these three functions that'll build the device we're ready to attach it to a domain.  We do this here by establishing a Libvirt connection and locating the domain we're interested in.  Next, we build the device and attach it.

Using the data
If you ran the above example and all went well, you now have a new disk device attached to your domain.  This is great — new data has been made available, no need to access a network and no need to reboot.  But inside the guest, applications need access to the data to make use of it.

Below is an example of how you could listen for the new device — from within the machine — and read it.

import os, os.path
import subprocess
import time

while True:
    if not os.path.exists('/dev/sdb'):
        time.sleep(10)
        continue
    
    subprocess.call(['mount', '/dev/sdb', '/mnt/sdb'])
    for fname in os.listdir('/mnt/sdb'):
        with open(os.path.join('/mnt/sdb', fname)) as ifile:
            print 'Got some data...'
            print ifile.read()

Here, all we're doing is checking, every ten seconds, if the device we just attached exists.  If it does, we're then able to mount it and read from the file within the ISO.

Libvirt And The Virtual Standard

Libvirt is more than just a virtualization library — it strives for standardization among virtual resources. This matters because hypervisors abound, they all have different capabilities.  The Libvirt library assimilates an interface for communicating with different hypervisor technology and treating them as one in the same.  But there is more to the Libvirt API than just functions that create and manipulate virtual machines.

More than machines
There is certainly more to operating a virtual infrastructure than provisionally instantiating and terminating virtual machines.  There are networks, there is storage, and there are the physical hosts themselves who carry out these operations.  Any virtualization library hoping to elucidate these things will need to differentiate between them — to create abstractions for them.

Considering the number of hypervisors that Libvirt actually supports, this is a challenging feat — the goal is for the developer using the library to have a seamless integration between the divergent hypervisors.

These abstractions are the virtual entities that comprise virtual infrastructures.  To be interoperable, they need to behave the same.  That is, their external interfaces must not differ per hypervisor connection.  If it were just the virtual machine control we're concerned with — starting, stopping, destroying, snap-shots — this would be no problem.  But the fact of the matter is that there are dozens of virtual entity types that need to be acknowledged.

In addition to other virtual entities such as networking and storage, the machine entities themselves have internal parts.  Machines are composites made up of several smaller components — CPUs, disk devices, memory.  These all fall outside of the core virtualization idea of emulating CPU instructions.  Effectively managing a large-scale, or even a mid-scale virtual environments, requires abstraction at a higher level.

Virtual documents
Inside code that uses Libvirt, inside the Python binding at least, these virtual entities are represented as traditional objects.  Just like a user object or a file object, virtual entities have methods and attributes — used to query the state of the object or to alter it in some fashion.  What is unique about how Libvirt deals with it's virtual entities is that they're described in an XML format.  Any given virtual entity can be reduced to an XML description.  A document.

Whats interesting to me about this approach is that XML has an inherent ability to be decomposed and rearranged.  Any programming language can parse and generate XML data — so with any virtual entity XML from Libvirt, we're able to take it's XML description and splice it up into several smaller documents should we choose to do so.

For example, the XML schema for domains — virtual machines — in Libvirt can be somewhat lengthy.  The XML description needs to tell Libvirt about everything the machine needs to operate, including all devices, CPU tuning, network connections, etc.  So of course it's going to have some heft to it.  Our software that uses Libvirt can take a given XML description for a domain an find the interesting sub-elements it needs — it can drill down and organize the domain data in it's own way.

A side-effect to using XML in this way is that it gives the various components of our virtual infrastructure the freedom to devise their own abstractions for virtual entities.  Let's say you want a memory abstraction — one for domains that you can use to attach additional behavior.  The memory abstraction is then constructed based on the parsed domain XML.  Additionally, this abstraction could play a role in constructing XML descriptions for domains.

Another corollary to XML documents is that they're documents — they've an inherent ability to be treated as shared resources that can be passed around in a distributed environment.

More than one host
Another, perhaps less obvious, reason for having XML documents describe the virtual entities in our infrastructure is that there are going to be multiple hosts.  The host is where the hypervisor manages virtual machines, so you're going to need more than one.  Libvirt really shines here because our hyperviser environment may be heterogeneous — many different hypervisors each with their own physical hardware.

Now imagine that we've got our own tailor made software for controlling various things — the management of the virtual infrastructure.  This means that we'll need to pass around virtual XML documents to various hosts that control the entity.

For example, let's say we want to define a new domain.  Our software builds the XML document that will tell Libvirt how to make it.  Our first attempt to create and start the domain on host A fails.  So we can try it again on another host in our infrastructure.  In theory, we shouldn't care what hypervisor is running on host B for our second attempt — the same XML document that defines the domain will do.  This is a key benefit of having a standardized XML schema for virtual entities — there is one schema for many hypervisors.

Outside of the XML standardization, Libvirt offers another type of virtualization standard — connecting to multiple hosts.  It is one thing to write some agent code that will use Libvirt to control our virtual infrastructure.  This code will need to live on each host where the hypervisor runs.  But, alas, we can use Libvirt to establish remote connections to the hypervisor in a heterogeneous way.  What this part of the virtual standard libvirt implements means is that we can put more focus on our controlling code — we can worry more about making stuff that best promotes our infrastructure.  The physical hosts we're using don't need our application running on them — only Libvirt and the hypervisor itself.

The absent stuff
The Libvirt library fills a large gap between disparate hypervisor technologies.  Perhaps most important, Libvirt unifies the virtual entities common in any infrastructure and abstracts them.  This is how we're able to use multiple hosts to deploy virtual machines without having to worry about the hypervisor installed on each.  Another innovation in the virtualization standard that we're sorely missing in this field is that of the XML document — used to describe virtual entities.  The concept of an XML document isn't new to most code and can be easily and quickly adapted for any software.

But despite these achievements, Libvirt isn't exactly a standard — although, there isn't anything else that better unifies virtualization technology.  There is still some betterment that needs to take place in order to better facilitate our infrastructures.  For example, we know little about the capabilities of the hypervisor's host — this XML schema still hasn't solidified.  Another hurdle in the virtual standard is that despite having abstract entities, not every hypervisor is going to support identical functionality that Libvirt offers.  Our code that uses Libvirt can sort these errors out, but one must ask, can there really be a standard where not all hypervisors agree on at least a minimal set of common features?

Polymorphic Libvirt

The Libvirt virtualization library is great for interacting with heterogeneous hypervisors. Depending on the hypervisor connection, most domains support the same virtual machine interfaces. For instance, you can you the same code to start and stop virtual machines using Libvirt for both KVM and Xen.

But when the operation isn't supported for a particular hypervisor type, you better be prepared to handle this scenario. Libvirt only defines a single exception type but we can determine that it is a hypervisor interface support issue by looking at the error code.

This is shown below. The following operation on the domain will fail on KVM.

import libvirt

if __name__ == "__main__":

   conn = libvirt.open("qemu:///system")

   for id in conn.listDomainsID():

       domain = conn.lookupByID(id)
      
       try:
           domain.reboot(0)
       except libvirt.libvirtError, e:
           if e.get_error_code() == libvirt.VIR_ERR_NO_SUPPORT:
               print "Cannot Reboot"

Python Libvirt Example

Just like block device statistics can be retrieved from libvirt guest domains, network interface statistics can also be retrieved. The main modification is that the XML data comes from a different element and a different method on the domain is invoked. Here is an example of how network interface statistics are retrieved from libvirt.

#Example; Libvirt network stats.

#We need libvirt and ElementTree.
import libvirt
from xml.etree import ElementTree

#Function to return a list of network devices used.
def get_target_devices(dom):
  #Create a XML tree from the domain XML description.
  tree=ElementTree.fromstring(dom.XMLDesc(0))

  #The list of network device names.
  devices=[]

  #Iterate through all network interface target elements of the domain.
  for target in tree.findall("devices/interface/target"):
      #Get the device name.
      dev=target.get("dev")
  
      #Check if we have already found the device name for this domain.
      if not dev in devices:
          devices.append(dev)
      
  #Completed device name list.
  return devices

if __name__=="__main__":
  #Connect to some hypervisor.
  conn=libvirt.open("qemu:///system")

  #Iterate through all available domains.
  for id in conn.listDomainsID():
      #Initialize the domain object.
      dom=conn.lookupByID(id)
  
      #Initialize our interface stat counters.
      rx_bytes=0
      rx_packets=0
      rx_errs=0
      rx_drop=0
      tx_bytes=0
      tx_packets=0
      tx_errs=0
      tx_drop=0
  
      #Iterate through each device name used by this domain.
      for dev in get_target_devices(dom):
          #Retrieve the interface stats for this device used by this domain.
          stats=dom.interfaceStats(dev)
      
          #Update the interface stat counters
          rx_bytes+=stats[0]
          rx_packets+=stats[1]
          rx_errs+=stats[2]
          rx_drop+=stats[3]
          tx_bytes+=stats[4]
          tx_packets+=stats[5]
          tx_errs+=stats[6]
          tx_drop+=stats[7]
      
      #Display the results for this domain.
      print "\n%s Interface Stats"%(dom.UUIDString())
      print "Read Bytes:      %s"%(rx_bytes)
      print "Read Packets:    %s"%(rx_packets)
      print "Read Errors:     %s"%(rx_errs)
      print "Read Drops:      %s"%(rx_drop)
      print "Written Bytes:   %s"%(tx_bytes)
      print "Written Packets: %s"%(tx_packets)
      print "Write Errors:    %s"%(tx_errs)
      print "Write Drops:     %s"%(tx_drop)

Python Libvirt Example

The libvirt virtualization library is a programming API used to manage virtual machines with a variety of hypervisors. There are several language bindings available for the libvirt library including Python. Within a given Python application that uses the libvirt library, the application can potentially control every virtual machine running on the host if used correctly. Libvirt also has the ability to assume control of remote hypervisors.

Virtual machines, or guest domains, have primary disks and potentially secondary disks attached to them. These block devices and even be added to a running virtual machine. But just like a physical host, it helps to know exactly how the virtual block devices for a given virtual machine are being utilized. This way, potential problems may be addressed before they occur. Libvirt provides the ability retrieve such statistics for these devices. Here is a Python example of how to do this.

#Example; Libvirt block stats.

#We need libvirt and ElementTree.
import libvirt
from xml.etree import ElementTree

#Function to return a list of block devices used.
def get_target_devices(dom):
   #Create a XML tree from the domain XML description.
   tree=ElementTree.fromstring(dom.XMLDesc(0))
  
   #The list of block device names.
   devices=[]
  
   #Iterate through all disk target elements of the domain.
   for target in tree.findall("devices/disk/target"):
       #Get the device name.
       dev=target.get("dev")
      
       #Check if we have already found the device name for this domain.
       if not dev in devices:
           devices.append(dev)
          
   #Completed device name list.
   return devices

if __name__=="__main__":
   #Connect to some hypervisor.
   conn=libvirt.open("qemu:///system")
  
   #Iterate through all available domains.
   for id in conn.listDomainsID():
       #Initialize the domain object.
       dom=conn.lookupByID(id)
      
       #Initialize our block stat counters.
       rreq=0
       rbytes=0
       wreq=0
       wbytes=0
      
       #Iterate through each device name used by this domain.
       for dev in get_target_devices(dom):
           #Retrieve the block stats for this device used by this domain.
           stats=dom.blockStats(dev)
          
           #Update the block stat counters
           rreq+=stats[0]
           rbytes+=stats[1]
           wreq+=stats[2]
           wbytes+=stats[3]
          
       #display the results for this domain.
       print "\n%s Block Stats"%(dom.UUIDString())
       print "Read Requests:  %s"%(rreq)
       print "Read Bytes:     %s"%(rbytes)
       print "Write Requests: %s"%(wreq)
       print "Written Bytes:  %s"%(wbytes)

Libvirt 0.6.1

Looks like the latest Libvirt is now available (0.6.1). Only a couple new API features and some maintainance tasks but nonetheless moving forward.

ECP update

Over the last week or so, the ECP development team has been fixing several issues brought about by the ECP user community. One issue that has been resolved I'd like to point out.

The problem here is that during the ECP installation procedure, any existing Libvirt domains will be imported into the database. Obviously, the machine table must already exist in the database. When the user hits the "/install" url, the installer is run. The user may perform this action even after all the ECP database tables have been defined. This way, any Libvirt machines that have been created by some other machines may be imported.

One of the problems with this method is that sometimes, the machine import functionality is executed before the machine database table exists. Another requirement of the machine import functionality is that the local machine database record has been inserted into the machine database table. This is needed to determine what hypervisors are available on the local machine.

The fix made in this case was to check in the machine table exists. ECP cannot execute this functionality without it. Second, ECP will no longer assume that the local machine exists in the database. It will now check for both the machine table and the local machine record. If either is false, no machines will be imported. In this case, the "/install" url can always be reloaded once the required tables and records have been created.

So, how do we end up in a situation like this in the first place? Shouldn't the ECP installation functionality always ensure that the required data exists before it is needed? This is a perfectly valid concern and the installer does work this way. The table creation is the first task executed by the installer. Then, important records such as the local machine are inserted. The only way this ordering can be altered is if some extension functionality "hooks" into the installer. The hooks have a choice as to the order in which the "hooked" method is executed. The original invocation may happen before the new functionality or after. It is entirely the responsibility of the extension module to ensure that nothing is interrupted in the original behaviour.

That being said, there are several extension modules distributed along with ECP and we'll be keeping a close eye on those as usual. If anyone has noticed a potential defect in a core extension module, feel free to report it here.

New ECP exception and enhanced state restoring behaviour

Over the past few days, the ECP team has made some notable enhancements in the trunk. The first being the addition of a new exception called E2LibvirtError. As the title suggests, this exception is raised for Libvirt-related issues. The Python Libvirt library already defines an exception class. However, there are many types of errors that can happen from within Libvirt. The idea behind this new E2LibvirtError class is to provide better information when something bad happens in libvirt. For instance, in the Python Libvirt library, there is only one exception type. If this exception gets raised, a short message is displayed. This is the default message that gets initialized with the Python base exception class.

The problem here being that Libvirt can manage several different hypervisors on any given system. Thus, there are several layers within Libvirt in which something can go wrong. In ECP, the Libvirt exception is caught, and the generic message is recorded.

The new E2LibvirtError exception exploits additional exception information encapsulated within the basic Libvirt exception instance. I don't mean encapsulated in the traditional object-oriented sense. I mean the information is there, and ECP should use it for the benefit of the end user. The new ECP exception, when instantiated, will take several error codes from the original Libvirt exception and produce a much more meaningful error message.

This leads me to the changes made in the restore_machines_state() functionality. The rationale hasn't changed, only the implementation. We simply handle table existence and local machine existence detection much better than the current version. If the function finds a machine that is not running and it should be (because that was the state the machine was in when ECP last shut down), it will attempt to start it. We've already added the new E2LibvirtError exception handling to this function when attempting to start the machine since this is a Libvirt operation. I've already been seeing much more useful error messages in the logs. These new error messages should also be viewable in the web front-end via the error dialog box when something Libvirt-related goes wrong.

This does increase the Libvirt coupling in ECP a considerable amount. However, given the level of functionality that ECP would have without Libvirt, I think it is a fair trade off.

libvirt 0.6.0

Looks like libvirt 0.6.0 is now available. There are several new features, bug fixes, and improvements. I just tested this version with ECP 2.2 and everything works just fine.

Libvirt 0.5.0

This may be old news for some but libvirt 0.5.0 is now available. As usual, the bug fixes and improvements are spread throughout the library. There looks to be some interesting new features in the Qemu/KVM driver. Namely, "domain lifecycle event support" and "migration support".

The domain life cycle support includes Python bindings according to the change log whereas nothing is mentioned about Python support regarding the migration support for Qemu/KVM. Also, the domain life cycle support mentions only Qemu and Xen. I haven't actually tried using 0.5.0 yet but if I find anything else interesting to write about I will.

Python libvirt example

Xen sucks for my purposes. Xen is an extremely powerful virtualization platform and is why I don't need it. Few people do for development or experimental purposes. This is where KVM, QEMU, and Libvirt come in handy. I can use Python to easily write a simple application to create virtual machines and manipulate them as needed using the Libvirt Python binding. So, here is a simple example of how to get started using Libvirt in Python.

#Simple Python libvirt example.

import libvirt

if __name__=='__main__':
   conn=libvirt.open('qemu:///system')
   print 'Listing running domains'
   for id in conn.listDomainsID():
       dom=conn.lookupByID(id)
       print dir(dom)
          
   print 'Listing defined domains'
   for id in conn.listDefinedDomains():
       dom=conn.lookupByName(id)
       print dir(dom)

The great thing about Libvirt is that, in theory, should decide I want to use Xen, this same code should work. Libvirt is still in its' infancy with a long way to go but the ideas are right.