Adding Xen VTPM Support to OpenStack

Previously, I wrote about how to survive OpenStack Development so now I’ll write about how I applied that knowledge. The goal of this project was to build infrastructure that would allow an untrusting cloud tenant to verify for themselves the integrity of their VM and the hypervisor it runs on. If you’ve done any work on cyber security you will understand that trust is very valuable these days, so this is a big step in allowing security sensitive entities the ability to make use of cloud computing infrastructure.

Building on that motivation, there’s this idea of Trusted Computing which is a set of technologies developed to make sure that a computer “behaves in an expected way”. For example, if there were a rootkit in the BIOS or kernel we don’t necessarily want to patch it up and carry on; we instead want to recognize that something unexpected happened and remove the system from the ring of trust before it can do any harm. I.e. we want to prevent this machine from accessing any higher level security services.

In order to do this we make use of a piece of hardware known as a TPM which has a set of registers known as PCRs and a hardware bound endorsement key (EK) for signing the PCRs. This configuration allows us to trust that the PCRs are coming from a particular TPM and that they haven’t been tampered with. To make this useful we measure (hash) any critical bit of software before running it and “extend” it into a PCR creating a hash chain. What you end up with is a set of registers that represent all the software that got you into user space. If you have a “white list” of PCR values, you can then immediately detect any anomalies. The caveat here is that this is only as good as your measurement and attestation infrastructure.

What we did extend the measurement infrastructure beyond the physical hardware and into VMs. What’s tricky is that you can no longer rely on a single physical TPM when you have multiple operating systems trying to measure things concurrently. To get around this hypervisors have implemented Virtual TPMs (VTPMs) which both provides a unique TPM to each VM and exposes the the physical TPM’s PCR values. That way, a tenant running in the VM’s user space can first attest the integrity of their hypervisor with the physical TPM and then extend that root of trust up to their OS with the VTPM. Currently, this is a fairly cumbersome process so we seek to automate all of this by allowing OpenStack to provision VTPM resources and integrating it with attestation infrastructure (Keylime) developed by this project’s mentor.

The diagram above illustrates our stack which consists of Xen, OpenStack, and Keylime. In Xen, you have domain0 which you can think of as the “root” user for the hypervisor. It exposes hypervisor management through a native library known as LibXL. In order to support multiple virtualization layers, OpenStack uses LibVirt as a common abstraction layer. What’s missing is support for VTPMs in everything above the LibXL layer.

Starting with the LibVirt layer, we needed a way to define our intent to spawn a VTPM so I wrote a specification for a new device in the domain configuration file. This was a matter of parsing parameters out of the XML file and into internal data structures. On the other end I then translated those internal configuration structures into the native Xen structures to spawn the VM. Overall, it was a very straightforward patch to carry out and my hope is to get it pushed upstream at some point.

The tricky bit was dealing with the OpenStack layer and its immensity. On top of spawning the VM it also had to provision the VTPM resources beforehand. As far as Xen was concerned, you needed a UUID from the VTPMMGR and a small backing image to create a VTPM. To get the UUID, we needed to expose a rest API on the Keylime VM which acted as a proxy to the VTPMMGR. This was necessary because Domain0’s kernel yields the physical TPM to the hypervisor by removing support for TPMs entirely. For the image, we simply had Nova create a file out of /dev/zero. Putting this all together, Nova provisions these resources and generates an XML file for the VTPM which goes into the patched LibVirt. After that, we generate the last XML file for the VM that connects it to the VTPM.

I learned a lot from this project and in some ways it may have changed my career trajectory. For one, I realize that I still love building backendy infrastructure type things. Unexpectedly, I also developed more of an appreciation for FOSS and its community. I’m now much more comfortable diving into these projects, reaching out for help, and contributing patches then I ever was before. Bugs that I would quietly complain to myself about before are now submitted as bug reports and I’d make an honest effort to patch myself. Most importantly though, this project and the class that came with it opened my eyes to the exciting work going on in cloud computing. While I missed being on the ground floor of this work, I believe were on the cusp of a Cambrian explosion of sorts in this field.

README and Nova Patch

LibVirt Patch

A Beginner’s Guide to OpenStack Development

Over my last semester of college I took on a project that involved making changes to OpenStack. For those that aren’t familiar, OpenStack is an open source infrastructure as a service (IaaS) platform that’s quickly taking over the private cloud space. In the process I had to learn how to bring up DevStack, make my changes, use Tox to run tests, deploy my changes, and then debug them with PDB. In hindsight it’s really not all that difficult to get the workflow down, but for someone who hasn’t worked on an open source project the size and sophistication of OpenStack it was a challenge getting the pieces together. Hopefully with this post, I’ll spare some other poor graduate student the trouble of learning all these lessons from the firehose of information that is the internet.

OpenStack Architecture

First off, you have to understand that OpenStack is a huge project that would be impossible to understand in its entirety. My mentor likened its structure and governance to an aircraft carrier. Spend a good chunk of time in discovery mode consuming everything you can on the project. The diagram above gives a nice overview of the architecture which consists of a number of loosely coupled and independently scalable software components. It’s also good to have a clear understanding of your end goal at this point. That way you can start identifying the relevant OpenStack projects and their sub-projects recursively. For me this meant learning about Nova, how it works, and how I can augment its resource provisioning capabilities.

On the other non-technical end of your research you should also gain a firm understanding of the community processes that govern OpenStack. If not for these community processes OpenStack would likely collapse under its own weight. While ideally everything would be well documented at all times, OpenStack is too large and this approach simply doesn’t scale well. The solution to this is to write code with consistent style, that is self documenting, and is thoroughly tested. Younger me came from the world of standard libraries with rich documentation and expected that of all software. After this project I now understand how unrealistic that is in a project that’s always in flux like OpenStack. With high quality code you can break into the source and understand exactly what’s going on in that vignette. This is a very powerful revelation and will crop up over and over when developing on OpenStack so get used to the idea that the developer docs will either not exist or become out of date very quickly. Rely solely on the source code and trust the processes that brought it to that state.

Since you’re reading this, one of your goals is development and you will have learned about DevStack during your research. Once cloned to a machine it can bring up a working deployment of OpenStack with one simple incantation of “./”. Before we get there though, make sure you are deploying on a “sacrificial” machine such as a VM. For my development, I used a 64-bit Ubuntu 14.04 VM from a production OpenStack environment. Installing DevStack on your personal computer is something you definitely do not want to do. runs with passwordless sudo taking a lot of liberties with the machine to deploy DevStack and it will break anything that gets in the way of that goal. I also suggest taking the time to skim through to get an idea of what I mean.

Once you understand the risks and prerequisites, setup the minimal configuration as described in the documentation. The local.conf file allows you to change anything about the default DevStack deployment that you need. Basically, it’s an INI file that turns into environmental variables when you run The documentation is sparse so you will have to read the source to understand what variables you need to set. For my project I had to set the hypervisor to Xen and the floating IP range so DevStack could create my VMs properly. You can also set the particular releases of each service to checkout which is handy if you want others to easily repeat your work. Finally, make sure that the user running has passwordless sudo by modifying your sudoers file.

With everything setup, simply run to deploy OpenStack. This will take about half an hour to run and if nothing goes wrong you will have a working OpenStack deployment in the end. If you’re using the minimal configuration this will setup a OpenStack with a KVM host backed by fully virtualized CPUs from QEMU. Glance will have a CirrOS image with a username “cirros” and password “cubswin:)” for testing purposes. To actually start using the OpenStack environment you can log into Horizon as the “demo” or “admin” users (demo being a normal user). For development though, it’s more convenient to use the command line clients so in the /devstack directory there’s a file called openrc that will store your Keystone credentials as environmental variables. Simply run “source openrc” and the clients will be able to use these variables from the shell to get authentication tokens until you log off.

Play around with OpenStack to the point where you’re creating clusters of VMs in virtual networks. When you’re ready to make changes to the OpenStack source, navigate to /opt/stack/ which is where DevStack has cloned all the OpenStack services. Note the folder for logs which is handy for debugging. Each service has its own Git repository and in that repository is a folder containing the Python source (ex. Nova source is in /opt/stack/nova/nova). Like I mentioned earlier, the documentation is sparse when it comes to development so start locating points of interest and setting breakpoints in the source by adding the line “import pdb; pdb.set_trace()”. These changes go in effect when the service restarts so don’t worry about breaking your DevStack.

Before we can deploy our changes, you should first run the unit tests by going to the root directory of the service you’re modifying and running “tox”. Tox will then go out and fetch all the dependencies for the tests and deploy them in a virtual environment, due to caching, this makes the first execution a little longer. With the environments setup Tox will iterate through all the tests for about 10-20 minutes. Thanks to the OpenStack community we have complete code coverage as well as regression tests. When contributing code you should adhere to this standard as well. It also performs a PEP8 style check which enforces the consistent look and feel of the code.

If the tests pass, you can now deploy your changes by restarting the service. What DevStack did to make all this work was launch all the services in a screen session which you can think of as a separate terminal. To reconnect to the session simply run “screen stack”. The last service launched should still be in the foreground so simply “ctrl-z” and “bg” it. The next few steps are a bit tricky and involve some guesswork, but don’t worry about breaking anything. If you run into trouble just run and then to start fresh. Now to restart, you first need to kill the OpenStack service you changed. To find Nova I used “pgrep -a nova” and took a quick note of the command I was about to kill. Then “sudo kill ” terminates the process. Afterwards, you need to run “history” and look for the command DevStack used to bring up the service which looks like the command you just took note of. All that’s left is to then run that command and your modified service should now be running. To detach from the screen session just hit “ctrl+d”.

Now that our service is running we need to trigger the breakpoints we set. For me, I broke into the VM spawning sequence which could be triggered by launching a “nova boot”. After that, the PDB console will be inside your screen session. With PDB you now have an interactive debug session. This is a great way to understanding how OpenStack works by examining the stack trace, the variables, and stepping deeper into the abyss. One thing worth exploring are the interfaces where OpenStack services meet. You probably won’t work on this layer, but it’s nice to have an understanding of where the edges of your black box are. Once you’re comfortable with using PDB to look at your program state, you’ll have all the tools necessary to begin development on OpenStack.

This is definitely a lot to take in so don’t get discouraged by the complexity. It took me the better part of a semester to actually put it all together in practice. The goal of this post was to present one possible workflow for OpenStack development and distill all the information I wish I knew at the onset of this project. Full disclosure though, I am by no means an expert on this subject but I do hope this was helpful for other novices such as myself.

Setting up a Cloud on the Cloud with SLURM

Awhile back I vowed to dip my feet into the world of cloud computing without really understanding what exactly the cloud was. In the simplest terms I can put it, the cloud provides virtual building blocks to build a data center on the cheap. The key word which makes this all possible is “virtual”. Real servers running all sorts of virtualization hardware/software significantly increases the number of services a single server can offer which drives down the cost. Since these services are virtual, software can then offer a level of speed/flexibility that’s simply not possible when dealing with real hardware. This is the primordial soup that enables the internet we know today. It’s the reason I can host my blog to the world without worrying about the cost is because I share this IP address with at least 210 other domains. It’s also how Netflix can adapt the number of servers in their cluster throughout the day to match the diurnal patterns of its human end users. Cloud computing is what’s driving the current revolution on the internet and I’m glad I was able to take on this project to understand how it works in the backend.

In this project my colleague and I presented an experimental method for analyzing scheduling policies for batch computing clusters. Basically this meant implementing benchmarks that stress test a scheduler and produce a metric for comparison. These benchmarks amounted to running a bunch of jobs that reserves a number of nodes for a certain amount of time. The paper we pulled the paremeters from called it the “Effective System Performance” (ESP) benchmark. My implementation of this was a Python script that submitted sleep jobs to the cluster with different reservation parameters. It also scaled the benchmark to the size of the cluster being tested so its performance was agnostic of the underlying hardware. We also implemented a few other benchmarks, but they did not end up producing any interesting data.

The fun part was setting up a cloud to test our methods. Through the university we had access to the Massachusetts Open Cloud (MOC) which was running Open Stack to provide a public cloud. On this cloud we spun up 8 servers to create a small cluster. We then setup SLURM on the cluster which is a batch job scheduler. In other words with SLURM you can submit work to a control daemon and it finds a suitable set of computer slaves to do that work. People that have worked in HPC or any kind of scientific computing should have experience in a system like this. Overall, it wasn’t too difficult to set up but it did test my competency in navigating and administrating Linux servers. By the end we had the cluster under control with near complete startup scripts that can immediately bring new nodes online. We also had systems in place to roll out changes to the entire cluster which was essential for test and development. My only wish would have been to have more time to play with it and make it closer to production ready.

I’ve included the full writeup and source code in the bottom of this post. It provides all the information you need to set up your own SLURM cluster. The paper provides a more in-depth view of the technologies involved, what we looked into, our methods, and our results. This was the first time I’ve used the IEEE LaTeX template for a paper and the resulting output was just beautiful.

presentation slides

slurm writeup