Over my last semester of college I took on a project that involved making changes to OpenStack. For those that aren’t familiar, OpenStack is an open source infrastructure as a service (IaaS) platform that’s quickly taking over the private cloud space. In the process I had to learn how to bring up DevStack, make my changes, use Tox to run tests, deploy my changes, and then debug them with PDB. In hindsight it’s really not all that difficult to get the workflow down, but for someone who hasn’t worked on an open source project the size and sophistication of OpenStack it was a challenge getting the pieces together. Hopefully with this post, I’ll spare some other poor graduate student the trouble of learning all these lessons from the firehose of information that is the internet.
First off, you have to understand that OpenStack is a huge project that would be impossible to understand in its entirety. My mentor likened its structure and governance to an aircraft carrier. Spend a good chunk of time in discovery mode consuming everything you can on the project. The diagram above gives a nice overview of the architecture which consists of a number of loosely coupled and independently scalable software components. It’s also good to have a clear understanding of your end goal at this point. That way you can start identifying the relevant OpenStack projects and their sub-projects recursively. For me this meant learning about Nova, how it works, and how I can augment its resource provisioning capabilities.
On the other non-technical end of your research you should also gain a firm understanding of the community processes that govern OpenStack. If not for these community processes OpenStack would likely collapse under its own weight. While ideally everything would be well documented at all times, OpenStack is too large and this approach simply doesn’t scale well. The solution to this is to write code with consistent style, that is self documenting, and is thoroughly tested. Younger me came from the world of standard libraries with rich documentation and expected that of all software. After this project I now understand how unrealistic that is in a project that’s always in flux like OpenStack. With high quality code you can break into the source and understand exactly what’s going on in that vignette. This is a very powerful revelation and will crop up over and over when developing on OpenStack so get used to the idea that the developer docs will either not exist or become out of date very quickly. Rely solely on the source code and trust the processes that brought it to that state.
Since you’re reading this, one of your goals is development and you will have learned about DevStack during your research. Once cloned to a machine it can bring up a working deployment of OpenStack with one simple incantation of “./stack.sh”. Before we get there though, make sure you are deploying on a “sacrificial” machine such as a VM. For my development, I used a 64-bit Ubuntu 14.04 VM from a production OpenStack environment. Installing DevStack on your personal computer is something you definitely do not want to do. stack.sh runs with passwordless sudo taking a lot of liberties with the machine to deploy DevStack and it will break anything that gets in the way of that goal. I also suggest taking the time to skim through stack.sh to get an idea of what I mean.
Once you understand the risks and prerequisites, setup the minimal configuration as described in the documentation. The local.conf file allows you to change anything about the default DevStack deployment that you need. Basically, it’s an INI file that turns into environmental variables when you run stack.sh. The documentation is sparse so you will have to read the stack.sh source to understand what variables you need to set. For my project I had to set the hypervisor to Xen and the floating IP range so DevStack could create my VMs properly. You can also set the particular releases of each service to checkout which is handy if you want others to easily repeat your work. Finally, make sure that the user running stack.sh has passwordless sudo by modifying your sudoers file.
With everything setup, simply run stack.sh to deploy OpenStack. This will take about half an hour to run and if nothing goes wrong you will have a working OpenStack deployment in the end. If you’re using the minimal configuration this will setup a OpenStack with a KVM host backed by fully virtualized CPUs from QEMU. Glance will have a CirrOS image with a username “cirros” and password “cubswin:)” for testing purposes. To actually start using the OpenStack environment you can log into Horizon as the “demo” or “admin” users (demo being a normal user). For development though, it’s more convenient to use the command line clients so in the /devstack directory there’s a file called openrc that will store your Keystone credentials as environmental variables. Simply run “source openrc” and the clients will be able to use these variables from the shell to get authentication tokens until you log off.
Play around with OpenStack to the point where you’re creating clusters of VMs in virtual networks. When you’re ready to make changes to the OpenStack source, navigate to /opt/stack/ which is where DevStack has cloned all the OpenStack services. Note the folder for logs which is handy for debugging. Each service has its own Git repository and in that repository is a folder containing the Python source (ex. Nova source is in /opt/stack/nova/nova). Like I mentioned earlier, the documentation is sparse when it comes to development so start locating points of interest and setting breakpoints in the source by adding the line “import pdb; pdb.set_trace()”. These changes go in effect when the service restarts so don’t worry about breaking your DevStack.
Before we can deploy our changes, you should first run the unit tests by going to the root directory of the service you’re modifying and running “tox”. Tox will then go out and fetch all the dependencies for the tests and deploy them in a virtual environment, due to caching, this makes the first execution a little longer. With the environments setup Tox will iterate through all the tests for about 10-20 minutes. Thanks to the OpenStack community we have complete code coverage as well as regression tests. When contributing code you should adhere to this standard as well. It also performs a PEP8 style check which enforces the consistent look and feel of the code.
If the tests pass, you can now deploy your changes by restarting the service. What DevStack did to make all this work was launch all the services in a screen session which you can think of as a separate terminal. To reconnect to the session simply run “screen stack”. The last service launched should still be in the foreground so simply “ctrl-z” and “bg” it. The next few steps are a bit tricky and involve some guesswork, but don’t worry about breaking anything. If you run into trouble just run unstack.sh and then stack.sh to start fresh. Now to restart, you first need to kill the OpenStack service you changed. To find Nova I used “pgrep -a nova” and took a quick note of the command I was about to kill. Then “sudo kill ” terminates the process. Afterwards, you need to run “history” and look for the command DevStack used to bring up the service which looks like the command you just took note of. All that’s left is to then run that command and your modified service should now be running. To detach from the screen session just hit “ctrl+d”.
Now that our service is running we need to trigger the breakpoints we set. For me, I broke into the VM spawning sequence which could be triggered by launching a “nova boot”. After that, the PDB console will be inside your screen session. With PDB you now have an interactive debug session. This is a great way to understanding how OpenStack works by examining the stack trace, the variables, and stepping deeper into the abyss. One thing worth exploring are the interfaces where OpenStack services meet. You probably won’t work on this layer, but it’s nice to have an understanding of where the edges of your black box are. Once you’re comfortable with using PDB to look at your program state, you’ll have all the tools necessary to begin development on OpenStack.
This is definitely a lot to take in so don’t get discouraged by the complexity. It took me the better part of a semester to actually put it all together in practice. The goal of this post was to present one possible workflow for OpenStack development and distill all the information I wish I knew at the onset of this project. Full disclosure though, I am by no means an expert on this subject but I do hope this was helpful for other novices such as myself.