One thing that was intended as new, and I think worked well, was the splitting of the conference and design summit. It really made it a lot easier to be involved in the technical details and not have to choose between that and listening to folks talk to what they’re Openstack experiences have been, governance models and choices, and so forth. As an active contributor to (and general jack-of-all-trades with) a number of the OpenStack projects, it’s hard enough to balance my focus between the technical sessions.

There was also a ton of news related to OpenStack that is just fantastic for the community. Rackspace is starting to form out a foundation to run OpenStack long term, and the governance round-table at the conference made it clear that pretty much everyone agreed to a thoughtful, careful process to set it up to really make a long term run with OpenStack. There was notable consensus on ideas and vision of how to set up governance, including taking some long and deep looks at how other successful open source foundations have been set up, and looking to those foundations to learn what’s been successful, and what they might have done differently.

There was also the entrance of Hewlett Packard very publicly actively into the OpenStack world. Back in September, they said they’d be there, and with the conference they have committed hardware and time to continuous integration efforts. They were notably involved in the design summit as well, hosting a couple of sessions and getting actively involved in others. And I think most pleasing to me, they are actively submitting code contributions to the OpenStack core projects. I think it’s a little odd that they’re all coming in as “HP Nova Contributors“, but I’m glad to see it.

With HP running their cloud offering on OpenStack, they’re getting deeper in the details of the code. Dell’s been focusing their efforts on the amazing OpenStack deployment tool: Crowbar, and it seems clear to me that HP is taking their interest quite a bit deeper into the core projects.

I won’t even do justice the the individual project roadmaps from the summit – I think that the most of the project technical leads are likely recovering this weekend from an intensely focused week. In the next couple of weeks, we’ll see them hit the mailing list – and from what I saw from the sessions in the conference, there’s going to be some hard choices coming up on where to expend resources. Every one of the projects has more that they want to do than they will be able to smoothly accomplish in the next 6 months. We’ll see the details hitting the blueprints in the very near future I’m sure. Suffice to say that if you’ve been thinking about getting involved in OpenStack, now is very definitely the time to jump in – there’s a ton to do, and a lot of really interesting problems to solve in getting it done.

These instructions are assuming you’re starting with an Ubuntu based system – either 10.10 or 11.04. I haven’t tried it as yet with 11.10.

First things first, I recommend you make sure you have the latest bits of everything:

sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get autoremove

Then we need to add the release “PPA” so that your system can grab the packages for Openstack:

sudo apt-get install python-software-properties
sudo add-apt-repository ppa:openstack-release/2011.3
sudo apt-get update

Now we get into the details. I’m going to drive out the instructions that will start with a single host, but are set up to add additional virtualization hosts as you need. I’m writing this assuming you’re working in a small network, and setting it up for FlatDHCP networking. Choosing the networking strategy and IP address space to use is actually one of the trickier parts of doing a reasonable install. For just testing something out in a test lab, this setup will work reasonable well – the only thing to really note is that this *will* install a DHCP server to provide IP addresses to the virtual instances, so if you have another DHCP server handing out addresses, you might need to get into the details and change some of these settings.

Installing the packages:

OpenStack relies on using MySQL as a data repository for information about the openstack configuration, so we’ll need to set up a MySQL server. Normally when you install the packages for MySQL, it’ll ask you about configuring a root password and such. We can make that hands-off by pre-answering some of those questions. To do this, make a file named “/tmp/mysql_preseed.txt” and put in it the following:

mysql-server-5.1 mysql-server/root_password password openstack
mysql-server-5.1 mysql-server/root_password_again password openstack
mysql-server-5.1 mysql-server/start_on_boot boolean true

Then we can get into the commands to install the packages:

cat /tmp/mysql_preseed.txt | debconf-set-selections
apt-get install mysql-server python-mysqldb
apt-get install rabbitmq-server
# ^^ pre-reqs for running controller nova instance
apt-get install euca2ools unzip
# ^^ for accessing nova through EC2 APIs
apt-get install nova-volume nova-vncproxy nova-api nova-ajax-console-proxy
apt-get install nova-doc nova-scheduler nova-objectstore
apt-get install nova-network nova-compute
apt-get install glance

That’s got all the packages installed onto your local system! Now we just need to configure it up and initialize some information (that’s the bit about networks, etc).

Before I get into changing configs, let me explain what I’ll be setting up. In this example, my internal “network” is 172.17.0.0/24 – and I have a dedicated IP address for this host that is 172.17.0.133. The virtual machines will be in their own network space (10.0.0.0 to 10.0.0.254), and (at this point) not visible from the local network, but will be able to access the local network through their virtualization hosts. The machine I’m using also only has a single NIC (eth0), which is fine for a little test bed, but not likely what you want to do in any sort of real setup.

/etc/nova/nova.conf

--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--flagfile=/etc/nova/nova-compute.conf
--verbose
#
--sql_connection=mysql://novadbuser:novaDBsekret@172.17.0.133/nova
#
--network_manager=nova.network.manager.FlatDHCPManager
--flat_network_bridge=br100
--flat_injected=False
--flat_interface=eth0
--public_interface=eth0
#
--vncproxy_url=http://172.17.0.133:6080
--daemonize=1
--rabbit_host=172.17.0.133
--osapi_host=172.17.0.133
--ec2_host=172.17.0.133
--image_service=nova.image.glance.GlanceImageService
--glance_api_servers=172.17.0.133:9292
--use_syslog

Now you might have noticed the MySQL connection string in there. We need to set up that user and password in MySQL to do what needs to be done. I also change the MySQL configuration so that remote systems can connect to MySQL. It’s not needed on a single host, but if you ever want to have more than one compute host, you need to make this change. In /etc/mysql/my.conf, find the line:

bind-address = 127.0.0.1

and change it to

bind-address 0.0.0.0

Now lets make the user in Mysql:

mysql -popenstack
CREATE USER 'novadbuser' IDENTIFIED BY 'novaDBsekret';
GRANT ALL PRIVILEGES ON *.* TO 'novadbuser'@'%' WITH GRANT OPTION;
FLUSH PRIVILEGES;

And set up the database:

mysql -popenstack -e 'CREATE DATABASE nova;'
nova-manage db sync

If that last command gives you any trouble, then we likely don’t have the MySQL system configured correctly – the user can’t access the tables or something. Check in the logs for MySQL to get a sense of what might have gone wrong.

At this point, it’s time to configure up the internals of openstack – create projects, networks, etc.
We’ll start by creating an admin user:

# create admin user called "cloudroot"
nova-manage user admin --name=cloudroot --secret=sekret

This should respond with something like:

export EC2_ACCESS_KEY=sekret
export EC2_SECRET_KEY=653f3fad-df22-449b-9e6a-ea6c81e32621

You can scratch that down, but we’ll be getting that same information again later and using it, so don’t worry too much about it.

Now we create a project:

# create project "cloudproject" with project mgr: "cloudroot"
nova-manage project create --project=cloudproject --user=cloudroot

And finally, a network configuration for those internal IP addresses:

nova-manage network create private \
    --fixed_range_v4=10.0.0.0/24 \
    --num_networks=1 \
    --network_size=256 \
    --bridge=br100 \
    --bridge_interface=eth0 \
    --multi_host=T
# gateway assumed at 10.0.0.1
# broadcast assumed at 10.0.0.255

Now I’m using the multi-host flag, which is new in the Diablo release. This makes each compute node it’s own networking host for the purposes of allowing the VM’s you spin up to access your network or the internet.

At this point, you’re system should be up and running, all systems operational. Let me walk you through the command steps to actually kick up a little test VM though. These commands are all meant to be done as a local user (not root!)

sudo nova-manage project zipfile cloudproject cloudroot /tmp/nova.zip
unzip -o /tmp/nova.zip -d ~/creds
cat creds/novarc >> ~/.bashrc
source creds/novarc
#
euca-add-keypair mykey > mykey.priv
chmod 600 mykey.priv
#
image="ttylinux-uec-amd64-12.1_2.6.35-22_1.tar.gz"
wget http://smoser.brickies.net/ubuntu/ttylinux-uec/$image
uec-publish-tarball $image mybucket
#
wget http://uec-images.ubuntu.com/releases/10.04/release/ubuntu-10.04-server-uec-amd64.tar.gz
uec-publish-tarball ubuntu-10.04-server-uec-amd64.tar.gz mybucket
...OUTPUT...
Thu Aug 18 14:02:20 PDT 2011: ====== extracting image ======
Warning: no ramdisk found, assuming '--ramdisk none'
kernel : lucid-server-uec-amd64-vmlinuz-virtual
ramdisk: none
image  : lucid-server-uec-amd64.img
Thu Aug 18 14:02:29 PDT 2011: ====== bundle/upload kernel ======
Thu Aug 18 14:02:34 PDT 2011: ====== bundle/upload image ======
Thu Aug 18 14:03:12 PDT 2011: ====== done ======
emi="ami-00000002"; eri="none"; eki="aki-00000001";
...OUTPUT...

And running the instances:

euca-run-instances ami-00000002 -k mykey -t m1.large
...OUTPUT...
RESERVATION r-1jj2a80v  cloudproject    default
INSTANCE    i-00000001  ami-00000002            scheduling  mykey (cloudproject, None)  0       m1.tiny2011-08-18T21:06:03Z unknown zone    aki-00000001    ami-00000000
...OUTPUT...
#
euca-describe-instances
...OUTPUT...
RESERVATION r-1jj2a80v  cloudproject    default
INSTANCE    i-00000001  ami-00000002    10.0.0.2    10.0.0.2    building    mykey (cloudproject, SIX)   0   m1.tiny 2011-08-18T21:06:03Z    nova    aki-00000001    ami-00000000
...OUTPUT...
#
euca-describe-instances
...OUTPUT...
RESERVATION r-1jj2a80v  cloudproject    default
INSTANCE    i-00000001  ami-00000002    10.0.0.2    10.0.0.2    running mykey (cloudproject, SIX)   0       m1.tiny 2011-08-18T21:06:03Z    nova    aki-00000001    ami-00000000
...OUTPUT...
#
euca-authorize -P tcp -p 22 default
ssh -i mykey.priv root@10.0.0.2

To add on additional hosts to support more VMs, you only need to install a few of the packages:

apt-get install nova-compute nova-network nova-api

You do need that exact same /etc/nova/nova.conf file though.

Note:
The default install of Glance expects the images that you’ve loaded up to be available on the local file system for every compute node at /var/lib/glance. Either NFS mount this directory from a central machine, or replicate the files underneath it to all your “compute hosts” when you upload a new image to be used in the virtual machines.

Also, the metadata URL needed for UEC images (169.154.169.154) may need help getting forwarded when running on a system with a single NIC. Two potential solutions: A) run nova-api on each of the compute nodes (quick and dirty) or B) specify the –ec2_dmz_host=$HOSTIP, and potentially invoke the command ip link set dev br100 promisc on to turn on promiscuous mode (per https://answers.launchpad.net/nova/+question/152528).

If you’re into OpenStack, come join us on September 28th to celebrate the Diablo release with other stackers in Seattle.

HP Cloud Services has graciously offered up their offices at 701 Pike St, Suite 1100, Seattle WA to host a meetup.

If you’re planning on coming, please stop by the Meetup link at http://www.meetup.com/OpenStack-Seattle/events/33922932/ and RSVP for us so we can get a sense of who might be wandering by chat and say hello!

We have Quantum and Glance shifting to use Github as a repository, new processes (that would be using Gerrit) wrapped around GitHub to allow the project to have a “gated trunk” methodology, and lots of “motion” within the various projects. It’s pretty easy to see on Github, a little harder with launchpad (I just don’t have the tools handy to create the pretty graphs) – you can see the impact graphs for swift, keystone, glance, quantum, and openstack-dashbaord to see what I mean.

The shifting to Gerrit hasn’t been without it’s trials, but is coming along pretty well now. I really wish the GitHub folks had been a bit more amenable to putting in a field that external folks could use to store metadata about a pull request. Several folks from the OpenStack project (including myself) reached out to them about this, all rebuffed (nicely, but still). In fact, one of the suggestions I got back from the github’r support was “Why don’t you set up Gerrit?”

With the changes in core repository, lots of dependencies are shifting as well. Dashboard was broken a bit this week we kicked things around to get the dependencies to match the new locations, I think we’ve got all those pieces worked around now (pull request outstanding for openstack-dashboard). The other piece that really shifted and broke with these changes were the install scripts that we’ve been using to build and work on a developer’s environment. The cloudbuilder team at Rackspace recently created a whole new setup that works very nicely, so I think we’re going to drop our older scripts (based on over-extended versions of Vish’s excellent nova.sh script) and move to using their new “stack.sh“. (And yeah, of course we’ll want to mess with it ourselves, so I’ve forked it…)

That I primarily work in two modes:

• “gettin’ shit done and stay the fuck outa my way”

Which is the rude way of saying I’m hammering out basic frameworks, making sure the theories of how something will go together work, mostly ignoring error conditions and error checking, and testing is manual and just as I need it. I sketch out structures, build blocks on each other, and hopefully end up with my arch/bridge/sofware component/etc – whatever construct I’m making.

• “making it correct”

In which I get totally OCD about understanding how something works – even something I’ve just created, down to creating tests to cover the logic, defining the failure conditions, and picking “just the right names” for objects, method calls, and general analogies.

The first shows more apparent progress – really short term progress – very quickly. But if that’s all you do, then I’m deep in the bug/muck in not very long at all. I’ve worked with a lot of people who don’t toggle between these modes – or I think even recognize them. I think that’s a bit of a tragedy, and something I encourage people to look at introspectively when they’re working for me.

Another realization:

I have far, far less patience with executives and managers than I do with technical folks/line workers.

The explanation into that is a bit more lengthy:

Being a good manager is very, very difficult. (Related reading that I highly recommend: Managing Humans and Behind Closed Doors. Being a good executive (director or whatever up) is even harder. Doing either of those jobs without resorting to being a dick – keeping an objective and fair eye out is damn near impossible. The dirty truth (that most folks in management positions realize pretty darn quickly) is that you get more money and tend to make more progress if you ARE a dick/backstabber/amoral sonuvabitch. Much of the thinking at an executive level is about capital, progress, and rewards – where anything you control is capital, what you output is progress, and what you get from all this is rewards. If you want to know in a general sense why an executive is doing anything, find out what their reward structure is. The track from there is usually abundantly clear.

Oh – and as for that ‘progress’ thing, that’s really naive of me to assert. Here’s the cynical me:

I’ve seen instances where output is status quo because that makes larger business sense: you want to keep some segment of the operation running solidly. Those are usually the pieces of the business that bore the shit out of me, and ironically the ones that are “looking for creative people” – usually to do mind-numbing work.

I’ve also seen directors having the output of one group to just fuck with other groups – not playing to win, but playing to have someone else loose. Why? Because that makes it easier for other resources you control appear to be making progress (even if it’s not the case). I’ve almost always seen this at the director and above level; managers just don’t have enough resources under their control to assign some of them to screwing with other folks, and it’s very, very rare that you’re overtly rewarded for screwing over the other guy. Okay, maybe not in Microsoft if you’re working for the “office” or “windows” franchise components. There’s another company (that I worked for directly – I haven’t worked at Microsoft) in which this pattern is abundant.

The end result of this cynicism (or realism, if you prefer) is that I have little patience with people whom I perceive as not “trying to make things better”. A bit idealistic of me, I know – a place where my idealism doesn’t match at all with reality. A little quote from a favorite movie (Secondhand Lions):

Hub: Sometimes the things that may or may not be true are the things a man needs to believe in the most. That people are basically good; that honor, courage, and virtue mean everything; that power and money, money and power mean nothing; that good always triumphs over evil; and I want you to remember this, that love… true love never dies. You remember that, boy. You remember that. Doesn’t matter if it’s true or not. You see, a man should believe in those things, because those are the things worth believing in.

But there is something about this game that just really annoys the crap out of me. The music theme is a (very!) thinnly veiled rip off from the Pirates of the Carribean theme. Its not the same, but its so close as to be nearly identical in my head. And that theme has been SO overworked in my hearing that I just want it to go away. I get that the harebrained folks were wanting to (I think, positively) riff on the genre, and that’s reasonably cool – but I’d really hoped for music that enhanced the game, rather than a piece which, probably because of my over exposure to it, pisses me off rather than adds. The game itself was pretty new and interesting for the gameplay – I wish they’d taken the trouble to make the music something of their own rather than relying on a “common theme” that most people will recognize to try and garner some emotion towards their game.

The artwork is a bit overdone for my tastes as well, but maybe it appeals to folks a bit more into steampunk than I am. It’s a fun, goofy genre – the artwork in the game reflects that.

The only other downside is that the game has a tendency to crash on me as well – usually right when you’re loading up a scenario to play it. I’m guessing they got a bit overexcited with the memory allocation or something, as restarting the iPad resolves it – makes me think they’re hitting upper limits on memory pressure, and just not dealing well when the iPad OS says “shrink or die!” under the covers. Their FAQ even tells you to do restart the iPad (or reinstall), which is an incredibly lame response. I hope they put a little more effort into tuning the memory use to avoid that crap in the future.

The game is Hnefatafl – not quite an ancestor of chess, but somewhat related. The board is 11×11, and the game is unusual in that it’s an asymmetric game. It’s harder to win as one player than the other.

I first learned about this game through a book (the Mabinogion) that had a reference to a game called Gwyddbwyll (from ‘the dream of rhonabwy’ – see how this crazy stuff all ties together?). Nobody really knows the details of gwyddbwyll, but there’s some reference to say its related to other games from that era – Irish and Welsh, all of which seem to be the asymmetric “save the king” tafl games.

Right now I’m building this on iPad and iPhone, but I’m not sure if I’ll release it under the iPhone – only because the squares are so damned small on that interface. I have a few ideas for how I want to make the interface function, and once I get beyond that I’m going to aim to make it a turn-based multiplayer game using Apple’s GameKit (just an excuse to use the new stuff, course). A multi-player only game seemed kind of limiting though, so I purchased a book on game AI (Artificial Intelligence for games) so that I could make a reasonable single-player game. It’s a darned good book – so we’ll see how I do implementing some of those algorithms in Objective-C. I bought the whole book for Chapter 8 (Board Games), and don’t regret a penny of the purchase.

We have unit testing up and running, and like most projects a few of those tests are more “functional” or “integration” tests than actually unit tests, but really the project’s been pretty darn good about keeping those separate. As it turns out, the Junit framework is truly awesome for fast, quick tests – but when you start to step into functional testing you want just a touch more linearity to the testing. The unit test frameworks that I’ve run into (nosetests, unittest2) don’t have much in their mechanisms that allow you to set up longer, repeated sequences or flows, so it was time to look around for what else might work there. (sidenote: If you know of some mechanisms that do longer, sequential tests and are based around unittest2 or nose, I’d love it hear about it! I’ll freely admit to not having searched every corner there…)

I’ve been watching and reading quite a bit on BDD and the cucumber framework for testing applications, so I thought it was time to try it out. Of course, being more of a python geek than a ruby geek, I’d heard about Lettuce, which is either a straight up port, or incredibly headily inspired by Cucumber.

Getting started with it was incredibly easy. Everything we’re doing is working from a virtualenv (which I highly recommend for development!), so I just added lettuce to the pip-requires list and it was pretty much ready to go.

In setting up lettuce, I probably spent as much time knocking together a quick “functional_tests.py” script as anything – making it responsible for the sequence of testing. The whole point there was to make a single script that could be run, returning an exit code that meant success or failure for the whole set, so it could be plugged into our local instance of Jenkins.

The core of that fabric script is really just simple fabric commands to set up, clean, and push code to a remote node. I ended up adding a script to our local repository to start up services on the far-end for functional testing – testsetup.sh – that runs the various pieces under “screen” – which makes it super easy to see what’s happened when something goes awry.

You’ll notice at the very end of that functional test script, I invoke “lettuce” on the local machine. That takes care of running the actual test sequences. The documentation for lettuce is pretty reasonable, so I recommend walking through that tutorial for a quick understanding. The super-basic gists is that lettuce uses a DSL that you basically create as you go along, and ends up being rather highly customized to your testing setup. You put together files in a “features” directory, and they use “steps.py” in that same directory to do what’s needed. The pattern that I’ve been using (and which seemed to be implied) was to use the “world” object in lettuce as a sort of black-board to get and store data as you walk through the steps in lettuce. There’s deeper integration into Django based projects (my example was using a little Flask based application run with gunicorn).

The key to setting up the functional testing was taking advantage of the file “terrain.py“, which lettuce uses to do global setup, and setup/teardown around steps, scenarios, features, or the whole shebang. Turns out I had several options for how to reach out and walk through the remote application, so I chose to use twill, although I could clearly have used the local test_client() with Flask, or even the web_driver setup from Selenium to drive Firefox.

One of the nicer things that I found while walking through setting this up was that I could write up a feature file and run lettuce, and it gave me really pleasant “this isn’t implemented yet” output that I could use to expand steps.py. For example, I created “example.feature” in the features directory:

and then ran “lettuce” and got the following output:

As you can see, it’s basically giving you the code to add to steps.py to make this work. As I went, I found myself refactoring steps.py to use some of lettuce more advanced mechanisms – regular expressions and grouping steps together in sequences. And to make sure I don’t leave you completely in the dark about setting up and using steps.py, here’s a snippet from my growing file.

This morning, I found a link that adds a new dimension into place: Raspberry Pi. Wikipedia has a good description of the details of the device, and there’s a BBC interview that shows it and gets into a nice overview.

It’s the price point of this device with the capabilities that changes the game a bit. It’s a computer the size of a tin of mints, with a processor capable of easily running higher level languages – python, ruby, java, whatever. There’s a lot out there now (Arduino) that is a roughly similar cost and form factor, but programming it requires working in C or C-like languages and some tools to get it all together.

Raspberry Pi BBC Interview:

While the interview and foundation talk to making low-cost computers available to kids, I think this kind of device has a much broader potential in personal computing. The tech specs of this device make it very similar to the computing power available in the iPad or iPhones. And while the first generation of this device is missing some things I’d like to see default (a networking port), it’s easily added with a USB hub. The device is also incredibly low power – so I could imagine a shoe-box sized container of these running a 10x the amount of compute, and generating less heat (using less power) than your average dell desktop.

For an always-on application, I want something that’s not drawing a lot of power and could conceivably run unattended in the closet all day and night. This enables it at a new, lower price point (~$25 compared to a Hawkboard SBC at ~$90). Now that’s cheaper (for the moment), than running an instance up on a cloud server…

As I got into the game, noodling on the design and how the interaction would work, I came to the realization that I really would like to have a reasonable single player game as well, and that I didn’t have a clue about how to do board-game AI to make a good game experience.

After looking around through a pile of game AI books, I came across “Artificial Intelligence for Games, Second Edition“. Heavy book, hardback – some $80 if you pick it up outside of Amazon’s discount mechanisms. It looked good after a glance-through and minimal reading, so I took it home to dig in depth. I’m really glad I did!

This book does an excellent job of describing the AI algorithms in pseudo good and good ole english, as opposed to many which seem to be littered with mostly un-parse-able fragments of C++. They reference a number of modern games, but rather than catalog “these used this type of AI, the others didn’t” (which I saw in a number of books), they took some time to explain the tradeoffs in choosing the AI mechanism in the game, even if they didn’t have the “why” of the game designer to pull from.

I liked it enough to put up a review on Amazon, and hey – I’m writing this too. If you’re looking for a good overview of AI algorithms and how they work, I recommend the book.

Quick recap: The service framework in nova is set up to make it easy to write your own services that interact with any other nova services (such as nova-network, nova-scheduler, etc). The service framework includes all the pieces to communicate with these other services (using a module called rpc.py that abstracts away the communications), a database connection for looking up data from the nova persistence store, and leaves the rest to you.

The service module is expecting to be told a manager class to load, and the framework will use that to do what it needs. There are only two required methods to overwrite:

• init_host
• periodic_tasks

There are two kinds of services – I’m going to focus on the stand-alone (not WSGI) service that expects to communicate and respond entirely through the message queue system in Nova.

So how does this service critter work? Well, when it’s initialized, it gets a number of attributes assigned to it. This is typically done from a class method on service.py:

from nova.service import Service
my_service = Service.create()

The create() method has a number of parameters:

• host – a string with the host this service is running on
• binary – a string with the binary name of this program
• topic – a string with a subset of the binary name, used to set up a message exchange in AMQP
• manager – a string with the class name to be loaded to do the ‘work’
• report_interval – an interval set from Flags, triggering a regular reporting loop
• periodic_interval – an interval set from Flags, triggering a periodic loop to do repeating tasks

If you don’t provide any of them, all these values get populated with defaults and from configuration detail (the flags) in the service.serve() method. Once serve() gets everything configured up, it calls .start() on each service to kick it into gear.

start() is where things really get moving. This is where the service manager class gets loaded, registered with the nova database (if it isn’t already), and the RPC mechanisms get spun up with Eventlet greenthreads to accept messages to this service. The topic (which is the name of the binary, minus any “nova-” in front of it) is used as an exchange. Through this mechanism, any service can talk to any other service (or set of services). Here’s how that works:

rpc.py has two methods: call() and cast() that do all the heavy lifting. When you use these, they take a “context” (i.e. authorization for who’s doing the call), a topic (the name of the service you’re calling), and a message. call() sends this message and waits for a response. cast() sends the message entirely asynchronously, not expecting a response.

The message is a JSON structure – a dictionary, and it’s expected that the dictionary will have a key “method” and another key “args”. method is expected to be a string, and args is expected to be another dictionary. The rpc module does the work of using that method string to look up and invoke the method on your manager.

An example of this operating is right in the code. In the nova-network API, there’s an rpc.cast():

rpc.cast(context,
         self.db.queue_get_for(context, FLAGS.network_topic, host),
         {'method': 'associate_floating_ip',
          'args': {'floating_address': floating_ip['address'],
                   'fixed_address': fixed_ip['address']}})

Through the service framework and the RPC mechanisms, this is calling associate_floating_ip() on the network service manager class.

The other nifty thing about service is that it’s keeping and managing a number of greenthreads from Eventlet to do it’s work. The basic bits are all encapsulated in that rpc.py mechanism – when it sets up connections to the message queue service to receive communications, that starts a greenthread rolling to watch out for, pull, and process any messages inbound. The two periodic interval pieces are also spun up on their own greenthreads – looping every “interval” (specified by the flags –report_interval and –periodic_interval, set at 10 and 60 seconds by default respectively). These run continuously until the service is terminated.

This little post is to document a benchmark that I did for an internal use case, in the hopes that it’ll be helpful for others. As a benchmark, I wasn’t attempting to fully characterize the performance of a specific system – I just wanted a pole against which to measure changes in the environment or underlying infrastructure. I was curious what the performance was of Celery, using RabbitMQ. The tests I ran were pretty much straight sample code of the project (a simple “add” task) and a client making multiple requests and combining the results with the veritable python “timeit“.

The code

All the code for this test (and more, I got excited…) is stashed up on Github: https://github.com/heckj/openstack-benchmarks/tree/CeleryBenchmark. Really, the parts you’re likely to be interested in is the worker class: tasks.py, the configuration: celeryconfig.py, and the actual benchmarking code: celery-benchmark.py.

And before you ask, no – the OpenStack project doesn’t currently use Celery – in fact they use Carrot right now. I’m just intending to add on more benchmarks and profile tools into this codebase around the OpenStack project in the future.

The config

The configuration was held constant – a stock Ubuntu 10.10 server with all the various dependencies installed. Since I’m sure someone will want to know about the versions:

• rabbitmq-server 1.8.0-1ubuntu2
• python2.6 2.6.6-5ubuntu1
• python-amqplib 0.6.1-1
• celery 2.2.7
• kombu 1.1.6
• anyjson 0.3.1

The host was a Shuttle PC, 8GB ram, Core 2 Duo processors. The host was never heavily burdened by the processing that took place (load < 1.0, no significant swaping). The benchmarking was done on the same host as RabbitMQ to remove any network latency effects.

The results

.

I was totally abusing MS Excel’s “stock graph” to show variability in the results (of which there wasn’t a hell of a lot). In the graph, the thin line represents the range (min to max) and the thicker box in the middle is standard deviation +/- the average result. The gist – the round trip time was pretty much straight up at 160ms per requests, and that sampled over 1,000,000 requests. The image above shows a portion of the sequence. The relevant code:

from benchmark.celerybench.tasks import add
 
result = add.apply_async(args=[4, 4]_
result.get()

(As I mentioned earlier, you can see the whole code on github).

I did more tests, but I need to keep some of those to myself, as they’re testing variations of configurations for my job.

Random side notes

I hadn’t done anything in depth with Celery before. I’d heard about it from friends, and in the community in general. The author pinged me a couple of times with help as well. Overall, I found the Celery setup to be incredibly easy to use and a very straightforward API (always nice). There were lots of options available, but everything was set with very usable defaults from the start. I’m totally looking forward to using Celery in some projects, as well as taking advantage of Kombu – a drop-in/compatibility layer for Carrot.

Update:

Ask mentioned some suggestions for optimizing in twitter – seemed a good place to put them. Try:

• CELERYD_PREFETCH_MULTIPLIER=0
• CELERY_DISABLE_RATE_LIMITS=True
• and BROKER_TRANSPORT=”librabbitmq” to use the pylibrabbitmq C library

Before I dive into the gritty details, it’s worth getting a high level overview so that some of this (hopefully) makes sense. OpenStack’s service architecture is made up of services that all talk with each other to get things done. nova-network, nova-scheduler, etc. There’s a lot of underpinning in the nova codebase to make those services relatively easy to write and work together – I was mostly curious about how they passed messages back and forth. As I dove in, the two pieces that stood out as needing to be understood first were the unified service framework in nova and configuration using flags (which it heavily depends upon).

Configuration – using the flags

The configuration for nova services – global or specific to a service – are all done with configuration files that can be over-ridden on the command line, taking advantage of python gflags to make it all work nice. I didn’t know much about the flags system, so I dug around in the python-gflags project. They have the documentation for how to use gFlags in the code itself: http://python-gflags.googlecode.com/svn/trunk/gflags.py.

To summarize it up:

Python modules in the codebase can define and use flags, and there is a general nova flags file that holds the cross-service (common) configuration settings. Nova defaults to looking for it’s configuration in a nova.conf file in the local directory. Failing that it looks for the nova.conf file in /etc/nova/nova.conf. Where it looks for the configuration file can be overridden (typically on the command line) by (--flagfile) and a location to a config file. The code that makes this happen nova.utils.default_flagfile().

To use the configuration from within code, you typically instantiate the global flags, add any flag definitions (with default values) that you care to add, and then use ‘em! Here’s a code snippet example:

from nova import flags
# import the nova wrapper around python gflags
#  .. there's some interesting wrapping for taking in arguments
#     and passing along extras values to your code
#  .. and it's where the global flag definitions reside
FLAGS = flags.FLAGS
# get the global instance
#  .. this attempts to read the /etc/nova/nova.conf for flags
# 
# You can define an additional flag here if you needed to...
flags.DEFINE_string('my_flag', 'default_value', \
        'human readable description of your flag')
# there's also flags.DEFINE_bool, flags.DEFINE_integer and more...
# 
# And then you can use the flags
#  .. the flags you defined show up as attributes 
#     on that FLAGS object
print FLAGS.my_ip

If you were happening to write a script that took in flags and worked with them for a command-line script, you might do something like:

from nova import flags
form nova import utils
utils.default_flagfile()
flags.FLAGS(sys.argv)
GLOBAL_FLAGS = flags.FLAGS
# ... and on to the rest of your code

There is some good end-user documentation on how to find the flags. The gist is – if you want to know what flags are there, the easiest way is to hand in the flag “–help” or “–shorthelp” from the command line. That is how the gFlags library is set up to tell you about the flags.

Update:

After a little digging down a side passage, I noticed that service.py had some debugging code in it that iterated through all the set flags. You iterate directly on FLAGS (treating it as an iterable thing) and use FLAGS.get() to retrieve the set values.

    logging.debug(_('Full set of FLAGS:'))
    for flag in FLAGS:
        flag_get = FLAGS.get(flag, None)
        logging.debug('%(flag)s : %(flag_get)s' % locals())

Services

There are two types of services in Nova: system services and web services. The code to use and launch them is basically the same, and Nova has this all bundled into a general service architecture and code base. The reason that configuration is so important is that the nova service framework has a convention of knowing how to run a service based on flags from the framework.

Here’s a bit of example code of a service to illustrate what I’m talking about.

nova-exampleservice:

import eventlet
eventlet.monkey_patch()
 
import sys
from nova import flags
from nova import service
from nova import utils
 
if __name__ == '__main__':
    utils.default_flagfile()
    flags.FLAGS(sys.argv)
    service.serve()
    service.wait()

The convention starts off by using the name of the script invoked – in this case “nova-exampleservice”. The scripts in bin/ (like nova-network) use this mechanism. This convention can be overridden, of course, but it does make things pretty straightforward once you know the convention. The key to this convention is that the code in nova.service looks in the configuration for a class to instantiate (expected to be a subclass of nova.manager.Manager) named after the service that was just invoked. (this convention is in code under the nova.service.create() method)

For our example of nova-exampleservice, the service is going to look in the configuration for exampleservice_manager, expecting the value to be a class that it can load that will be a subclass of nova.manager.Manager and will be responsible for running the service.

This code is invoked from service.serve() from our example above. Again, it looks for the flag “exampleservice_manager” and try to load that class to do the work.

An updated example that sets a default manager that will attempt to load the class mymodule.exampleservice.ExampleServiceManager by default:

nova-exampleservice:

import eventlet
eventlet.monkey_patch()
 
import sys
from nova import flags
from nova import service
from nova import utils
 
if __name__ == '__main__':
    utils.default_flagfile()
    flags.FLAGS(sys.argv)
    flags.DEFINE_string('exampleservice_manager',
            'mymodule.exampleservice.ExampleServiceManager',
            'Default manager for the nova-exampleservice')
    service.serve()
    service.wait()

The manager class has two classes that you override to get your stuff done:

• init_host
• periodic_tasks

There are also some conventions around adding methods to your manager and invoking them using the service framework’s RPC mechanism, which I’ll dig into with another post.

Ref: Nova Developer Documentation
Ref: OpenStack Compute (Nova) Administration Manual
Ref: Openstack Wiki: Unified Service Architecture

April started showing me a need to change things up. There’s not too much to say about the whys and wherefores of deciding to leave Disney that are relevant (or appropriate) for a public forum like my blog. Suffice to say the impetus hit. In late April I attended the second OpenStack design summit in Santa Clara, and then a vacation up to Alaska to settle down and reset. At the end of that, I was ready and took on a new position at a new company where I can combine a number of pieces that Im passionate about: DevOps and OpenStack. I guess that misses out a bit on the Mac/iPhone development, but it would be an immensely rare gig that could cover all three of those.

I can’t yet speak of the new company, save to say that it is exciting, challenging, and I really enjoy the crew that I’m working with. We are all top-secret and stealth startup at the moment… although the word should be coming out at OSCON 2011 with some pretty amazing announcements. In the mean time, I’m very happy to be back in the middle of actively working on open source projects – most specifically OpenStack.

It all revolves around the development toolchain, and the base components to cobble together full desktop-UI quality applications. This happens to overlap extensively with the server side world too – but the details started really becoming clear after digging a bit into Sencha, SproutCore, and JavascriptMVC. And it’s Javascript MVC that stands out to me.

If you look at the home page for the site, it presents itself as a collection of tools and mechanisms to do a full-bore standard application:

• core generators
• dependency management
• build scripts
• testing
• templating
• code cleaning and linting

and the list goes on. All of those components in javascript, for javascript. It’s still incredibly fragmented as a language, but the parallel efforts moving forward to drive automated testing, including unit testing on the server side. Node has it’s own built-in stuff, but there’s also a lot of effort to bring QUnit from the JQuery world into that same server side world.

Python and Ruby still have a dominant hold on the server side development, but I wonder for how much longer…

I had a previous pedometer (kept running out of battery), but I’m finding that after a week with the fitbit, the big difference is the dashboard that it automatically keeps and maintains. You can sign up for the dashboard even without having one – although without the fitbit what it really amounts to is an online journal for tracking food intake, weight, and the other kinds of common measurements. With the fitbit, and some of the data going in on a regular basis, it’s really become a bit more of a personal dashboard.

I’ve been trying to track food intake (i.e. calories) and weight manually. That’s going reasonably well, although I find it rather tricky to track calories for some of the food I eat – there isn’t an easy lookup. Ironically, getting breakfast at a health club cafe last weekend was the hardest – they couldn’t tell me anything about the nutritional value of what I ate. Ironic that it was a health club too…

The fitbit cost $99, and I frankly bought it on a whim because I saw other people saying great things about it. I wanted to check it – having liked having a pedometer before – and I’ve got to say, this particular gadget buy has really worked out. I’m now much more aware of how much I’m walking (and how much I’m not walking), and it’s even doing some numbers like “how many calories I’ve burned” so that making attempts to track how much I’m taking in with food have something nice to balance against.

I’m pretty happy with the first week – and I’d recommend it to anyone wanting to get a track on their day to day activity. Now we’ll see what happens after a few weeks…

I’ve been following SproutCore for a while, sort of thinking about that as an interesting mechanism, and then more recently I saw the announcements about the jQueryMobile framework setup. Having done some iPhone and iPad development, the common metaphors of list views, tableviews, etc. felt very familiar. jQueryMobile isn’t all that to be perfectly honest, but the basic design feel is reasonably close.

I made a branch on Eyes and started laying it out – and I’ve been going gangbusters since. Laying in the constraints of the framework has really made the choices much easier to work out. Not that I haven’t run down some dead ends and had to back out, but I feel like I have better sense of how the pages flow and work together.

And best of all, I have something that I’m comfortable will work on mobile devices, while not looking like crap on a larger view.

I’m still learning the various tidbits of the framework CSS stylings, and I’m already seeing some elements where I’m going to want to dig deeper. The only real struggle I’ve had is working with Ajax based forms submissions and debugging them when I bork it up. Doing so much javascript work and debugging is still pretty unfamiliar to me. I’m ending up with a lot of print(“..”) debugging on the back end, and alert(…) debugging in the javascript itself.

All in all, I’m very pleased with the effects and results of using jQueryMobile. I do rather wish I hadn’t left my jQuery book at work for the holiday weekend though – I’m certainly not a jQuery expect, and it would have come in very handy more than once already. Thank god for excellent online documentation.

So here’s my annoyance – all of these systems, even the open source ones, are really set up to be managed by people, not other systems. They’re not built with API’s to be able to create, update, check on, and delete monitors. Some of them come darned close – Zenoss, SiteScope, etc. Others have this sort of worked around from the back side – i.e. Puppet or Chef recipes that generate Nagios configuration files.

So a year ago I decided that I could probably put something together that does the same basics, but has API’s built in from scratch. I did a lot of noodling on the idea, scratching ideas in notebooks and such, before I really kicked things off. Then I decided to see what I could do to wrap up a new system. I built it around Nagios plugins – mostly because there’s a lot of them and they do some pretty good stuff right off the bat. And that’s all open source as well. And I wrapped that around with a web application based on Django, because – well – I know Django reasonably well. The result is a basic system that I’m calling “Eyes”.

The whole source, from the very beginning, is on Bitbucket at http://bitbucket.org/heckj/eyes. And after poking at it on and off for nearly 12 months, I decided to wrap this up a bit and get it out there – so as of today I have my first release on PyPi, with documentation. And I created a mailing list – eyes-monitoring – on Google Groups to have some place to talk around it.

I very intentionally included documentation for the project from the very beginning, embedded with the project. I also very intentionally wrapped a large number of tests around the project, and have been checking and watching test coverage as the project grows and changes. The state today? Basically functional, but you have to know the innards to do much more with it.

Today it doesn’t have anything related to alerting or escalation within it. It doesn’t have much of anything around a user interface, and the data is stored all in RRD files. It’s at the point now where the basic framework is in place – and there’s a lot more to go: design of the user interface, blocking in alternate mechanisms to do monitoring, and fleshing out higher level features like notifications and alerts – both via email and alternative mechanisms (like web-hooks, or XMPP event streams).

What it is ready for is outside eyes. If you’re interested in contributing, or even sharing some ideas about what to do with Eyes, I’d love to hear from you. Fork the code and try it out, or join the mailing list (http://groups.google.com/group/eyes-monitoring) and share some of your ideas or thoughts.

I’m saying all this after a week with the OpenStack guys – both in design sessions and just chillin’ out. Focused, intelligent, demanding conversations scattered through the week with an amazing “no-ego” attitude presenting itself. Not that there weren’t some good ole technical “best way to do it” or “which is better” fights, but given the breadth of this project and the open nature with vendors lurking all around the corners – well, frankly I expected a lot more “special interest” to be clearly showing itself. Everyone at that conference was interested in making OpenStack better at every turn.

250 people, 12 countries, 90 companies/organizations – all that after 3 months from being publicly announced. And they’re going it without any prior structure – building up an OpenStack foundation, doing all the legal and community building, right from scratch. And yeah – that’s showing right now.

The first thing I see that will provide the biggest gains:

• “How do we all work together?”

Some of the best sessions were around “What does the status X of bugs mean” and talking through the development and release process. At this point I’m convinced the core folks are reasonably comfortable with LaunchPad (the platform the system is hosted on) – and being at the conference really taught me a great deal about how OpenStack is effectively using it. Prior, it wasn’t comfortable or familiar to me at all. The object store and compute (swift and nova, respectively) core groups are really quite separate teams, all trying to figure out how to get some common ground in re-using code, libraries, and even setting up documentation.

The second:

• “Show me it’s workin’, again and again”

OpenStack is quickly heading to be the kernel or core of a platform. You could see it in the twinkle of Eucalyptus’ eye when they talked about Swift (the object store), or chatting with the folks from Scalr or RightScale. The whole system is being built with API in mind from the ground up, and while there is some pretty good unit testing in play and continuous integration, it was clear that installing this sucker was a PITA – and the documentation to really pull that all together starting coming together in the documentation sprint and install fest at the summit. One of the “blueprints” of the design summit (i.e. “Things we want to do, and how we want to do it for the next release”) is to get some fully automated integration testing as well as track the metrics on how the system is operating. There were a lot of folks that have some cross over into the Drizzle project, and the ideas of running and tracking benchmark data on every revision is darned power.

Add to that the benefits of a constant flow of functional testing against a couple of pre-defined clusters of both compute and object store, and you have a powerful engine to make sure trouble is spotted early and can be resolved quickly.

The third:

• “How’s this thing tick?”

One of the admitted weak points is that some small, damned effective core teams have done most of the work – and if you want to understand the system, well… you’ve just got to read the code. That is a huge investment – and frankly a barrier to entry into the project that can be avoided with some effort towards docs and discussion. Again, great progress was made there (I learned what the “project” concept was in Nova at the summit) – but the interactions between components, what the components are responsible for, and what they’re *not* responsible for, are all kind of tricky to learn right now.

This extends down into digging into the code, where docstrings could be better (and are getting better!) so that if you wanted to go help with something specific, you didn’t have to grok a broad codebase to get a handle on what the impacts are of the changes you’ll need to make.

And the last thing I’ll throw in here:

• “What OpenStack isn’t, or won’t do…”

The project is still in a lot of flux. There were some great components that were shown off at the summit that ride over the top of the infrastructure, or work with it through APIs. Should those be a part of OpenStack, or on the side? Some service providers were very interested in more platform-kind of elements – a common logging infrastructure, a common authentication, ID, and authorization infrastructure. Should that be a part, or on the side? How tightly or loosely do we want to couple some of these elements? The philosophy is there and forming up, but the real truth of it all will be over the next 6-12 months of the project when decisions are made, reviewed, and a core forms out of it. There have been a few architectural decisions made early: “Don’t mandate anything in the client”, “If a feature would restrict scale, it MUST be optional”, etc. that I absolutely applaud. I think it will form up more as projects apply to join the OpenStack umbrella and either make it or don’t. It will become clear what’s common, and what isn’t, pretty darn quickly.

I’m pumped about this project, the people, and it’s future. The core openstacker’s have clearly been driving up a steep hill to get to where they can level out a bit and move into more of a marathon mode. Really, it feels like we’re nearly at the top of that first hill.

The way I’m thinking about it, GAE probably isn’t the right thing. They don’t easily let you run repeated cron jobs, making an agent style application (i.e. proactive, rather than reactive) not quite a good fit. So what would be? A VM. Sure, you could slice it down farther and get a slicehost style thing – they’re excellent, but many of them have restrictions about “long running processes” – and that is exactly what I want.

So what’s it take to run a VM? There’s an interesting question. The cheapest I found was Rackspace – that offered a 256MB Linux hosting solution for $10/mo. Linode starts at 512MB for $19.95. Of course you need to at least make a vague attempt to compare to an EC2 instance. The always on really bites you there – the cheapest I could get my numbers was around $6/mo (a single micro instance, 1 yr commitment – minimal to no bandwidth).

I have to wonder – how well would something like that be received? Say you could offer a private agent/proxy service for $10/mo. I suspect you could get it cheaper – the compute needed for this kind of service would seem to smaller than the minimum you can purchase as a slice of compute. But people aren’t generally buying like this – at least not today.

I think most folks who have a desire for something like this would be more likely to have a desktop computer at home they could dedicate to leaving on. That’s assuming they have an internet service to their house too – so the real cost is more than the hosted compute, but since people often have it anyway… I’m guessing they’d be more likely to want to just use it.

The closest to paying as you go for an application gets to the Platform as a Service – Google App Engine or Microsoft Azure. Frankly neither of which I particularly want because they both represent a sort of technology lock in. In development now is OpenStack – Rackspace & NASA doing some of that base line commoditization work to drive down the costs of running small VM slices. We’re already on the sweet side of Moore’s Law for the commoditization of compute resources – it’s just a few more years until it’s a complete no brainer.