I’m a huge fan of CloudEnvy, so I’ve created a public github repository with the envy configuration and setup scripts to be able to set up a VM and completely build out all the existing documentation in roughly 20-25 minutes.

First, we install cloudenvy. It’s a python module, so it’s really easy to install with pip. My recommended installation process:

pip install -U cloudenvy

If you’re working on a mac laptop (like I do), you may need to use

sudo pip install -U cloudenvy

Once cloudenvy is installed, you need to set up the credentials to your handy-dandy local OpenStack cloud (y’all have one of those, don’t you?). For cloudenvy, you create a file in your home directory named .cloudenvy akin to this:

cloudenvy:
  clouds:
    cloud01:
      os_username: username
      os_password: password
      os_tenant_name: tenant_name
      os_auth_url: http://keystone.example.com:5000/v2.0/

Obviously, put in the proper values for your cloud.

Now you just need to clone the doctools envyfile setup, switch to that directory, and kick off Envy!

git clone https://github.com/heckj/envyfile-openstack-docs.git
cd envyfile-openstack-docs
envy up

20-25 minutes later, you’ll have a virtual machine running with all the tooling installed, run-through, and the output generated for all the documentation in the openstack manuals repository. The envyfile puts all this into your virtual machine at

~/src/openstack-manuals

To get there, you can use the command envy ssh to connect to the machine and do what you need.

For more on the how-to with contributing to OpenStack documentation, check out the wiki page https://wiki.openstack.org/wiki/Documentation/HowTo.

Looking at what glanceclient did, and had to do, I got really annoyed and wanted the client to have a much easier to use interface. At the same time, I was also digging around trying to allow the keystoneclient CLI to accept and use an override for the endpoint from the command line. Turns out the various mechanations to make the original client setup work with a system with two distinct URL endpoints was quite a mess under the covers, and that mess just propagated through to anyone trying to use the library.

We just landed some new code updates with keystoneclient to make it much easier to use. So this little article is intended to be a quick guide to using the python keystoneclient library and some of it’s new features. While we’re getting v3 API support installed, we’re still very actively using v2 apis, so we’ll use v2 API examples throughout.

The first is just getting a client object established.


>>> from keystoneclient.v2_0 import client
>>> help(client)


We’ve expanded the documentation extensively to make it easier to use the library. The base client is still working from httplib2 – I didn’t rage-change it into the requests library (although it was damned close).

There’s a couple of common things that you’ll want to do with initializing the client. The first is to authorize the client with the bootstrapping pieces so you can use it to configure keystone. In general, I’m sort of expecting this to be done mostly from the CLI, but you can also do it from the python code directly. To use this setup, you’ll need to initialize the client with two pieces of data:

• token
• endpoint

Token is what you’ll have configured in your keystone.conf file under admin_token and endpoint is the URL to your keystone service. If you were using devstack, it would be http://localhost:35357/v2.0

A bit of example code (making up the admin_token)

from keystone client.v2_0 import client

adminclient = client.Client(token='9fc31e32f61e78f114a40999fbf594c2',
                            endpoint='http://localhost:35357/v2.0')

Now at this point, you’ll have an instance of the client, and can start interacting with all the internal structures in keystone. adminclient.tenants.list() for example.

You may have spotted the authenticate() method on the client. If you’re using the token/endpoint setup, you do not want to call this method. When you’re using the admin_token setup, you don’t
have a full authorization token as retrieved from keystone, you’re short-cutting the system. This mode is really only intended to be used to bootstrap in projects, users, etc. Once you’ve done that, you’re better using the username/password setup with the client.

To do that, you minimally need to know the username, the password, and the “public” endpoint of Keystone. With the v2 API, the public and administrative endpoints are separate. With devstack, the example public API endpoint is http://localhost:5000/v2.0.

A bit of an example:

from keystoneclient.v2_0 import client
kc = client.Client(username='heckj', password='e2112EFFd3ff',
                   auth_url='http://localhost:5000/v2.0')

At this point, the client has been initialized, and as a default it will immediately attempt to authenticate() to the endpoint, so it already has some authorization data. With the updated keystoneclient library, this authorization info is stashed into an attribute “auth_ref”. You can check out the code in more detail – the class is keystoneclient.access.AccessInfo, and this represents the token that we retrieved after calling authenticate() to auth against keystone.

With only providing a username and password, the token is really only useful for about two things – getting a list of clients that this user can authorize to (getting a ‘scoped’ token – where the token represents authorization to a project), and then retrieving that token.

>>> kc.username
'heckj'
>>> kc.auth_ref
{u'token': {u'expires': u'2012-11-12T23:28:58Z', u'id': u'97913f8839634946afab2897ac19908d'}, u'serviceCatalog': {}, u'user': {u'username': u'heckj', u'roles_links': [], u'id': u'c8d112a0932a454097dfba0f3b598bdc', u'roles': [], u'name': u'heckj'}}
>>> kc.auth_ref.scoped
False
>>> kc.tenants.list()
[<Tenant {u'id': u'7dbf826d086c4580a28cf860a6d13046', u'enabled': True, u'description': u'', u'name': u'heckj-project'}>]
>>> kc.authenticate(tenant_name='heckj-project')
True
>>> kc.auth_ref.scoped
True
>>> kc.auth_ref
{u'token': {u'expires': u'2012-11-12T23:37:10Z', u'id': u'6d811d7c39034813b6cab2ad083cdf3e', u'tenant': {u'id': u'7dbf826d086c4580a28cf860a6d13046', u'enabled': True, u'description': u'', u'name': u'heckj-project'}}, u'serviceCatalog': [{u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://localhost:8776/v1/7dbf826d086c4580a28cf860a6d13046', u'region': u'RegionOne', u'internalURL': u'http://localhost:8776/v1/7dbf826d086c4580a28cf860a6d13046', u'publicURL': u'http://localhost:8776/v1/7dbf826d086c4580a28cf860a6d13046'}], u'type': u'volume', u'name': u'Volume Service'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://localhost:9292/v1', u'region': u'RegionOne', u'internalURL': u'http://localhost:9292/v1', u'publicURL': u'http://localhost:9292/v1'}], u'type': u'image', u'name': u'Image Service'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://localhost:8774/v2/7dbf826d086c4580a28cf860a6d13046', u'region': u'RegionOne', u'internalURL': u'http://localhost:8774/v2/7dbf826d086c4580a28cf860a6d13046', u'publicURL': u'http://localhost:8774/v2/7dbf826d086c4580a28cf860a6d13046'}], u'type': u'compute', u'name': u'Compute Service'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://localhost:8773/services/Admin', u'region': u'RegionOne', u'internalURL': u'http://localhost:8773/services/Cloud', u'publicURL': u'http://localhost:8773/services/Cloud'}], u'type': u'ec2', u'name': u'EC2 Service'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://localhost:35357/v2.0', u'region': u'RegionOne', u'internalURL': u'http://localhost:5000/v2.0', u'publicURL': u'http://localhost:5000/v2.0'}], u'type': u'identity', u'name': u'Identity Service'}], u'user': {u'username': u'heckj', u'roles_links': [], u'id': u'c8d112a0932a454097dfba0f3b598bdc', u'roles': [{u'name': u'Member'}], u'name': u'heckj'}, u'metadata': {u'is_admin': 0, u'roles': [u'08ccc339c0074a548104b9050bdf9492']}}

You might have noticed that you can now call authenticate() on the client and just pass in values that are missing from previous authenticate() calls, or you can switch them out entirely. You can change the username, password, project, etc – anything that you’d otherwise normally initialize with the client to do what you need.

Here is an overview of what’s looking to happen in Keystone over the grizzly release cycle.

From the summit, we had the state of the project slides, which might be of interest: http://www.slideshare.net/ccjoe/oct-2012-state-of-project-keystone

Since then, we’ve been working on fleshing out more details around those initial discussions, and we’ve been correlating who’s working on what to get an overview of what’s coming up for Keystone. If you’re into reading raw notes, take a look at https://etherpad.openstack.org/keystone-grizzly-plans.

For those looking for more of a tl;dr:

grizzly-1 plans:
* merging in V3 API work – “tech preview”
https://blueprints.launchpad.net/keystone/+spec/implement-v3-core-api

* move auth_token middleware to keystoneclient repo
https://blueprints.launchpad.net/keystone/+spec/authtoken-to-keystoneclient-repo

* AD LDAP extensions
https://blueprints.launchpad.net/keystone/+spec/ad-ldap-identity-backend

* enabling policy & RBAC access for V3 API
https://blueprints.launchpad.net/keystone/+spec/rbac-keystone-api

grizzly-2 plans:
* pre-authenticated token
https://blueprints.launchpad.net/keystone/+spec/pre-auth

* plugable authentication handlers
https://blueprints.launchpad.net/keystone/+spec/pluggable-identity-authentication-handlers

* consolidated policy documentation/recommendations
https://blueprints.launchpad.net/keystone/+spec/document-deployment-suggestions-policy

* PKI future work
https://blueprints.launchpad.net/keystone/+spec/delegation
– starting into delegation, signing of tokens
– annotations on signing for authorization

grizzly-3 plans:
* delegation
https://blueprints.launchpad.net/keystone/+spec/delegation

* multifactor authN
https://blueprints.launchpad.net/keystone/+spec/multi-factor-authn

Much of the work and desires around Delegation has yet to be fully defined and nailed down, and relies on a lot of additions in making PKI based tokens a stable, solid, default mechanism. I’m sure there will be some redirection once we get a few weeks down the road and see what’s happening with the V3 API rollout and PKI token extensions to support delegation, pre-auth, and so forth.

CloudEnvy (https://github.com/cloudenvy/cloudenvy) is what started that cascade. Brian Waldon started it some time ago as a script that emulated the ease of spinning up VM’s with vagrant, except wrapped over the OpenStack clients. I always wanted to like Vagrant, but it never really synced for me. I think mostly because I was in never-ending kernel panic hell with virtualbox. CloudEnvy is a different story.

I think the most interesting illustration of CloudEnvy is using it to spin up an instance in a cloud, and then run devstack in that instance.

CloudEnvy relies on a starter of two files – one that’s specific to the project (DevStack in this case): Envyfile, and one for your personal cloud configuration (~/.cloudenvy).

Here’s my .cloudenvy file (with the hostname and password redacted):

And the Envyfile I use with devstack:

You’ll notice that the Envyfile references a script I named “cloudenvy-setup.sh” – this is the basic script that cloudenvy uploads to the instance it creates to automatically provision things up. You could easily replace this with Puppet, Chef, or whatever it is you like to configure VMs in your world.

Here’s what I’m doing:

(all three of these files are in the gist https://gist.github.com/3969250)

The Envyfile also refers to a image_name. I’m using a stock UEC precise image that I uploaded to our instance of OpenStack. Pretty shortly, CloudEnvy will be replacing “image_name” with just “image”, and they recommend that you use an image ID (guaranteed uniqueness) over a name. For my immediate use, the name works pretty well.

Once this is all in place:


envy up


Creates the instance, assigns it a floating IP address, SSH’s into the instance, uploads the provision script, and starts cranking on the provisioning. 763 seconds later, a fully operational devstack in an instance running on OpenStack.


envy ssh


Gets you in, lets you do what you want.


envy list


Shows you the instance(s) you have running.

There’s more, a lot more, but hopefully this is sufficient to get you started.

The design sessions are definitely less about actually hacking code than they were a year ago, offset though with the increasing diversity of backgrounds and interests participating in the sessions. The core team developers joined me through-out Thursday and drove most of the discussions, with fantastic input from David Chadwick, Khaja, Ryan Lane. There were way more people in the sessions than that, but to me these three represent a set of fresh inputs from folks with a deep academic background, previous experience building identity systems, and active operator points of view. They and the the previous contributors provided tremendous feedback, asked great questions, and set the stage for a lot of interesting ideas.

This year all the project technical leads gave a “state of the project” overview, but we did that on Tuesday – so like John Griffith (the project technical lead for cinder), I was doing the ‘state of the project’ routine prior to getting the feedback and doing the brainstorming in the sessions. The slides from that presentation are online at http://www.slideshare.net/ccjoe/oct-2012-state-of-project-keystone if you’re interested. The coordinators all video-taped those segments, as I understand it, they should be appearing on the OpenStack channel in Youtube in the next couple of days.

There was also a very active session led by Gabriel Hurley seeking to drive more continuity into the OpenStack APIs, and a matching session by Doug Hellman and Dean Troyer for the OpenStack CLIs. The continued focus on bringing in new ideas while keeping the interfaces consistent and clear is a great sign for the project overall, and I was pleased to see a large number of like minded folks wanting to continue to move things forward in those areas.

This was also the first summit under the auspices of the OpenStack Foundation – they all met for an extended period of time early in the conference, and the Technical Committee managed to pull of a first all-in-person meeting over dinner and scattered conversation Tuesday evening.

And not at all related to any core projects or overall effort: check out the very creative riff on the OpenStack theme Dope’n'Stack, (video on YouTube as well). Gabriel and Erik were working their tails off prior to and during the conference to pull this off, culminating in a great presentation Wednesday evening at Piston’s party. (I was disappointed that Gabriel lost the mohawk for the summit, but he said he was sick of wearing it after three weeks).

We rolled RC1 for Keystone Folsom release out the door this past week, and at this point I think all the projects have an initial release candidate out the door. The original release date is 5 days away, and it’s looking pretty good for hitting it. If you want a quick overview of what’s coming in this release, I’d recommend a look at Emilien Macchi’s Folsom overview, which is a pretty nice high level summary.

While we’ve been busy nailing down bugs and wrapping this release together, the OpenStack Foundation has finally come into form. As the Keystone project technical lead, I’m on the OpenStack technical committee – picture and title, but I haven’t written a bio yet. (Sorry Lauren). I find it really quite difficult to write a bio at myself. Regardless, it’s great to finally see this moving into a foundation external to any single corporate interest. That’s not to say it’s all in the land of milk and cookies, there’s just a lot of people with all slightly different interests jumping into the pool to push this little rowboat in different directions.

One thing that we did early out of excellent foresight was keeping the direction of the core projects democratically oriented by the contributors to those projects. The people that show up to write, update, and support the code are the ones that are ultimately making the decisions on what features get implemented, and when. Lots of folks talk about what could be, but it’s the contributors that make it happen.

A perfect example of this is Adam Young, who in the past 6 months drove the implementation of PKI based tokens in Keystone. I’m not even sure I’ve met Adam face to face, but I definitely know him – and he’s been a fantastic contributor, and in the Folsom development cycle was promoted to the core Keystone team.

To say it’s been a busy six months since the last post is really quite an understatement. What we released in the past month, and the work that went in to getting us there, is nothing short of phenomenal.

The OpenStack Essex release is out, and I’m still sort of catching my breath from that. Just six withs prior to the release I was elected as the PTL for Keystone. I had no idea how much additional work it was to wrangle the bugs, approvals, and the other sundry details that go into getting the release out the door. I rather wish there was a “new PTL user’s manual” for the process. It was a lot of learning, most of it last-minute and ad-hoc around our project management, release process and associated the mechanics. I spent a lot of time trying to figure out how to wrangle Launchpad into some semblance of useful tracking, etc. (To be fair, it was mostly a matter of learning what Launchpad could and couldn’t do, and figuring out the conventions that were already in place).

The past month was terrifically busy with getting OpenStack deployed and tested – running in small environments (devstack), and large. I’ve been working from the Ubuntu distributions myself, and the whole gang that has been packaging OpenStack (Chuck, Adam, Kiall, and others – inside Canonical and out) with the downstream distributions has done a tremendous job wrapping together the changes and making it into a deployable release from packages.

A much better release:

Where I was, er, more than disappointed with the Diablo, I’m significantly happier with the Essex release. Keystone (the project for which I am now the PTL) didn’t advance this release cycle so much as retrench and prepare for advancement this next release cycle. Horizon made huge leaps and bounds, which I think is fantastic as the face of OpenStack to many users. Nova and Swift advanced, and the integration work and definition going into making Quantum (and Melange) a reality has been terrific.

I think if I were to pull out a star of the show for the Essex release, then I’ve really got to point to the combined work for the crew formulating and keeping DevStack solid, and the CI team that integrated it into our development and review process so that we had a minimum of guaranteed interoperability. Where I was screaming into the wind during the last milestone of Diablo for continued breakages between the components, the integration of devstack has demanded that changes be rolled in with though to an overall use case and interop. With 200+ developers all kicking this ball down the field, the guaranteed CI has been the piece that has done us the most good.

A little about Keystone:

Keystone, as we retrenched and simplified the codebase, also has a significant advancement in integration testing. We replaced the entire codebase based on a series of integration tests that verified and guaranteed API compatibility with the Diablo and trunk releases as well as the client while we made that switch. The underlying code base is now significantly simplified, and the internal architecture of that service now wrapped around some core “internal service” concepts to allow us to have drivers that cleanly back-end into external systems to support identity and authorization.

I’ve received a number of questions about “why the API still sucks” six months later after Diablo. There’s still no obvious means for a user to “log out” (invalidate their own token), or change their own password (assuming the back-end were to support it). In switching out the underlying code base and architecture we needed to keep the API stable so that we could make sure we had the internals correct. With the internals switched out, it is now time to revisit the API and take the lessons we’ve learned from 6+ months of using the v2.0 API and improve upon it.

Looking towards the summit:

While I’m much, much happier with the Essex release, there are still plenty of places for improvement – many that I remember from the Diablo summit, and some new things that I’d love to see some focus on for the upcoming six months.

While Horizon has given us a dramatically improved user experience for OpenStack, there’s much more that we could do there – both with a web based UI, and with our command line interfaces to OpenStack. One thing that could use some explicit attention is the proliferation of “clients” needed to interact with OpenStack. As projects are splintering off Nova into their own domains, we have a number of new command line clients (nova, keystone, glance, quantum) – and they don’t all act the same. There is some great work on driving them to consistency, but I wonder if we shouldn’t bag all the individual clients and roll them together into one “OpenStack” client that is consistent in how it handles command line options, what the “commands” look like, and generally how you interact with them.

I hate to admit this, but even as the PTL of Keystone, I ran into a brick wall of old docs referring to using the command of “nova-manage project list” to see the projects, when I knew darn well and good they were all in the keystone system, and to see a list of them I should use the command “keystone tenant-list“. Not so hot, huh? Quantum and the nova-network components are going to really come into their own in this release, so we’ll have more shattering of the user experience from a deployer’s point of view unless we do something to be specific about tying it together.

There are more changes to be made in Keystone that I’m paying a lot of attention to – some of which are honestly going to require buy-in from the entire community to make happen. Where we place the boundaries of role based access, and how we deal with trust and information sharing about identity between the projects is probably going to need some changes that will ripple all the way down into the API’s of nova, swift, glance, quantum, melange, etc. Getting the brainstorming around those concepts and desired features is what I primarily want to accomplish at the Design Summit, with follow on in the milestone timeframe thereafter to bring the new features into existence and integrate them across OpenStack. My ideal goal is to get all the heavy features implemented early in the Folsom release cycle so that they’re solid and nailed down for the later milestones, and we can tweak and fix what may be broken long before any release points.

About that CloudStack thing:

Seems like in the last week, everybody (or at least the pundits) got their knickers in a twist about Citrix finally being more open about their dual-interest in CloudStack and OpenStack. Regardless of Gartner pundit sensationalistic babbling, the release of CloudStack as an Apache licensed project is nothing but goodness for the whole community. The Apache license makes me feel more comfortable with reviewing their code and looking at how they attacked the same issues and problems OpenStack has, and I expect they’ll look closely at how OpenStack has done the same. I applaud any company trying to build an open-source community, and I’m looking forward to seeing what Citrix does to actually do that community building. If it gains ground and folks contribute to CloudStack, we’re all better off – it’s more ideas hitting the street and becoming reality.

Finally, for anyone that didn’t think that the user-facing EC2 API was the defacto API, wake the F* up. It is reasonable, solid, and nobody is going to be turning it off any time soon. It also doesn’t allow/enable some things that a lot of cloud administrators would like to do – so it shouldn’t surprise anyone that CloudStack has their “api” and OpenStack has their “api”. The APIs are another place where we can learn from each other, as we add some value beyond what the bookseller down street might want to impose and expose.

If you want a little pundit action from this source, watch the API’s of both CloudStack and OpenStack (not just the EC2 compatibility layer) over the next six months. I think you’ll be seeing some very interesting steps forward.

One thing that was intended as new, and I think worked well, was the splitting of the conference and design summit. It really made it a lot easier to be involved in the technical details and not have to choose between that and listening to folks talk to what they’re Openstack experiences have been, governance models and choices, and so forth. As an active contributor to (and general jack-of-all-trades with) a number of the OpenStack projects, it’s hard enough to balance my focus between the technical sessions.

There was also a ton of news related to OpenStack that is just fantastic for the community. Rackspace is starting to form out a foundation to run OpenStack long term, and the governance round-table at the conference made it clear that pretty much everyone agreed to a thoughtful, careful process to set it up to really make a long term run with OpenStack. There was notable consensus on ideas and vision of how to set up governance, including taking some long and deep looks at how other successful open source foundations have been set up, and looking to those foundations to learn what’s been successful, and what they might have done differently.

There was also the entrance of Hewlett Packard very publicly actively into the OpenStack world. Back in September, they said they’d be there, and with the conference they have committed hardware and time to continuous integration efforts. They were notably involved in the design summit as well, hosting a couple of sessions and getting actively involved in others. And I think most pleasing to me, they are actively submitting code contributions to the OpenStack core projects. I think it’s a little odd that they’re all coming in as “HP Nova Contributors“, but I’m glad to see it.

With HP running their cloud offering on OpenStack, they’re getting deeper in the details of the code. Dell’s been focusing their efforts on the amazing OpenStack deployment tool: Crowbar, and it seems clear to me that HP is taking their interest quite a bit deeper into the core projects.

I won’t even do justice the the individual project roadmaps from the summit – I think that the most of the project technical leads are likely recovering this weekend from an intensely focused week. In the next couple of weeks, we’ll see them hit the mailing list – and from what I saw from the sessions in the conference, there’s going to be some hard choices coming up on where to expend resources. Every one of the projects has more that they want to do than they will be able to smoothly accomplish in the next 6 months. We’ll see the details hitting the blueprints in the very near future I’m sure. Suffice to say that if you’ve been thinking about getting involved in OpenStack, now is very definitely the time to jump in – there’s a ton to do, and a lot of really interesting problems to solve in getting it done.

These instructions are assuming you’re starting with an Ubuntu based system – either 10.10 or 11.04. I haven’t tried it as yet with 11.10.

First things first, I recommend you make sure you have the latest bits of everything:

sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get autoremove

Then we need to add the release “PPA” so that your system can grab the packages for Openstack:

sudo apt-get install python-software-properties
sudo add-apt-repository ppa:openstack-release/2011.3
sudo apt-get update

Now we get into the details. I’m going to drive out the instructions that will start with a single host, but are set up to add additional virtualization hosts as you need. I’m writing this assuming you’re working in a small network, and setting it up for FlatDHCP networking. Choosing the networking strategy and IP address space to use is actually one of the trickier parts of doing a reasonable install. For just testing something out in a test lab, this setup will work reasonable well – the only thing to really note is that this *will* install a DHCP server to provide IP addresses to the virtual instances, so if you have another DHCP server handing out addresses, you might need to get into the details and change some of these settings.

Installing the packages:

OpenStack relies on using MySQL as a data repository for information about the openstack configuration, so we’ll need to set up a MySQL server. Normally when you install the packages for MySQL, it’ll ask you about configuring a root password and such. We can make that hands-off by pre-answering some of those questions. To do this, make a file named “/tmp/mysql_preseed.txt” and put in it the following:

mysql-server-5.1 mysql-server/root_password password openstack
mysql-server-5.1 mysql-server/root_password_again password openstack
mysql-server-5.1 mysql-server/start_on_boot boolean true

Then we can get into the commands to install the packages:

cat /tmp/mysql_preseed.txt | debconf-set-selections
apt-get install mysql-server python-mysqldb
apt-get install rabbitmq-server
# ^^ pre-reqs for running controller nova instance
apt-get install euca2ools unzip
# ^^ for accessing nova through EC2 APIs
apt-get install nova-volume nova-vncproxy nova-api nova-ajax-console-proxy
apt-get install nova-doc nova-scheduler nova-objectstore
apt-get install nova-network nova-compute
apt-get install glance

That’s got all the packages installed onto your local system! Now we just need to configure it up and initialize some information (that’s the bit about networks, etc).

Before I get into changing configs, let me explain what I’ll be setting up. In this example, my internal “network” is 172.17.0.0/24 – and I have a dedicated IP address for this host that is 172.17.0.133. The virtual machines will be in their own network space (10.0.0.0 to 10.0.0.254), and (at this point) not visible from the local network, but will be able to access the local network through their virtualization hosts. The machine I’m using also only has a single NIC (eth0), which is fine for a little test bed, but not likely what you want to do in any sort of real setup.

/etc/nova/nova.conf

--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--flagfile=/etc/nova/nova-compute.conf
--verbose
#
--sql_connection=mysql://novadbuser:novaDBsekret@172.17.0.133/nova
#
--network_manager=nova.network.manager.FlatDHCPManager
--flat_network_bridge=br100
--flat_injected=False
--flat_interface=eth0
--public_interface=eth0
#
--vncproxy_url=http://172.17.0.133:6080
--daemonize=1
--rabbit_host=172.17.0.133
--osapi_host=172.17.0.133
--ec2_host=172.17.0.133
--image_service=nova.image.glance.GlanceImageService
--glance_api_servers=172.17.0.133:9292
--use_syslog

Now you might have noticed the MySQL connection string in there. We need to set up that user and password in MySQL to do what needs to be done. I also change the MySQL configuration so that remote systems can connect to MySQL. It’s not needed on a single host, but if you ever want to have more than one compute host, you need to make this change. In /etc/mysql/my.conf, find the line:

bind-address = 127.0.0.1

and change it to

bind-address 0.0.0.0

Now lets make the user in Mysql:

mysql -popenstack
CREATE USER 'novadbuser' IDENTIFIED BY 'novaDBsekret';
GRANT ALL PRIVILEGES ON *.* TO 'novadbuser'@'%' WITH GRANT OPTION;
FLUSH PRIVILEGES;

And set up the database:

mysql -popenstack -e 'CREATE DATABASE nova;'
nova-manage db sync

If that last command gives you any trouble, then we likely don’t have the MySQL system configured correctly – the user can’t access the tables or something. Check in the logs for MySQL to get a sense of what might have gone wrong.

At this point, it’s time to configure up the internals of openstack – create projects, networks, etc.
We’ll start by creating an admin user:

# create admin user called "cloudroot"
nova-manage user admin --name=cloudroot --secret=sekret

This should respond with something like:

export EC2_ACCESS_KEY=sekret
export EC2_SECRET_KEY=653f3fad-df22-449b-9e6a-ea6c81e32621

You can scratch that down, but we’ll be getting that same information again later and using it, so don’t worry too much about it.

Now we create a project:

# create project "cloudproject" with project mgr: "cloudroot"
nova-manage project create --project=cloudproject --user=cloudroot

And finally, a network configuration for those internal IP addresses:

nova-manage network create private \
    --fixed_range_v4=10.0.0.0/24 \
    --num_networks=1 \
    --network_size=256 \
    --bridge=br100 \
    --bridge_interface=eth0 \
    --multi_host=T
# gateway assumed at 10.0.0.1
# broadcast assumed at 10.0.0.255

Now I’m using the multi-host flag, which is new in the Diablo release. This makes each compute node it’s own networking host for the purposes of allowing the VM’s you spin up to access your network or the internet.

At this point, you’re system should be up and running, all systems operational. Let me walk you through the command steps to actually kick up a little test VM though. These commands are all meant to be done as a local user (not root!)

sudo nova-manage project zipfile cloudproject cloudroot /tmp/nova.zip
unzip -o /tmp/nova.zip -d ~/creds
cat creds/novarc >> ~/.bashrc
source creds/novarc
#
euca-add-keypair mykey > mykey.priv
chmod 600 mykey.priv
#
image="ttylinux-uec-amd64-12.1_2.6.35-22_1.tar.gz"
wget http://smoser.brickies.net/ubuntu/ttylinux-uec/$image
uec-publish-tarball $image mybucket
#
wget http://uec-images.ubuntu.com/releases/10.04/release/ubuntu-10.04-server-uec-amd64.tar.gz
uec-publish-tarball ubuntu-10.04-server-uec-amd64.tar.gz mybucket
...OUTPUT...
Thu Aug 18 14:02:20 PDT 2011: ====== extracting image ======
Warning: no ramdisk found, assuming '--ramdisk none'
kernel : lucid-server-uec-amd64-vmlinuz-virtual
ramdisk: none
image  : lucid-server-uec-amd64.img
Thu Aug 18 14:02:29 PDT 2011: ====== bundle/upload kernel ======
Thu Aug 18 14:02:34 PDT 2011: ====== bundle/upload image ======
Thu Aug 18 14:03:12 PDT 2011: ====== done ======
emi="ami-00000002"; eri="none"; eki="aki-00000001";
...OUTPUT...

And running the instances:

euca-run-instances ami-00000002 -k mykey -t m1.large
...OUTPUT...
RESERVATION r-1jj2a80v  cloudproject    default
INSTANCE    i-00000001  ami-00000002            scheduling  mykey (cloudproject, None)  0       m1.tiny2011-08-18T21:06:03Z unknown zone    aki-00000001    ami-00000000
...OUTPUT...
#
euca-describe-instances 
...OUTPUT...
RESERVATION r-1jj2a80v  cloudproject    default
INSTANCE    i-00000001  ami-00000002    10.0.0.2    10.0.0.2    building    mykey (cloudproject, SIX)   0   m1.tiny 2011-08-18T21:06:03Z    nova    aki-00000001    ami-00000000
...OUTPUT...
#
euca-describe-instances 
...OUTPUT...
RESERVATION r-1jj2a80v  cloudproject    default
INSTANCE    i-00000001  ami-00000002    10.0.0.2    10.0.0.2    running mykey (cloudproject, SIX)   0       m1.tiny 2011-08-18T21:06:03Z    nova    aki-00000001    ami-00000000
...OUTPUT...
#
euca-authorize -P tcp -p 22 default
ssh -i mykey.priv root@10.0.0.2

To add on additional hosts to support more VMs, you only need to install a few of the packages:

apt-get install nova-compute nova-network nova-api

You do need that exact same /etc/nova/nova.conf file though.

Note:
The default install of Glance expects the images that you’ve loaded up to be available on the local file system for every compute node at /var/lib/glance. Either NFS mount this directory from a central machine, or replicate the files underneath it to all your “compute hosts” when you upload a new image to be used in the virtual machines.

Also, the metadata URL needed for UEC images (169.154.169.154) may need help getting forwarded when running on a system with a single NIC. Two potential solutions: A) run nova-api on each of the compute nodes (quick and dirty) or B) specify the –ec2_dmz_host=$HOSTIP, and potentially invoke the command ip link set dev br100 promisc on to turn on promiscuous mode (per https://answers.launchpad.net/nova/+question/152528).

If you’re into OpenStack, come join us on September 28th to celebrate the Diablo release with other stackers in Seattle.

HP Cloud Services has graciously offered up their offices at 701 Pike St, Suite 1100, Seattle WA to host a meetup.

If you’re planning on coming, please stop by the Meetup link at http://www.meetup.com/OpenStack-Seattle/events/33922932/ and RSVP for us so we can get a sense of who might be wandering by chat and say hello!

We have Quantum and Glance shifting to use Github as a repository, new processes (that would be using Gerrit) wrapped around GitHub to allow the project to have a “gated trunk” methodology, and lots of “motion” within the various projects. It’s pretty easy to see on Github, a little harder with launchpad (I just don’t have the tools handy to create the pretty graphs) – you can see the impact graphs for swift, keystone, glance, quantum, and openstack-dashbaord to see what I mean.

The shifting to Gerrit hasn’t been without it’s trials, but is coming along pretty well now. I really wish the GitHub folks had been a bit more amenable to putting in a field that external folks could use to store metadata about a pull request. Several folks from the OpenStack project (including myself) reached out to them about this, all rebuffed (nicely, but still). In fact, one of the suggestions I got back from the github’r support was “Why don’t you set up Gerrit?”

With the changes in core repository, lots of dependencies are shifting as well. Dashboard was broken a bit this week we kicked things around to get the dependencies to match the new locations, I think we’ve got all those pieces worked around now (pull request outstanding for openstack-dashboard). The other piece that really shifted and broke with these changes were the install scripts that we’ve been using to build and work on a developer’s environment. The cloudbuilder team at Rackspace recently created a whole new setup that works very nicely, so I think we’re going to drop our older scripts (based on over-extended versions of Vish’s excellent nova.sh script) and move to using their new “stack.sh“. (And yeah, of course we’ll want to mess with it ourselves, so I’ve forked it…)

Quick recap: The service framework in nova is set up to make it easy to write your own services that interact with any other nova services (such as nova-network, nova-scheduler, etc). The service framework includes all the pieces to communicate with these other services (using a module called rpc.py that abstracts away the communications), a database connection for looking up data from the nova persistence store, and leaves the rest to you.

The service module is expecting to be told a manager class to load, and the framework will use that to do what it needs. There are only two required methods to overwrite:

• init_host
• periodic_tasks

There are two kinds of services – I’m going to focus on the stand-alone (not WSGI) service that expects to communicate and respond entirely through the message queue system in Nova.

So how does this service critter work? Well, when it’s initialized, it gets a number of attributes assigned to it. This is typically done from a class method on service.py:

from nova.service import Service
my_service = Service.create()

The create() method has a number of parameters:

• host – a string with the host this service is running on
• binary – a string with the binary name of this program
• topic – a string with a subset of the binary name, used to set up a message exchange in AMQP
• manager – a string with the class name to be loaded to do the ‘work’
• report_interval – an interval set from Flags, triggering a regular reporting loop
• periodic_interval – an interval set from Flags, triggering a periodic loop to do repeating tasks

If you don’t provide any of them, all these values get populated with defaults and from configuration detail (the flags) in the service.serve() method. Once serve() gets everything configured up, it calls .start() on each service to kick it into gear.

start() is where things really get moving. This is where the service manager class gets loaded, registered with the nova database (if it isn’t already), and the RPC mechanisms get spun up with Eventlet greenthreads to accept messages to this service. The topic (which is the name of the binary, minus any “nova-” in front of it) is used as an exchange. Through this mechanism, any service can talk to any other service (or set of services). Here’s how that works:

rpc.py has two methods: call() and cast() that do all the heavy lifting. When you use these, they take a “context” (i.e. authorization for who’s doing the call), a topic (the name of the service you’re calling), and a message. call() sends this message and waits for a response. cast() sends the message entirely asynchronously, not expecting a response.

The message is a JSON structure – a dictionary, and it’s expected that the dictionary will have a key “method” and another key “args”. method is expected to be a string, and args is expected to be another dictionary. The rpc module does the work of using that method string to look up and invoke the method on your manager.

An example of this operating is right in the code. In the nova-network API, there’s an rpc.cast():

rpc.cast(context,
         self.db.queue_get_for(context, FLAGS.network_topic, host),
         {'method': 'associate_floating_ip',
          'args': {'floating_address': floating_ip['address'],
                   'fixed_address': fixed_ip['address']}})

Through the service framework and the RPC mechanisms, this is calling associate_floating_ip() on the network service manager class.

The other nifty thing about service is that it’s keeping and managing a number of greenthreads from Eventlet to do it’s work. The basic bits are all encapsulated in that rpc.py mechanism – when it sets up connections to the message queue service to receive communications, that starts a greenthread rolling to watch out for, pull, and process any messages inbound. The two periodic interval pieces are also spun up on their own greenthreads – looping every “interval” (specified by the flags –report_interval and –periodic_interval, set at 10 and 60 seconds by default respectively). These run continuously until the service is terminated.

Before I dive into the gritty details, it’s worth getting a high level overview so that some of this (hopefully) makes sense. OpenStack’s service architecture is made up of services that all talk with each other to get things done. nova-network, nova-scheduler, etc. There’s a lot of underpinning in the nova codebase to make those services relatively easy to write and work together – I was mostly curious about how they passed messages back and forth. As I dove in, the two pieces that stood out as needing to be understood first were the unified service framework in nova and configuration using flags (which it heavily depends upon).

Configuration – using the flags

The configuration for nova services – global or specific to a service – are all done with configuration files that can be over-ridden on the command line, taking advantage of python gflags to make it all work nice. I didn’t know much about the flags system, so I dug around in the python-gflags project. They have the documentation for how to use gFlags in the code itself: http://python-gflags.googlecode.com/svn/trunk/gflags.py.

To summarize it up:

Python modules in the codebase can define and use flags, and there is a general nova flags file that holds the cross-service (common) configuration settings. Nova defaults to looking for it’s configuration in a nova.conf file in the local directory. Failing that it looks for the nova.conf file in /etc/nova/nova.conf. Where it looks for the configuration file can be overridden (typically on the command line) by (--flagfile) and a location to a config file. The code that makes this happen nova.utils.default_flagfile().

To use the configuration from within code, you typically instantiate the global flags, add any flag definitions (with default values) that you care to add, and then use ‘em! Here’s a code snippet example:

from nova import flags
# import the nova wrapper around python gflags
#  .. there's some interesting wrapping for taking in arguments
#     and passing along extras values to your code
#  .. and it's where the global flag definitions reside
FLAGS = flags.FLAGS
# get the global instance
#  .. this attempts to read the /etc/nova/nova.conf for flags
# 
# You can define an additional flag here if you needed to...
flags.DEFINE_string('my_flag', 'default_value', \
        'human readable description of your flag')
# there's also flags.DEFINE_bool, flags.DEFINE_integer and more...
# 
# And then you can use the flags
#  .. the flags you defined show up as attributes 
#     on that FLAGS object
print FLAGS.my_ip

If you were happening to write a script that took in flags and worked with them for a command-line script, you might do something like:

from nova import flags
form nova import utils
utils.default_flagfile()
flags.FLAGS(sys.argv)
GLOBAL_FLAGS = flags.FLAGS
# ... and on to the rest of your code

There is some good end-user documentation on how to find the flags. The gist is – if you want to know what flags are there, the easiest way is to hand in the flag “–help” or “–shorthelp” from the command line. That is how the gFlags library is set up to tell you about the flags.

Update:

After a little digging down a side passage, I noticed that service.py had some debugging code in it that iterated through all the set flags. You iterate directly on FLAGS (treating it as an iterable thing) and use FLAGS.get() to retrieve the set values.

    logging.debug(_('Full set of FLAGS:'))
    for flag in FLAGS:
        flag_get = FLAGS.get(flag, None)
        logging.debug('%(flag)s : %(flag_get)s' % locals())

Services

There are two types of services in Nova: system services and web services. The code to use and launch them is basically the same, and Nova has this all bundled into a general service architecture and code base. The reason that configuration is so important is that the nova service framework has a convention of knowing how to run a service based on flags from the framework.

Here’s a bit of example code of a service to illustrate what I’m talking about.

nova-exampleservice:

import eventlet
eventlet.monkey_patch()

import sys
from nova import flags
from nova import service
from nova import utils

if __name__ == '__main__':
    utils.default_flagfile()
    flags.FLAGS(sys.argv)
    service.serve()
    service.wait()

The convention starts off by using the name of the script invoked – in this case “nova-exampleservice”. The scripts in bin/ (like nova-network) use this mechanism. This convention can be overridden, of course, but it does make things pretty straightforward once you know the convention. The key to this convention is that the code in nova.service looks in the configuration for a class to instantiate (expected to be a subclass of nova.manager.Manager) named after the service that was just invoked. (this convention is in code under the nova.service.create() method)

For our example of nova-exampleservice, the service is going to look in the configuration for exampleservice_manager, expecting the value to be a class that it can load that will be a subclass of nova.manager.Manager and will be responsible for running the service.

This code is invoked from service.serve() from our example above. Again, it looks for the flag “exampleservice_manager” and try to load that class to do the work.

An updated example that sets a default manager that will attempt to load the class mymodule.exampleservice.ExampleServiceManager by default:

nova-exampleservice:

import eventlet
eventlet.monkey_patch()

import sys
from nova import flags
from nova import service
from nova import utils

if __name__ == '__main__':
    utils.default_flagfile()
    flags.FLAGS(sys.argv)
    flags.DEFINE_string('exampleservice_manager',
            'mymodule.exampleservice.ExampleServiceManager',
            'Default manager for the nova-exampleservice')
    service.serve()
    service.wait()

The manager class has two classes that you override to get your stuff done:

• init_host
• periodic_tasks

There are also some conventions around adding methods to your manager and invoking them using the service framework’s RPC mechanism, which I’ll dig into with another post.

Ref: Nova Developer Documentation
Ref: OpenStack Compute (Nova) Administration Manual
Ref: Openstack Wiki: Unified Service Architecture

I’m saying all this after a week with the OpenStack guys – both in design sessions and just chillin’ out. Focused, intelligent, demanding conversations scattered through the week with an amazing “no-ego” attitude presenting itself. Not that there weren’t some good ole technical “best way to do it” or “which is better” fights, but given the breadth of this project and the open nature with vendors lurking all around the corners – well, frankly I expected a lot more “special interest” to be clearly showing itself. Everyone at that conference was interested in making OpenStack better at every turn.

250 people, 12 countries, 90 companies/organizations – all that after 3 months from being publicly announced. And they’re going it without any prior structure – building up an OpenStack foundation, doing all the legal and community building, right from scratch. And yeah – that’s showing right now.

The first thing I see that will provide the biggest gains:

• “How do we all work together?”

Some of the best sessions were around “What does the status X of bugs mean” and talking through the development and release process. At this point I’m convinced the core folks are reasonably comfortable with LaunchPad (the platform the system is hosted on) – and being at the conference really taught me a great deal about how OpenStack is effectively using it. Prior, it wasn’t comfortable or familiar to me at all. The object store and compute (swift and nova, respectively) core groups are really quite separate teams, all trying to figure out how to get some common ground in re-using code, libraries, and even setting up documentation.

The second:

• “Show me it’s workin’, again and again”

OpenStack is quickly heading to be the kernel or core of a platform. You could see it in the twinkle of Eucalyptus’ eye when they talked about Swift (the object store), or chatting with the folks from Scalr or RightScale. The whole system is being built with API in mind from the ground up, and while there is some pretty good unit testing in play and continuous integration, it was clear that installing this sucker was a PITA – and the documentation to really pull that all together starting coming together in the documentation sprint and install fest at the summit. One of the “blueprints” of the design summit (i.e. “Things we want to do, and how we want to do it for the next release”) is to get some fully automated integration testing as well as track the metrics on how the system is operating. There were a lot of folks that have some cross over into the Drizzle project, and the ideas of running and tracking benchmark data on every revision is darned power.

Add to that the benefits of a constant flow of functional testing against a couple of pre-defined clusters of both compute and object store, and you have a powerful engine to make sure trouble is spotted early and can be resolved quickly.

The third:

• “How’s this thing tick?”

One of the admitted weak points is that some small, damned effective core teams have done most of the work – and if you want to understand the system, well… you’ve just got to read the code. That is a huge investment – and frankly a barrier to entry into the project that can be avoided with some effort towards docs and discussion. Again, great progress was made there (I learned what the “project” concept was in Nova at the summit) – but the interactions between components, what the components are responsible for, and what they’re *not* responsible for, are all kind of tricky to learn right now.

This extends down into digging into the code, where docstrings could be better (and are getting better!) so that if you wanted to go help with something specific, you didn’t have to grok a broad codebase to get a handle on what the impacts are of the changes you’ll need to make.

And the last thing I’ll throw in here:

• “What OpenStack isn’t, or won’t do…”

The project is still in a lot of flux. There were some great components that were shown off at the summit that ride over the top of the infrastructure, or work with it through APIs. Should those be a part of OpenStack, or on the side? Some service providers were very interested in more platform-kind of elements – a common logging infrastructure, a common authentication, ID, and authorization infrastructure. Should that be a part, or on the side? How tightly or loosely do we want to couple some of these elements? The philosophy is there and forming up, but the real truth of it all will be over the next 6-12 months of the project when decisions are made, reviewed, and a core forms out of it. There have been a few architectural decisions made early: “Don’t mandate anything in the client”, “If a feature would restrict scale, it MUST be optional”, etc. that I absolutely applaud. I think it will form up more as projects apply to join the OpenStack umbrella and either make it or don’t. It will become clear what’s common, and what isn’t, pretty darn quickly.

I’m pumped about this project, the people, and it’s future. The core openstacker’s have clearly been driving up a steep hill to get to where they can level out a bit and move into more of a marathon mode. Really, it feels like we’re nearly at the top of that first hill.

“Ahh!”, I thought – an easy intro to getting into the project and it’ll give me an excuse to really read the code. So this weekend I started taking a stab at doing a little light buff and puff on the code to get the PEP8 and pylint code scores up a bit.

What I found is that it took some work to get everything from the codebase ready to really do some work on it. And the notes aren’t all in the same places on how to do that – what notes are there are all written mostly for Ubuntu. I was pretty sure most of this could be done on a Mac too – at least based on the dependency documentation, so I cobbled up some notes on getting rolling with Launchpad, this code base, and being able to run the tests to verify that my cleaning didn’t really break anything.

I put the resulting notes on the OpenStack wiki page http://wiki.openstack.org/HackingNovaMacOSX.

The big thing that I’m not sure about is the testing. Just a stock install is failing on one unit test, and it’s in an area I’m not very familiar with (auth & creating certificates). If any the OpenStack folk are reading this, here’s the error I’m seeing:

[ERROR]: nova.tests.auth_unittest.AuthTestCase.test_209_can_generate_x509

Traceback (most recent call last):
  File "/Users/heckj/Documents/code/nova/nova/test.py", line 222, in run
    d = self._maybeInlineCallbacks(testMethod)
  File "/Users/heckj/Documents/code/nova/nova/test.py", line 182, in _maybeInlineCallbacks
    g = f()
  File "/Users/heckj/Documents/code/nova/nova/tests/auth_unittest.py", line 162, in test_209_can_generate_x509
    signed_cert = X509.load_cert_string(cert_str)
  File "/Users/heckj/Documents/code/nova/.nova-venv/lib/python2.6/site-packages/M2Crypto/X509.py", line 655, in load_cert_string
    return load_cert_bio(bio, format)
  File "/Users/heckj/Documents/code/nova/.nova-venv/lib/python2.6/site-packages/M2Crypto/X509.py", line 639, in load_cert_bio
    raise X509Error(Err.get_error())
M2Crypto.X509.X509Error: 140735090166816:error:0906D06C:PEM routines:PEM_read_bio:no start line:/SourceCache/OpenSSL098/OpenSSL098-32/src/crypto/pem/pem_lib.c:650:Expecting: CERTIFICATE

update: Turns out the error was directly related to the version of OpenSSL installed on my laptop. I had version 1.0.0a from MacPorts installed and in my default path, which caused the error. Version 0.9.8l (base install in MacOS X) works fine.

sudo port deactivate openssl

did the trick and the tests are all running now. I updated the bug against Nova with those details, leaving it open – it ought to at least fail reasonably.