One thing that was intended as new, and I think worked well, was the splitting of the conference and design summit. It really made it a lot easier to be involved in the technical details and not have to choose between that and listening to folks talk to what they’re Openstack experiences have been, governance models and choices, and so forth. As an active contributor to (and general jack-of-all-trades with) a number of the OpenStack projects, it’s hard enough to balance my focus between the technical sessions.

There was also a ton of news related to OpenStack that is just fantastic for the community. Rackspace is starting to form out a foundation to run OpenStack long term, and the governance round-table at the conference made it clear that pretty much everyone agreed to a thoughtful, careful process to set it up to really make a long term run with OpenStack. There was notable consensus on ideas and vision of how to set up governance, including taking some long and deep looks at how other successful open source foundations have been set up, and looking to those foundations to learn what’s been successful, and what they might have done differently.

There was also the entrance of Hewlett Packard very publicly actively into the OpenStack world. Back in September, they said they’d be there, and with the conference they have committed hardware and time to continuous integration efforts. They were notably involved in the design summit as well, hosting a couple of sessions and getting actively involved in others. And I think most pleasing to me, they are actively submitting code contributions to the OpenStack core projects. I think it’s a little odd that they’re all coming in as “HP Nova Contributors“, but I’m glad to see it.

With HP running their cloud offering on OpenStack, they’re getting deeper in the details of the code. Dell’s been focusing their efforts on the amazing OpenStack deployment tool: Crowbar, and it seems clear to me that HP is taking their interest quite a bit deeper into the core projects.

I won’t even do justice the the individual project roadmaps from the summit – I think that the most of the project technical leads are likely recovering this weekend from an intensely focused week. In the next couple of weeks, we’ll see them hit the mailing list – and from what I saw from the sessions in the conference, there’s going to be some hard choices coming up on where to expend resources. Every one of the projects has more that they want to do than they will be able to smoothly accomplish in the next 6 months. We’ll see the details hitting the blueprints in the very near future I’m sure. Suffice to say that if you’ve been thinking about getting involved in OpenStack, now is very definitely the time to jump in – there’s a ton to do, and a lot of really interesting problems to solve in getting it done.

These instructions are assuming you’re starting with an Ubuntu based system – either 10.10 or 11.04. I haven’t tried it as yet with 11.10.

First things first, I recommend you make sure you have the latest bits of everything:

sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get autoremove

Then we need to add the release “PPA” so that your system can grab the packages for Openstack:

sudo apt-get install python-software-properties
sudo add-apt-repository ppa:openstack-release/2011.3
sudo apt-get update

Now we get into the details. I’m going to drive out the instructions that will start with a single host, but are set up to add additional virtualization hosts as you need. I’m writing this assuming you’re working in a small network, and setting it up for FlatDHCP networking. Choosing the networking strategy and IP address space to use is actually one of the trickier parts of doing a reasonable install. For just testing something out in a test lab, this setup will work reasonable well – the only thing to really note is that this *will* install a DHCP server to provide IP addresses to the virtual instances, so if you have another DHCP server handing out addresses, you might need to get into the details and change some of these settings.

Installing the packages:

OpenStack relies on using MySQL as a data repository for information about the openstack configuration, so we’ll need to set up a MySQL server. Normally when you install the packages for MySQL, it’ll ask you about configuring a root password and such. We can make that hands-off by pre-answering some of those questions. To do this, make a file named “/tmp/mysql_preseed.txt” and put in it the following:

mysql-server-5.1 mysql-server/root_password password openstack
mysql-server-5.1 mysql-server/root_password_again password openstack
mysql-server-5.1 mysql-server/start_on_boot boolean true

Then we can get into the commands to install the packages:

cat /tmp/mysql_preseed.txt | debconf-set-selections
apt-get install mysql-server python-mysqldb
apt-get install rabbitmq-server
# ^^ pre-reqs for running controller nova instance
apt-get install euca2ools unzip
# ^^ for accessing nova through EC2 APIs
apt-get install nova-volume nova-vncproxy nova-api nova-ajax-console-proxy
apt-get install nova-doc nova-scheduler nova-objectstore
apt-get install nova-network nova-compute
apt-get install glance

That’s got all the packages installed onto your local system! Now we just need to configure it up and initialize some information (that’s the bit about networks, etc).

Before I get into changing configs, let me explain what I’ll be setting up. In this example, my internal “network” is 172.17.0.0/24 – and I have a dedicated IP address for this host that is 172.17.0.133. The virtual machines will be in their own network space (10.0.0.0 to 10.0.0.254), and (at this point) not visible from the local network, but will be able to access the local network through their virtualization hosts. The machine I’m using also only has a single NIC (eth0), which is fine for a little test bed, but not likely what you want to do in any sort of real setup.

/etc/nova/nova.conf

--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--flagfile=/etc/nova/nova-compute.conf
--verbose
#
--sql_connection=mysql://novadbuser:novaDBsekret@172.17.0.133/nova
#
--network_manager=nova.network.manager.FlatDHCPManager
--flat_network_bridge=br100
--flat_injected=False
--flat_interface=eth0
--public_interface=eth0
#
--vncproxy_url=http://172.17.0.133:6080
--daemonize=1
--rabbit_host=172.17.0.133
--osapi_host=172.17.0.133
--ec2_host=172.17.0.133
--image_service=nova.image.glance.GlanceImageService
--glance_api_servers=172.17.0.133:9292
--use_syslog

Now you might have noticed the MySQL connection string in there. We need to set up that user and password in MySQL to do what needs to be done. I also change the MySQL configuration so that remote systems can connect to MySQL. It’s not needed on a single host, but if you ever want to have more than one compute host, you need to make this change. In /etc/mysql/my.conf, find the line:

bind-address = 127.0.0.1

and change it to

bind-address 0.0.0.0

Now lets make the user in Mysql:

mysql -popenstack
CREATE USER 'novadbuser' IDENTIFIED BY 'novaDBsekret';
GRANT ALL PRIVILEGES ON *.* TO 'novadbuser'@'%' WITH GRANT OPTION;
FLUSH PRIVILEGES;

And set up the database:

mysql -popenstack -e 'CREATE DATABASE nova;'
nova-manage db sync

If that last command gives you any trouble, then we likely don’t have the MySQL system configured correctly – the user can’t access the tables or something. Check in the logs for MySQL to get a sense of what might have gone wrong.

At this point, it’s time to configure up the internals of openstack – create projects, networks, etc.
We’ll start by creating an admin user:

# create admin user called "cloudroot"
nova-manage user admin --name=cloudroot --secret=sekret

This should respond with something like:

export EC2_ACCESS_KEY=sekret
export EC2_SECRET_KEY=653f3fad-df22-449b-9e6a-ea6c81e32621

You can scratch that down, but we’ll be getting that same information again later and using it, so don’t worry too much about it.

Now we create a project:

# create project "cloudproject" with project mgr: "cloudroot"
nova-manage project create --project=cloudproject --user=cloudroot

And finally, a network configuration for those internal IP addresses:

nova-manage network create private \
    --fixed_range_v4=10.0.0.0/24 \
    --num_networks=1 \
    --network_size=256 \
    --bridge=br100 \
    --bridge_interface=eth0 \
    --multi_host=T
# gateway assumed at 10.0.0.1
# broadcast assumed at 10.0.0.255

Now I’m using the multi-host flag, which is new in the Diablo release. This makes each compute node it’s own networking host for the purposes of allowing the VM’s you spin up to access your network or the internet.

At this point, you’re system should be up and running, all systems operational. Let me walk you through the command steps to actually kick up a little test VM though. These commands are all meant to be done as a local user (not root!)

sudo nova-manage project zipfile cloudproject cloudroot /tmp/nova.zip
unzip -o /tmp/nova.zip -d ~/creds
cat creds/novarc >> ~/.bashrc
source creds/novarc
#
euca-add-keypair mykey > mykey.priv
chmod 600 mykey.priv
#
image="ttylinux-uec-amd64-12.1_2.6.35-22_1.tar.gz"
wget http://smoser.brickies.net/ubuntu/ttylinux-uec/$image
uec-publish-tarball $image mybucket
#
wget http://uec-images.ubuntu.com/releases/10.04/release/ubuntu-10.04-server-uec-amd64.tar.gz
uec-publish-tarball ubuntu-10.04-server-uec-amd64.tar.gz mybucket
...OUTPUT...
Thu Aug 18 14:02:20 PDT 2011: ====== extracting image ======
Warning: no ramdisk found, assuming '--ramdisk none'
kernel : lucid-server-uec-amd64-vmlinuz-virtual
ramdisk: none
image  : lucid-server-uec-amd64.img
Thu Aug 18 14:02:29 PDT 2011: ====== bundle/upload kernel ======
Thu Aug 18 14:02:34 PDT 2011: ====== bundle/upload image ======
Thu Aug 18 14:03:12 PDT 2011: ====== done ======
emi="ami-00000002"; eri="none"; eki="aki-00000001";
...OUTPUT...

And running the instances:

euca-run-instances ami-00000002 -k mykey -t m1.large
...OUTPUT...
RESERVATION r-1jj2a80v  cloudproject    default
INSTANCE    i-00000001  ami-00000002            scheduling  mykey (cloudproject, None)  0       m1.tiny2011-08-18T21:06:03Z unknown zone    aki-00000001    ami-00000000
...OUTPUT...
#
euca-describe-instances
...OUTPUT...
RESERVATION r-1jj2a80v  cloudproject    default
INSTANCE    i-00000001  ami-00000002    10.0.0.2    10.0.0.2    building    mykey (cloudproject, SIX)   0   m1.tiny 2011-08-18T21:06:03Z    nova    aki-00000001    ami-00000000
...OUTPUT...
#
euca-describe-instances
...OUTPUT...
RESERVATION r-1jj2a80v  cloudproject    default
INSTANCE    i-00000001  ami-00000002    10.0.0.2    10.0.0.2    running mykey (cloudproject, SIX)   0       m1.tiny 2011-08-18T21:06:03Z    nova    aki-00000001    ami-00000000
...OUTPUT...
#
euca-authorize -P tcp -p 22 default
ssh -i mykey.priv root@10.0.0.2

To add on additional hosts to support more VMs, you only need to install a few of the packages:

apt-get install nova-compute nova-network nova-api

You do need that exact same /etc/nova/nova.conf file though.

Note:
The default install of Glance expects the images that you’ve loaded up to be available on the local file system for every compute node at /var/lib/glance. Either NFS mount this directory from a central machine, or replicate the files underneath it to all your “compute hosts” when you upload a new image to be used in the virtual machines.

Also, the metadata URL needed for UEC images (169.154.169.154) may need help getting forwarded when running on a system with a single NIC. Two potential solutions: A) run nova-api on each of the compute nodes (quick and dirty) or B) specify the –ec2_dmz_host=$HOSTIP, and potentially invoke the command ip link set dev br100 promisc on to turn on promiscuous mode (per https://answers.launchpad.net/nova/+question/152528).

We have Quantum and Glance shifting to use Github as a repository, new processes (that would be using Gerrit) wrapped around GitHub to allow the project to have a “gated trunk” methodology, and lots of “motion” within the various projects. It’s pretty easy to see on Github, a little harder with launchpad (I just don’t have the tools handy to create the pretty graphs) – you can see the impact graphs for swift, keystone, glance, quantum, and openstack-dashbaord to see what I mean.

The shifting to Gerrit hasn’t been without it’s trials, but is coming along pretty well now. I really wish the GitHub folks had been a bit more amenable to putting in a field that external folks could use to store metadata about a pull request. Several folks from the OpenStack project (including myself) reached out to them about this, all rebuffed (nicely, but still). In fact, one of the suggestions I got back from the github’r support was “Why don’t you set up Gerrit?”

With the changes in core repository, lots of dependencies are shifting as well. Dashboard was broken a bit this week we kicked things around to get the dependencies to match the new locations, I think we’ve got all those pieces worked around now (pull request outstanding for openstack-dashboard). The other piece that really shifted and broke with these changes were the install scripts that we’ve been using to build and work on a developer’s environment. The cloudbuilder team at Rackspace recently created a whole new setup that works very nicely, so I think we’re going to drop our older scripts (based on over-extended versions of Vish’s excellent nova.sh script) and move to using their new “stack.sh“. (And yeah, of course we’ll want to mess with it ourselves, so I’ve forked it…)

That I primarily work in two modes:

• “gettin’ shit done and stay the fuck outa my way”

Which is the rude way of saying I’m hammering out basic frameworks, making sure the theories of how something will go together work, mostly ignoring error conditions and error checking, and testing is manual and just as I need it. I sketch out structures, build blocks on each other, and hopefully end up with my arch/bridge/sofware component/etc – whatever construct I’m making.

• “making it correct”

In which I get totally OCD about understanding how something works – even something I’ve just created, down to creating tests to cover the logic, defining the failure conditions, and picking “just the right names” for objects, method calls, and general analogies.

The first shows more apparent progress – really short term progress – very quickly. But if that’s all you do, then I’m deep in the bug/muck in not very long at all. I’ve worked with a lot of people who don’t toggle between these modes – or I think even recognize them. I think that’s a bit of a tragedy, and something I encourage people to look at introspectively when they’re working for me.

Another realization:

I have far, far less patience with executives and managers than I do with technical folks/line workers.

The explanation into that is a bit more lengthy:

Being a good manager is very, very difficult. (Related reading that I highly recommend: Managing Humans and Behind Closed Doors. Being a good executive (director or whatever up) is even harder. Doing either of those jobs without resorting to being a dick – keeping an objective and fair eye out is damn near impossible. The dirty truth (that most folks in management positions realize pretty darn quickly) is that you get more money and tend to make more progress if you ARE a dick/backstabber/amoral sonuvabitch. Much of the thinking at an executive level is about capital, progress, and rewards – where anything you control is capital, what you output is progress, and what you get from all this is rewards. If you want to know in a general sense why an executive is doing anything, find out what their reward structure is. The track from there is usually abundantly clear.

Oh – and as for that ‘progress’ thing, that’s really naive of me to assert. Here’s the cynical me:

I’ve seen instances where output is status quo because that makes larger business sense: you want to keep some segment of the operation running solidly. Those are usually the pieces of the business that bore the shit out of me, and ironically the ones that are “looking for creative people” – usually to do mind-numbing work.

I’ve also seen directors having the output of one group to just fuck with other groups – not playing to win, but playing to have someone else loose. Why? Because that makes it easier for other resources you control appear to be making progress (even if it’s not the case). I’ve almost always seen this at the director and above level; managers just don’t have enough resources under their control to assign some of them to screwing with other folks, and it’s very, very rare that you’re overtly rewarded for screwing over the other guy. Okay, maybe not in Microsoft if you’re working for the “office” or “windows” franchise components. There’s another company (that I worked for directly – I haven’t worked at Microsoft) in which this pattern is abundant.

The end result of this cynicism (or realism, if you prefer) is that I have little patience with people whom I perceive as not “trying to make things better”. A bit idealistic of me, I know – a place where my idealism doesn’t match at all with reality. A little quote from a favorite movie (Secondhand Lions):

Hub: Sometimes the things that may or may not be true are the things a man needs to believe in the most. That people are basically good; that honor, courage, and virtue mean everything; that power and money, money and power mean nothing; that good always triumphs over evil; and I want you to remember this, that love… true love never dies. You remember that, boy. You remember that. Doesn’t matter if it’s true or not. You see, a man should believe in those things, because those are the things worth believing in.

The game is Hnefatafl – not quite an ancestor of chess, but somewhat related. The board is 11×11, and the game is unusual in that it’s an asymmetric game. It’s harder to win as one player than the other.

I first learned about this game through a book (the Mabinogion) that had a reference to a game called Gwyddbwyll (from ‘the dream of rhonabwy’ – see how this crazy stuff all ties together?). Nobody really knows the details of gwyddbwyll, but there’s some reference to say its related to other games from that era – Irish and Welsh, all of which seem to be the asymmetric “save the king” tafl games.

Right now I’m building this on iPad and iPhone, but I’m not sure if I’ll release it under the iPhone – only because the squares are so damned small on that interface. I have a few ideas for how I want to make the interface function, and once I get beyond that I’m going to aim to make it a turn-based multiplayer game using Apple’s GameKit (just an excuse to use the new stuff, course). A multi-player only game seemed kind of limiting though, so I purchased a book on game AI (Artificial Intelligence for games) so that I could make a reasonable single-player game. It’s a darned good book – so we’ll see how I do implementing some of those algorithms in Objective-C. I bought the whole book for Chapter 8 (Board Games), and don’t regret a penny of the purchase.

We have unit testing up and running, and like most projects a few of those tests are more “functional” or “integration” tests than actually unit tests, but really the project’s been pretty darn good about keeping those separate. As it turns out, the Junit framework is truly awesome for fast, quick tests – but when you start to step into functional testing you want just a touch more linearity to the testing. The unit test frameworks that I’ve run into (nosetests, unittest2) don’t have much in their mechanisms that allow you to set up longer, repeated sequences or flows, so it was time to look around for what else might work there. (sidenote: If you know of some mechanisms that do longer, sequential tests and are based around unittest2 or nose, I’d love it hear about it! I’ll freely admit to not having searched every corner there…)

I’ve been watching and reading quite a bit on BDD and the cucumber framework for testing applications, so I thought it was time to try it out. Of course, being more of a python geek than a ruby geek, I’d heard about Lettuce, which is either a straight up port, or incredibly headily inspired by Cucumber.

Getting started with it was incredibly easy. Everything we’re doing is working from a virtualenv (which I highly recommend for development!), so I just added lettuce to the pip-requires list and it was pretty much ready to go.

In setting up lettuce, I probably spent as much time knocking together a quick “functional_tests.py” script as anything – making it responsible for the sequence of testing. The whole point there was to make a single script that could be run, returning an exit code that meant success or failure for the whole set, so it could be plugged into our local instance of Jenkins.

The core of that fabric script is really just simple fabric commands to set up, clean, and push code to a remote node. I ended up adding a script to our local repository to start up services on the far-end for functional testing – testsetup.sh – that runs the various pieces under “screen” – which makes it super easy to see what’s happened when something goes awry.

You’ll notice at the very end of that functional test script, I invoke “lettuce” on the local machine. That takes care of running the actual test sequences. The documentation for lettuce is pretty reasonable, so I recommend walking through that tutorial for a quick understanding. The super-basic gists is that lettuce uses a DSL that you basically create as you go along, and ends up being rather highly customized to your testing setup. You put together files in a “features” directory, and they use “steps.py” in that same directory to do what’s needed. The pattern that I’ve been using (and which seemed to be implied) was to use the “world” object in lettuce as a sort of black-board to get and store data as you walk through the steps in lettuce. There’s deeper integration into Django based projects (my example was using a little Flask based application run with gunicorn).

The key to setting up the functional testing was taking advantage of the file “terrain.py“, which lettuce uses to do global setup, and setup/teardown around steps, scenarios, features, or the whole shebang. Turns out I had several options for how to reach out and walk through the remote application, so I chose to use twill, although I could clearly have used the local test_client() with Flask, or even the web_driver setup from Selenium to drive Firefox.

One of the nicer things that I found while walking through setting this up was that I could write up a feature file and run lettuce, and it gave me really pleasant “this isn’t implemented yet” output that I could use to expand steps.py. For example, I created “example.feature” in the features directory:

and then ran “lettuce” and got the following output:

As you can see, it’s basically giving you the code to add to steps.py to make this work. As I went, I found myself refactoring steps.py to use some of lettuce more advanced mechanisms – regular expressions and grouping steps together in sequences. And to make sure I don’t leave you completely in the dark about setting up and using steps.py, here’s a snippet from my growing file.

This morning, I found a link that adds a new dimension into place: Raspberry Pi. Wikipedia has a good description of the details of the device, and there’s a BBC interview that shows it and gets into a nice overview.

It’s the price point of this device with the capabilities that changes the game a bit. It’s a computer the size of a tin of mints, with a processor capable of easily running higher level languages – python, ruby, java, whatever. There’s a lot out there now (Arduino) that is a roughly similar cost and form factor, but programming it requires working in C or C-like languages and some tools to get it all together.

Raspberry Pi BBC Interview:

While the interview and foundation talk to making low-cost computers available to kids, I think this kind of device has a much broader potential in personal computing. The tech specs of this device make it very similar to the computing power available in the iPad or iPhones. And while the first generation of this device is missing some things I’d like to see default (a networking port), it’s easily added with a USB hub. The device is also incredibly low power – so I could imagine a shoe-box sized container of these running a 10x the amount of compute, and generating less heat (using less power) than your average dell desktop.

For an always-on application, I want something that’s not drawing a lot of power and could conceivably run unattended in the closet all day and night. This enables it at a new, lower price point (~$25 compared to a Hawkboard SBC at ~$90). Now that’s cheaper (for the moment), than running an instance up on a cloud server…

As I got into the game, noodling on the design and how the interaction would work, I came to the realization that I really would like to have a reasonable single player game as well, and that I didn’t have a clue about how to do board-game AI to make a good game experience.

After looking around through a pile of game AI books, I came across “Artificial Intelligence for Games, Second Edition“. Heavy book, hardback – some $80 if you pick it up outside of Amazon’s discount mechanisms. It looked good after a glance-through and minimal reading, so I took it home to dig in depth. I’m really glad I did!

This book does an excellent job of describing the AI algorithms in pseudo good and good ole english, as opposed to many which seem to be littered with mostly un-parse-able fragments of C++. They reference a number of modern games, but rather than catalog “these used this type of AI, the others didn’t” (which I saw in a number of books), they took some time to explain the tradeoffs in choosing the AI mechanism in the game, even if they didn’t have the “why” of the game designer to pull from.

I liked it enough to put up a review on Amazon, and hey – I’m writing this too. If you’re looking for a good overview of AI algorithms and how they work, I recommend the book.

It all revolves around the development toolchain, and the base components to cobble together full desktop-UI quality applications. This happens to overlap extensively with the server side world too – but the details started really becoming clear after digging a bit into Sencha, SproutCore, and JavascriptMVC. And it’s Javascript MVC that stands out to me.

If you look at the home page for the site, it presents itself as a collection of tools and mechanisms to do a full-bore standard application:

• core generators
• dependency management
• build scripts
• testing
• templating
• code cleaning and linting

and the list goes on. All of those components in javascript, for javascript. It’s still incredibly fragmented as a language, but the parallel efforts moving forward to drive automated testing, including unit testing on the server side. Node has it’s own built-in stuff, but there’s also a lot of effort to bring QUnit from the JQuery world into that same server side world.

Python and Ruby still have a dominant hold on the server side development, but I wonder for how much longer…

I’ve been following SproutCore for a while, sort of thinking about that as an interesting mechanism, and then more recently I saw the announcements about the jQueryMobile framework setup. Having done some iPhone and iPad development, the common metaphors of list views, tableviews, etc. felt very familiar. jQueryMobile isn’t all that to be perfectly honest, but the basic design feel is reasonably close.

I made a branch on Eyes and started laying it out – and I’ve been going gangbusters since. Laying in the constraints of the framework has really made the choices much easier to work out. Not that I haven’t run down some dead ends and had to back out, but I feel like I have better sense of how the pages flow and work together.

And best of all, I have something that I’m comfortable will work on mobile devices, while not looking like crap on a larger view.

I’m still learning the various tidbits of the framework CSS stylings, and I’m already seeing some elements where I’m going to want to dig deeper. The only real struggle I’ve had is working with Ajax based forms submissions and debugging them when I bork it up. Doing so much javascript work and debugging is still pretty unfamiliar to me. I’m ending up with a lot of print(“..”) debugging on the back end, and alert(…) debugging in the javascript itself.

All in all, I’m very pleased with the effects and results of using jQueryMobile. I do rather wish I hadn’t left my jQuery book at work for the holiday weekend though – I’m certainly not a jQuery expect, and it would have come in very handy more than once already. Thank god for excellent online documentation.

I’m saying all this after a week with the OpenStack guys – both in design sessions and just chillin’ out. Focused, intelligent, demanding conversations scattered through the week with an amazing “no-ego” attitude presenting itself. Not that there weren’t some good ole technical “best way to do it” or “which is better” fights, but given the breadth of this project and the open nature with vendors lurking all around the corners – well, frankly I expected a lot more “special interest” to be clearly showing itself. Everyone at that conference was interested in making OpenStack better at every turn.

250 people, 12 countries, 90 companies/organizations – all that after 3 months from being publicly announced. And they’re going it without any prior structure – building up an OpenStack foundation, doing all the legal and community building, right from scratch. And yeah – that’s showing right now.

The first thing I see that will provide the biggest gains:

• “How do we all work together?”

Some of the best sessions were around “What does the status X of bugs mean” and talking through the development and release process. At this point I’m convinced the core folks are reasonably comfortable with LaunchPad (the platform the system is hosted on) – and being at the conference really taught me a great deal about how OpenStack is effectively using it. Prior, it wasn’t comfortable or familiar to me at all. The object store and compute (swift and nova, respectively) core groups are really quite separate teams, all trying to figure out how to get some common ground in re-using code, libraries, and even setting up documentation.

The second:

• “Show me it’s workin’, again and again”

OpenStack is quickly heading to be the kernel or core of a platform. You could see it in the twinkle of Eucalyptus’ eye when they talked about Swift (the object store), or chatting with the folks from Scalr or RightScale. The whole system is being built with API in mind from the ground up, and while there is some pretty good unit testing in play and continuous integration, it was clear that installing this sucker was a PITA – and the documentation to really pull that all together starting coming together in the documentation sprint and install fest at the summit. One of the “blueprints” of the design summit (i.e. “Things we want to do, and how we want to do it for the next release”) is to get some fully automated integration testing as well as track the metrics on how the system is operating. There were a lot of folks that have some cross over into the Drizzle project, and the ideas of running and tracking benchmark data on every revision is darned power.

Add to that the benefits of a constant flow of functional testing against a couple of pre-defined clusters of both compute and object store, and you have a powerful engine to make sure trouble is spotted early and can be resolved quickly.

The third:

• “How’s this thing tick?”

One of the admitted weak points is that some small, damned effective core teams have done most of the work – and if you want to understand the system, well… you’ve just got to read the code. That is a huge investment – and frankly a barrier to entry into the project that can be avoided with some effort towards docs and discussion. Again, great progress was made there (I learned what the “project” concept was in Nova at the summit) – but the interactions between components, what the components are responsible for, and what they’re *not* responsible for, are all kind of tricky to learn right now.

This extends down into digging into the code, where docstrings could be better (and are getting better!) so that if you wanted to go help with something specific, you didn’t have to grok a broad codebase to get a handle on what the impacts are of the changes you’ll need to make.

And the last thing I’ll throw in here:

• “What OpenStack isn’t, or won’t do…”

The project is still in a lot of flux. There were some great components that were shown off at the summit that ride over the top of the infrastructure, or work with it through APIs. Should those be a part of OpenStack, or on the side? Some service providers were very interested in more platform-kind of elements – a common logging infrastructure, a common authentication, ID, and authorization infrastructure. Should that be a part, or on the side? How tightly or loosely do we want to couple some of these elements? The philosophy is there and forming up, but the real truth of it all will be over the next 6-12 months of the project when decisions are made, reviewed, and a core forms out of it. There have been a few architectural decisions made early: “Don’t mandate anything in the client”, “If a feature would restrict scale, it MUST be optional”, etc. that I absolutely applaud. I think it will form up more as projects apply to join the OpenStack umbrella and either make it or don’t. It will become clear what’s common, and what isn’t, pretty darn quickly.

I’m pumped about this project, the people, and it’s future. The core openstacker’s have clearly been driving up a steep hill to get to where they can level out a bit and move into more of a marathon mode. Really, it feels like we’re nearly at the top of that first hill.

The way I’m thinking about it, GAE probably isn’t the right thing. They don’t easily let you run repeated cron jobs, making an agent style application (i.e. proactive, rather than reactive) not quite a good fit. So what would be? A VM. Sure, you could slice it down farther and get a slicehost style thing – they’re excellent, but many of them have restrictions about “long running processes” – and that is exactly what I want.

So what’s it take to run a VM? There’s an interesting question. The cheapest I found was Rackspace – that offered a 256MB Linux hosting solution for $10/mo. Linode starts at 512MB for $19.95. Of course you need to at least make a vague attempt to compare to an EC2 instance. The always on really bites you there – the cheapest I could get my numbers was around $6/mo (a single micro instance, 1 yr commitment – minimal to no bandwidth).

I have to wonder – how well would something like that be received? Say you could offer a private agent/proxy service for $10/mo. I suspect you could get it cheaper – the compute needed for this kind of service would seem to smaller than the minimum you can purchase as a slice of compute. But people aren’t generally buying like this – at least not today.

I think most folks who have a desire for something like this would be more likely to have a desktop computer at home they could dedicate to leaving on. That’s assuming they have an internet service to their house too – so the real cost is more than the hosted compute, but since people often have it anyway… I’m guessing they’d be more likely to want to just use it.

The closest to paying as you go for an application gets to the Platform as a Service – Google App Engine or Microsoft Azure. Frankly neither of which I particularly want because they both represent a sort of technology lock in. In development now is OpenStack – Rackspace & NASA doing some of that base line commoditization work to drive down the costs of running small VM slices. We’re already on the sweet side of Moore’s Law for the commoditization of compute resources – it’s just a few more years until it’s a complete no brainer.

I also like using the desktop and iOS applications for watching twitter. They’re generally really good – give me a nice sliding window of news. But they only “go back” so far. The essential problem is that twitter has a limit to the history they’ll let you easily interrogate for your feed. My feed is wide enough that I’m starting to miss some pieces, especially as more of it is coming from around the globe and different time zones.

In thinking about it, we I think I need architecturally is something that’s “always on” on the Internet that can grab and watch my feed. Or, more efficiently, accept a twitter stream as events if they have a pub/sub model that is user specific (I don’t think so, but I didn’t look).

To me, that means leaving a desktop or something on all the time – kind of a pain, really. It’s way more compute than this little task needs. And my head is into cloud computing and various infrastructures – so what would it take to have something running – application as a service if you will – that I could cobble to do this for me? I think it would be pretty easy to arrange this with Google App Engine – and that feels roughly about right. A platform as a service engine – I can write and lay down the code on top of it, and then access it from the web through a browser or specifically out of an iOS application. I suppose someone could do this on a slicehost or other thin virtualization platform as well. This could all go into a dedicated JEOS VM and run on Amazon too.

What we’re down to is what’s it cost to run this sort of thing on a regular basis. Always on applications – can you do it cheaper “in the cloud” than you can at home?

And this whole thing is getting darned close to the concept of a personal agent – dealing with (aggregating, filtering) higher speed feeds on the internet than I want to dedicate my attention to. If you go beyond twitter, there’s a whole pile of social networking feeds that I review and look at with a variety of tools today – Netnewswire, browser, other dedicated applications… but all about consuming, filtering, and aggregating information I’m interested in. I get a heady feeling looking forward at how we might use it all.

The amazing natural beauty is what I went for, and definitely what I received. The bass note for the tour was news that my grandmother, Lela, was going downhill fast and was in the hospital (received just after we left Seattle), and passed away when we were in Juneau.

She was an inspiration to me in so many ways – grew up in Burlington, Iowa, stopped at a 6th grade education, lived through the depression, World War II, survived two husbands, and I think that shows she’s the most adaptable – bought her first computer 10 years ago in 2000, even had a facebook page and presence. I abused people complaining about “being too old” for computing with her accomplishments. Her passing wasn’t really unexpected – not at 97 years old, and when I visited her this past June she was definitely looking very frail. Aside from being very tired, and the various physical complications of just being old, she was sharp minded and had a great laugh through it all.

I got back from this all last week, had a day off at home, and then back to work for a couple of days. Now the weekend, where I’m writing this and reflecting a bit on the past two weeks. It was a really good vacation, definitely well timed, as the upcoming weeks and months are going to be darned busy.

Economic pressures being what they are, it is getting more effective to own what you absolutely need of infrastructure, and rent the rest. Shoot – if you’re small enough, owning anything at all for infrastructure may not make sense.

The problem comes when you want to start moving between those network oasis’ of happiness. Like, oh say, to a cloud provider. The end result is our services have all become addicted to this concept of secure, high-bandwidth, reliable access between tiers – that happy oasis. And that addiction is one that isn’t healthy – at least not in a world of elastic scaling architectures. We’re getting used to breaking that addiction when we integrate with external services – facebook, twitter, etc. The mashups are breaking the mold of how this has been done in the past. And we need to break our addiction!

Why am I asserting this? As I look at a large number of applications, they don’t fit a “cloud provider” very well – at least not when you start to get into the realm of dynamic or elastic scaling. Most providers have something akin to an “internal network” which we can leverage as consumers. As we get to the logical conclusion we will find ourselves wanting to shift work between one “oasis of network happiness” to another. The solution today? Expand to fill all available resources and then buy some more at the same place.

That ain’t gunna to cut it.

For us as consumers of infrastructure of a service, we want one provider to be as good and flexibly as another. That means commoditization of the infrastructure (which I believe we’re starting to see now, although the infrastructure providers will fight it tooth and nail). That means we need to be able to shift from one to the other at a moments notice.

Here’s an example:

You have enough resources for 100 units at provider A, and you are running up to a high level usage that looks like it’ll exceed that. You call up Mr Infrastructure A, who reluctantly informs you “Sorry, all sold out – it’ll take 6 weeks to add capacity”. You also happen to have a not-quite-as-good deal with provider B that costs a little more. All sounds good so far – your uber-cool software provisioning system smacks down a couple of VM images and spins them up…. except it’s a got a problem: connecting the stuff running at provider A to the new stuff you’ve just spun up.

And that’s assuming that you’ve solved all sorts of somewhat evil configuration problems knowing to indicate to a component what it is and who it should be talking to to get its work done. That’s a topic for a different post though.

I don’t have THE answer, but I do have AN answer. Follow the mashup leaders: take the OAuth & REST pattern back into your office, workspace, colleagues, whatever. That’s what we’re doing after all – mashing up some services. It’s just that we normally think mashup means “with someone else’s stuff”.

OAuth and REST work together beautifully and with some coordination can be the answer to providing the answer to the security question “Should I allow this external request to access these resources?” The OAuth 2.0 spec has a segment in it – section 1.4.4 (version 10) – that walks through the flow for this to happen. Routing over SSL does a pretty reasonable job of getting the data encrypted as well. The cost: running an Authorization server.

(You can pull off this same trick with OAuth 1.0: the not-quite-standard-but-defacto “2 legged OAuth” routing. The problem: not everyone and all the libraries agree on how to make that work seamlessly.)

In order to allow our ourselves to start treating everything – infrastructure as well – as software with it’s speed of change we need to be able to dynamically allocate resources. Once we have services that are all chatting across OAuth authorized links (encrypted or not), we’ve removed a huge impediment to being able to elastically scale our services.

Yeah, configuration. Kind of bitch, ain’t it. Another post, eh?

The thing that we’ll label as a “CMDB”, to me, isn’t. It isn’t a configuration management database. For us, it’s an inventory of assets – digital and physical. It’s a metadata store that allows us to assign ownership, and an ancillary data set that makes categorizing incidents, requests, and changes in the classic service management sense a bit easier across a very wide organization. If any small company came to me and said “yeah, gotta have a CMDB!” I’d be looking very closely at how potentially insane they were. Most small orgs and companies just don’t need it. It’s honestly only useful when you breach some amount of scale.

The worst part is the ITIL definition of what should be in a CMDB has been effectively unachievable because of the costs associated with it. The classic ITIL world of CMDB has this data repository being updated with process (typically manual) as changes are approved – it’s meant to represent the “desired state” of an operational world. Only it doesn’t. Really, it never has. And even with the highest priced tools on the market today never will. At best it’s an audit-against tool that you can see “yeah, it matched or didn’t match when we ran that scan a few days ago”.

It doesn’t have to be this way though. What most of us want from a CMDB is what we get implicitly, to some degree, from many of our monitoring solutions – a digital map of our environment. The monitoring pieces created their own version – typically configured by hand, or sometimes configured automatically with a hand to help guide (Zenoss and Hyperic both do a pretty good job at this). The monitoring systems then use that data model to know who to alert when something goes wrong, or if they’re really good – to share some set of analysis around “service X is down because the component A that it relies on is down”.

Virtualization is pushing this all right over a critical tipping point. The “old” CMDB is dead, lets jump to the new. We need a model of our environment that maps our physical and digital assets. We need it to show us dependencies in an ever increasing world, and we need it to help inform us – especially in a larger organization, who to contact if there’s an issue with a service. If we have to fill out all this data and information by hand, we’re lost. The rate of change is increasing, and we *want it* to increase. Look to the model of continuous deployment, the natural successor to the software development process that is continuous integration. Now in classic #devops style, let’s apply that right on through and into operations and running our services.

What doesn’t exist today, in our collective musings about a DevOps toolchain, are (currently) the tools to integrate the knowledge that the deployment tools have into updating a digital asset model . Even these tools don’t know the dependencies (i.e. what database is this rails app using, which memcache server/port combo is being used, etc) – but it’s there, just slightly under the covers.

The other place where we want/need this knowledge stashed? Our monitoring systems. The continuos tests against our live services to assert they’re OK. Many open source systems include some level of a model just implicitly in their configurations. Nagios, Munin, Zenoss, Hyperic, etc. I am still struggling to find monitoring systems that have the concept of dynamic configuration through an API access built in to the base of them. Still – much of their configuration is something that we might naturally want to store in a map of our services.

The way to get this data? Drive it from the tools that are implementing the changes. Have it as a service that can be updated and modified through simple API’s that Buildout, Func, Capistrano, Fabric, ControlTier, or whatever can access and inform. Use the manifests and details that the system configuration tiers (BCFG, cfengine, Chef, Puppet, SmartFrog) have been built with to populate this map as they deploy and invoke their services.

This is all a step to moving all of our infrastructure, historically so very physical, into the digital world. There are tremendous efficiencies to be gained – both financially (using our physical assets more effectively, or just using what you need from an existing infrastructure provider) and from a service perspective (being able to reconfigure and deploy your services to match the market needs).

Much of devops is focused on deployment, because that’s where we spend most of our time today. That’s good – but we can not forget that it is just one small part of the overall process for these services from inception to retirement.

…

And before any of the classic CMDB folks find me and start shooting, yes – I’m very aware of the work that is going on at CMDBf around federating CMDBs. The idea there is good – they’re heading in the right direction. At a 10,000 foot view they’re going the right direction with their standard. The foundation that it is built on is – in my opinion – now outdated and needs to be revisited. The implementation needs to be simplified. I would recommend you look at the labels of the members that are coming together around the federated CMDB concept. Do you see anything in there that shouts of “open, adaptable, flexible”? I don’t. I see the same kind of collaboration that led to J2EE standards and the W* standards around “web services”. What is needed is something simpler, more open, and with publicly available implementations. I would never expect BMC, CA, HP, IBM, or Microsoft to help provide that – it’s just not in their best interests when they have revenue tied to the services and software they provide in this space today.

Since I started using WordPress several years ago, it’s really come along solidly. I upgrade it these days directly from subversion. My hosting solution doesn’t have FTP access, which makes upgrading the plugins a bit trickier, but all in all it’s been really slick and clean for the using.

It’s no secret that most of the conference focused on the iPhone and iPad – what they’re now calling “iOS” (and yeah, I thought “Hey, doesn’t Cisco own that brand name…” when I first heard it). The desktop/laptop Mac OS X didn’t get much play time, the iOS operating system instead getting the red carpet treatment.

The engineers at Apple have managed to build, update, and ship an entire OS for specific platforms once a year for the past 3-4 years. I’ve got imagine that they’re nearly, if not completely, mentally bust at this point. I’m not at all surprised that “iOS 4″ won’t be out until Fall for the iPad – I suspect they’ll call it iOS 4.1 or 4.2. Apple’s not growing like mad (at least that I can tell), which means to me that they have been and continue to push at a fevered pitched for the accomplishments they’re making. I hope they don’t push so hard they fry their best and brightest that make up the core of the culture that’s doing all this impressive lifting.

The “other focus” of the conference was around new developer tools – Xcode 4 (developer preview) is quite a bit of engineering and accomplishment. To me it’s centrally a fulfillment of an idea that Chris Lattner presented and posed for possible futures in an LLVM session two years ago. It is more than that, and I’d love to wax a bit more enthusiastic about it, but I’m afraid that runs into NDA covered territory, so I’ll stop here on that topic. I did come away from the conference having submitted more enhancement requests than ever before around that tool chain.

Dr. Michael Johnson (@drwave) gave a talk in the middle of the conference, impressive in display and content, and really talking to the “why” and “how” of making tools. The message I took away this year from his talk was akin to an artist extolling “know your medium”. He didn’t outline the pros/cons of each medium that he works in per se, but just expressed what he could do in each and why knowing that medium was important. I’m still reflecting this year’s talk off the meme that he was touting a year or more ago (and was reflected a bit in this year’s talk too): “using tools to remove the tension in the room”. I guess it was two years ago at WWDC, we talked a bit. He related a viewpoint that “tools shift power” and was being very mindful of wanting to shift it in a way that was ultimately positive, not negative. Being a bit of a tools guy myself, I find myself reflecting on that quite a bit. The real bitch is that it’s often hard to predict which way the “right way” is between organizational politics and getting our jobs done.

For all the cool and amazing stuff that I saw at WWDC, I’m finding that I’m not walking away from this one inspired, scheming, and planning for something new, cool, etc. Oh, there’s the idea for the iPhone or iPad client application to this or that, but really I find myself instead thinking about my day job.

This week has given me some much needed time to step back and away from the office. I deleted my mail accounts from my iPad and iPhone to make sure I wasn’t even tempted to check them (those red badges can be like a red flag to a bull when you’re a somewhat compulsive person). What are we really focusing on and what difference is it going to make? How do we push or pull the whole organization forward to make it better? I have ideas. Some “maybe 60-70% right” ideas but no definitive “I’m 100% sure of it” answers right now.

Abby is 16, and was going for a record: youngest solo around the world. Pretty damned impressive.

At the article in the Seattle PI online, the first comment was made by someone calling themselves “Bobert”. He wrote:

I don’t know who this person is, but whomever they are – they symbolize something that is very, very wrong with our society today. Preserve and protect at all costs. What the fuck have we come to? Our rites of passage are down to “driving”, “graduating from high school”, “getting laid for the first time”, and sometimes “graduating college”. What happened to Thomas Jefferson’s american ideals – where the hell did we loose them? Maybe what passes for mainline culture thinks a rite of passage is a barbaric, untidy thing. It’s “dangerous”, and not for those who haven’t “proven themselves responsible”. They might get hurt.

How the hell do you prove you can be responsible in this day and age? By looking down, fitting in, not making waves, and getting by? We need to be looking up and out, trying things, and reaching out for our limits. We need to think, plan, and move ourselves forward, not just follow along.

I don’t know about being in the Indian Ocean during the winter months down there, but to my mind Abby was taking a rite of passage. Also trying to get the youngest-solo-around-the-world record is neat, but to me somewhat irrelevant. She was going out and doing something that I’m sure has touched her in some significant ways, and none of the externally visible.

One of the best things I did around College was go one a four-month walk-about in Europe. By myself. Right after college, I got a backpack and my parents got me the tickets and a Eurail pass. I was off. I think that was my most significant rite of passage. Being by myself for months, in countries where I didn’t even speak the language, teaches you something about yourself.

So to the Bobert’s out there – wake up and fucking smell the coffee. We are not going to make a difference in the world by being sheep. Going on a rite of passage, by whatever you want to call it, should be praised for the effort, not belittled.