Most reasonably (and larger) sized operations organizations have a pretty standard networking setup – or at least some close variation on the theme. ARIN wants public IP addresses behind load balancers, so most orgnizations front up their services through software or hardware load balancers. From there, it goes back to a highly response “web tier” – the spewing of content and the CDN source systems. Those back in to application tiers, and behind those are data persistence tiers (typically, your classic RDBMS). The only thing on the internet is often those load balancers. The networks are often segmented between app and data as well, often with firewalls, to “reduce potential intrusion”. It’s a good plan and pattern – it generally works. And while you own it all, you own your own network oasis of happiness.

Economic pressures being what they are, it is getting more effective to own what you absolutely need of infrastructure, and rent the rest. Shoot – if you’re small enough, owning anything at all for infrastructure may not make sense.

The problem comes when you want to start moving between those network oasis’ of happiness. Like, oh say, to a cloud provider. The end result is our services have all become addicted to this concept of secure, high-bandwidth, reliable access between tiers – that happy oasis. And that addiction is one that isn’t healthy – at least not in a world of elastic scaling architectures. We’re getting used to breaking that addiction when we integrate with external services – facebook, twitter, etc. The mashups are breaking the mold of how this has been done in the past. And we need to break our addiction!

Why am I asserting this? As I look at a large number of applications, they don’t fit a “cloud provider” very well – at least not when you start to get into the realm of dynamic or elastic scaling. Most providers have something akin to an “internal network” which we can leverage as consumers. As we get to the logical conclusion we will find ourselves wanting to shift work between one “oasis of network happiness” to another. The solution today? Expand to fill all available resources and then buy some more at the same place.

That ain’t gunna to cut it.

For us as consumers of infrastructure of a service, we want one provider to be as good and flexibly as another. That means commoditization of the infrastructure (which I believe we’re starting to see now, although the infrastructure providers will fight it tooth and nail). That means we need to be able to shift from one to the other at a moments notice.

Here’s an example:

You have enough resources for 100 units at provider A, and you are running up to a high level usage that looks like it’ll exceed that. You call up Mr Infrastructure A, who reluctantly informs you “Sorry, all sold out – it’ll take 6 weeks to add capacity”. You also happen to have a not-quite-as-good deal with provider B that costs a little more. All sounds good so far – your uber-cool software provisioning system smacks down a couple of VM images and spins them up…. except it’s a got a problem: connecting the stuff running at provider A to the new stuff you’ve just spun up.

And that’s assuming that you’ve solved all sorts of somewhat evil configuration problems knowing to indicate to a component what it is and who it should be talking to to get its work done. That’s a topic for a different post though.

I don’t have THE answer, but I do have AN answer. Follow the mashup leaders: take the OAuth & REST pattern back into your office, workspace, colleagues, whatever. That’s what we’re doing after all – mashing up some services. It’s just that we normally think mashup means “with someone else’s stuff”.

OAuth and REST work together beautifully and with some coordination can be the answer to providing the answer to the security question “Should I allow this external request to access these resources?” The OAuth 2.0 spec has a segment in it – section 1.4.4 (version 10) – that walks through the flow for this to happen. Routing over SSL does a pretty reasonable job of getting the data encrypted as well. The cost: running an Authorization server.

(You can pull off this same trick with OAuth 1.0: the not-quite-standard-but-defacto “2 legged OAuth” routing. The problem: not everyone and all the libraries agree on how to make that work seamlessly.)

In order to allow our ourselves to start treating everything – infrastructure as well – as software with it’s speed of change we need to be able to dynamically allocate resources. Once we have services that are all chatting across OAuth authorized links (encrypted or not), we’ve removed a huge impediment to being able to elastically scale our services.

Yeah, configuration. Kind of bitch, ain’t it. Another post, eh?

I work in an environment that has an existing CMDB. Over the past year, I’ve spent a fair number of man hours from my team and an equal number of hours of my own thinking about what it is, what we want from it, and how so much of what’s available today just doesn’t cut it.

The thing that we’ll label as a “CMDB”, to me, isn’t. It isn’t a configuration management database. For us, it’s an inventory of assets – digital and physical. It’s a metadata store that allows us to assign ownership, and an ancillary data set that makes categorizing incidents, requests, and changes in the classic service management sense a bit easier across a very wide organization. If any small company came to me and said “yeah, gotta have a CMDB!” I’d be looking very closely at how potentially insane they were. Most small orgs and companies just don’t need it. It’s honestly only useful when you breach some amount of scale.

The worst part is the ITIL definition of what should be in a CMDB has been effectively unachievable because of the costs associated with it. The classic ITIL world of CMDB has this data repository being updated with process (typically manual) as changes are approved – it’s meant to represent the “desired state” of an operational world. Only it doesn’t. Really, it never has. And even with the highest priced tools on the market today never will. At best it’s an audit-against tool that you can see “yeah, it matched or didn’t match when we ran that scan a few days ago”.

It doesn’t have to be this way though. What most of us want from a CMDB is what we get implicitly, to some degree, from many of our monitoring solutions – a digital map of our environment. The monitoring pieces created their own version – typically configured by hand, or sometimes configured automatically with a hand to help guide (Zenoss and Hyperic both do a pretty good job at this). The monitoring systems then use that data model to know who to alert when something goes wrong, or if they’re really good – to share some set of analysis around “service X is down because the component A that it relies on is down”.

Virtualization is pushing this all right over a critical tipping point. The “old” CMDB is dead, lets jump to the new. We need a model of our environment that maps our physical and digital assets. We need it to show us dependencies in an ever increasing world, and we need it to help inform us – especially in a larger organization, who to contact if there’s an issue with a service. If we have to fill out all this data and information by hand, we’re lost. The rate of change is increasing, and we *want it* to increase. Look to the model of continuous deployment, the natural successor to the software development process that is continuous integration. Now in classic #devops style, let’s apply that right on through and into operations and running our services.

What doesn’t exist today, in our collective musings about a DevOps toolchain, are (currently) the tools to integrate the knowledge that the deployment tools have into updating a digital asset model . Even these tools don’t know the dependencies (i.e. what database is this rails app using, which memcache server/port combo is being used, etc) – but it’s there, just slightly under the covers.

The other place where we want/need this knowledge stashed? Our monitoring systems. The continuos tests against our live services to assert they’re OK. Many open source systems include some level of a model just implicitly in their configurations. Nagios, Munin, Zenoss, Hyperic, etc. I am still struggling to find monitoring systems that have the concept of dynamic configuration through an API access built in to the base of them. Still – much of their configuration is something that we might naturally want to store in a map of our services.

The way to get this data? Drive it from the tools that are implementing the changes. Have it as a service that can be updated and modified through simple API’s that Buildout, Func, Capistrano, Fabric, ControlTier, or whatever can access and inform. Use the manifests and details that the system configuration tiers (BCFG, cfengine, Chef, Puppet, SmartFrog) have been built with to populate this map as they deploy and invoke their services.

This is all a step to moving all of our infrastructure, historically so very physical, into the digital world. There are tremendous efficiencies to be gained – both financially (using our physical assets more effectively, or just using what you need from an existing infrastructure provider) and from a service perspective (being able to reconfigure and deploy your services to match the market needs).

Much of devops is focused on deployment, because that’s where we spend most of our time today. That’s good – but we can not forget that it is just one small part of the overall process for these services from inception to retirement.

…

And before any of the classic CMDB folks find me and start shooting, yes – I’m very aware of the work that is going on at CMDBf around federating CMDBs. The idea there is good – they’re heading in the right direction. At a 10,000 foot view they’re going the right direction with their standard. The foundation that it is built on is – in my opinion – now outdated and needs to be revisited. The implementation needs to be simplified. I would recommend you look at the labels of the members that are coming together around the federated CMDB concept. Do you see anything in there that shouts of “open, adaptable, flexible”? I don’t. I see the same kind of collaboration that led to J2EE standards and the W* standards around “web services”. What is needed is something simpler, more open, and with publicly available implementations. I would never expect BMC, CA, HP, IBM, or Microsoft to help provide that – it’s just not in their best interests when they have revenue tied to the services and software they provide in this space today.

Somewhat related to my previous post – I’ve been thinking a lot about packaging lately. At work, I’ve jumped into a team that’s responsible for a whole bunch of infrastructure – envisioning it, building it, and running it.

Two months ago I thought I’d jump in and start building some really awesome spires. Spires in a metaphorical sense anyway – I guess if I was really sticking to an architectural theme, it should be something else – something more broadly useful but still fantastically impressive. I found myself focusing on the infrastructure instead. The infrastructure wouldn’t support what I wanted to build. It’s missing pieces – pieces at the very base of the whole setup. So I’m mucking around in the foundations, shoring up some systems, adding others – getting the basics nailed down with my team to build some of these cool spires. And thinking. A lot of thinking – thinking about what’s most critical, what provides the most value, and what needs to be done in what order to make this team really successful.

One of these foundational pieces is packaging – application packaging, to be specific. The more I think about it, the more I think I need to do something about it – but outside of the walls of my job. The reason for going outside my job for this? I honestly think the world needs another application packaging standard.

I could build a company standard for application packaging. I’m pretty convinced of it anyway… but I don’t think it would be successful in a long term (i.e. more than 1-2 years). For an application packaging standard to be really effective, it has to have a broad support base, will require a lot of effort, and for an effective standard I philosophically believe it must be an open standard.

The use case that’s driving this thinking is being able to programmatically deploy new servers, drop down applications, and configure them in a very fast time frame (in a manner of minutes). The use case is one of the classic jet-pack and flying cars dreams of elastic/cloud computing.

There are some enterprise tool chains that do this today. Sort of. If you squint real hard. Most of these tools are born from the world of policy based computing, IT control and compliance, and configuration management. They don’t match up with the use cases real well, and the enterprise software solutions in this space and horrifically and tragically expensive. I can’t even blame them – clearly that’s where the money’s been for these tools.

The marketplace is pushing – there’s a need, there’s a HUGE opportunity here. It’s where applications become components that can be, and are, programmatically deployed and controlled. It’s the next step up in the abstraction layer for the systems we use every day.

There is some work towards solutions to this problem today. It’s not like I’m the only guy seeing this problem (if I was, I’d likely be delusional).

• RightScale has a great product that layers on top of Amazon’s EC2. It’s damned impressive.
• 3Tera has a tool chain that starts to deal treat virtual machines as packages to be deployed, connected, and replicated at need.
• Parallels has created AppStandard.org, which is really the closest anyone’s come to what I’m aiming at – but even it’s not quite there – not to mention copyrighted and proprietary.
• VMware is making some aggressive moves into the tools space as well, propagating the OVF standard (with the DMTF) and the concepts that we’ll be deploying VMs, not applications, in the very near future. I’m not sure they’re wrong – but I think they’re missing half the picture.
• The DMTF has standards that are skipping around the ideas, but not quite nailing them on the head.
• In the open source world Puppet and CfEngine are the kingpins – programmatically defining infrastructure. They fail to take the next obvious step into application deployment – but I think only because they lack the infrastructure components needed to do it in an effective way.
• At Velocity 2008 (which I attended last year) Adam Jacob gave the talk Building an Automated Infrastructure. They used these tools, and one of the points of the talk was the applications and their configuration weren’t yet well dealt with.

I think the solution to this problem is an application packaging standard that includes configuration and parameters into its metadata.

Like other packaging standards, the packaging needs to:

• allow the code to be dropped into place
• define dependencies that the application requires
• have a common and open format that can be stored and shared in open repositories

What it needs in addition is:

• the package should contain parameters (configuration elements) that can be programmatically determined
• once determined, the application should be able to be deployed with parameterized configuration – details updated at deploy time
• ideally this will enable the capability to update that same configuration programmatically, idempotently and remotely

What I think it needs to be successful:

• An open format with at least one, preferably multiple, implementations enabling that standard.
• A base set of common applications packaged in this format, available for immediate use
• a tool to create the packages and verify they’re ‘good’
• a tool to deploy the packages overlaying configurations – at least on a local basis
• a public repository of common packages to build a dependency library upon
• a way to copy/mirror that repository cheaply and easily in any organization to reduce cost of uptake
• a damned impressive example solution that takes advantage of the kit

In short, a crap-ton of work. For a company to want to go do these things, they’ve have to be pretty damn visionary. And I’d expect that they’d want to keep this kind of IP to themselves. I need this kind of solution, the economic times are rough to chase visionary dreams outside of the core competencies of the job, and I think a successful solution would benefit everybody. To me – this whole setup screams “open source project” to make it work.

So now I’ve defined it – put it out there.

Are you interested in making something like this? Contributing to a open solution? Willing to put some sweat equity behind a project that could revolutionize things, but is essentially building bricks for a foundation?

I have some projects on my plate right now that I can’t just drop – so it’ll be a month or two before I start pursuing this one. Let me know if you’re interested… haven’t even decided on the structure it’ll need around it to work, but I’m heading in that direction.

(image used with permission: http://www.flickr.com/photos/kamoda/3184396852/)

I’m taking some time today to read up on the events and talks at PyCon 2009 happening this past week in Chicago. I’ve been watching a few folks on twitter who are there (John DeRosa and Ted Leung). It’s been interesting to see what’s happening through their comments and links. There’s the inevitable talks that I wished I’d attended, hopefully slide decks will appear for a good number of those and I can catch up even after the conference content. PyCon is one of the best there – they go out of their way to make the talk content available as quickly as possible with the right set of controls for content (the speaker’s are the ones responsible for uploading their content).

Anyway, all this led me around to the conversations that have been happening around packaging with python. It was a topic of some conversation (and has been needed for a while now, in my opinion) at the Python Language summit (notes from Ted about it). Tarek Ziade did a survey recently of what folks were using, and posted his results, which were interesting. zc.buildout and virtualenv got some attention from multiple folks. I think the most crucial was the summit talks themselves, which Tarek summarized on the mailing list. The gist – there is a high-level plan at improving the packaging world around Python:

• standardize more metadata, especially including dependencies, and APIs for accessing the metadata and dependencies.
• there should be an API to get metadata for a package without actually executing any of the package’s installation script.
• this will go into the stdlib for Python 2.7 and 3.1
• it will be provided as separately downloadable backports for prior versions (maybe using a bootstrap not unlike ez_install)
• keep distutils, but start deprecating certain higher-level functionality in it (e.g. bdist_rpm)
• don’t try to provide higher-level functionality in the stdlib, but instead let third party tools built on top of these core APIs compete

When I was working more extensively with Django, that was the biggest headache – getting a packaged, ready to roll quickly environment. I’ve trailed off in the past while with the Django project (not actively using it at the moment, although I think that’ll likely switch around again) – but the lessons about packaging have remained with me.