Rhonabwy

I really like the concept of Pinax, but I’m beginning to see the darker sides of the project. Here’s the fundamental issue:

Pinax does a completely kick ass job of making a combined and base project that pulls together a lot of pretty darn good re-usable django “applications” into a single project. What it doesn’t do well is manage any of the dependencies in that tree, or make it easy to use only a portion of some app. If you want to use Pinax the way Pinax is already developed, you’re rockin’. If not, you’re re-working quite a bit of code. And the worst part is that code is almost all opaque to you and hidden in the pinax framework.

The application I wanted to pull in to my basic “pinax” project was “photos” – some simple photo uploading. Yeah, cool – uses photologue under the covers. What’s not clear is that photos also has dependencies on grouping and tagging, and if you review the dependencies documentation, it looks like there’s circular dependencies embedded in there.

What I ended up doing was to take the “photos” application from the Pinax project directly, fork that, and drop it into my project. Then I had to do a shit-load of renaming and stripping because “photos” is built in to Pinax as a default – so “photos” in my app became “photo” and I was scrambling through code to remove the bindings into commenting, threaded commenting, grouping, and so forth.

I did get it working, and really all told I spent less time stripping out stuff than I would have putting it together from scratch. But the only reason I was able to do it that efficiently is because I’ve already been through the pinax project’s source, had it on my laptop, and was a fairly comfortable and competent Django programmer. Any newcomer hitting this would have been toast, and badly confused.

When I asked around the Seattle Django user’s group related to Pinax, several of them said they’d been shying away from something like Pinax because it was easier for them to simply including the good open-source django apps into their work already. At the time, I thought “Well, maybe that just makes Pinax easier and more effective for newbies into Django.” Now I’m reconsidering.

I don’t have a good answer to the problem, and I’m sure I’ll continue to use Pinax and maybe even dig deeper and see what can be done to resolve this kind of issue. I think the idea of the Pinax project is fantastic, I just wish it was a little less “tightly coupled” under the covers and provided more transparency into it’s depths.

I have a few friends that I’m bringing up to speed pretty quickly on Django. The basics are going well, and I’m providing the foundations for the effort. To take advantage of all the good stuff out there, I’ve started diving deeply into Pinax, and then wrapping some automation around it with Fabric to make it really easy to work on new apps and projects.

That’s the whole reason I posted gist 284927 yesterday on setting up a remote app server for a pinax project with Fabric.

The biggest hurdle that I have with the fellows I’m bringing up to speed is them understanding what’s IN the project that I’ve already slapped down. Even a basic pinax project has a huge amount of complexity for the newer django developer – it’s pretty overwhelming. To try and mitigate that complexity, I’ve started a side project making Pinax cheat sheets. (http://github.com/heckj/pinax_cheat_sheets).

This is not intended to replace Pinax documentation, and in fact I hope that I can eventually tweak these around and put them into the pinax project. I have a huge respect for it (and I use it), so here’s a little bit of detail for contributing back.

I am very happy to take on any changes, edits, updates, etc – from others. There’s a lot of folks who’ve worked very hard to build all these components, and I’m not writing from the perspective of the creator, but of the “code spelunker”. I’m sure I’ll get some of this stuff wrong – so if you see something, please let me know.

I’d ideally love to receive contributions in the form of pull requests from GitHub. Just fork that project, make your edits, and when you’ve pushed them back up – send me a pull request. I don’t promise that I’ll include everyone’s contributions, but to be honest I don’t expect to be too darn picky.

I’ve written the cheat sheets in markdown, mostly because I know it, and secondarily because it renders nicely as HTML when viewed through GitHub.

The start of the project has the following cheat sheets up:

• django.contrib.auth
• django.contrib.contenttypes
• django.contrib.humanize
• django.contrib.sessions
• django.contrib.sites
• pinax.templatetags
• notification

I have a bunch more external apps to go before I hit the internal Pinax apps. I’m working through everything that’s included in a Pinax “basic_project”.

• django_openid
• emailconfirmation
• mailer
• announcements
• pagination
• timezones
• ajax_validation
• uni_form
• staticfiles

And then the internal apps:

• basic_profiles
• account
• signup_codes
• about
• django.contrib.admin

I spent the better part of this morning screwing around with Fabric to get even a basic deployment working correctly with Pinax. I may be doing this horrifically backwards… Since I finally nailed something that worked for me and I couldn’t find much out there on the web, I wanted to post up a result so that other folks could use/riff etc.

http://gist.github.com/284927

Please feel free to replicate/clone/use/etc whatever to your heart’s content. It’s built around deploying a Pinax 0.71 project clone that also uses South for schema migrations, and the project is housed in BitBucket.

The other fabric file I found that’s referenced frequently was a part of the inspiration here: http://gist.github.com/212366. It’s a fabfile that’s intended to provide some simple rollback and expects to be used from GitHub – and has some really nice documentation associated with it. Maybe I’ll work on mine to add some of that doc… but mostly I’m just making it available as is.

Ironically, the choices for an in-house integrated project management toolchain are, to my mind, significantly more limited than the straight up open-source market. I have been looking and watching several projects over the past months for something that I could use for my development team at work (i.e. behind a corporate firewall and not publicly available) to support a more open, transparent view of what was happening.

Trac, of course, is one I’ve used and really enjoy. Trac, however, doesn’t really support multiple projects in any integrated way – and we’ve got way more than one project. From the activity I’ve seen, it’s not about to any time soon. Variations on that theme include DrProject, and more recently Basie. They’re all interesting, but I didn’t see either DrProject or Basie as really ready for prime time and I wanted to be able to quickly grow beyond the basics. Two others that are obviously available are Redmine and Retrospectiva.

Ultimately, I’m heading down the road with Redmine for the following reasons:

1. It’s a decent and clean user interface that has several years of polish now around it
2. The system support plugins, and there’s several plugins that I really want to take advantage of
3. Redmine and Trac appear to have the most active communities around them and driving them

The story cards and burn-down charts from Retrospectiva’s AgilePM addition were darned compelling. But ultimately it has a more limited set of folks focusing on that project, and I think I can achieve the same effects with plugins that are available from the Redmine community. Many of these are hosted on GitHub, and even more are in development or some middlin’ shape at the same site. And finally, I’d much rather deal with a partially implemented plugin than a partially implemented core system.

If I could host our development outside of the corporate firewall, I’d likely be looking at Bitbucket or Firefly for the work. I like Bitbucket partially because it’s all mercurial based, and I need to support some Microsoft based development. Git will work with enough effort, but I don’t want to go down the road of trying to explain all the detail needed there. I love the functionality in GitHub, and frankly think it’s a slightly better and more aggressively updated solution that BitBucket, but the git thing – while I’m happy enough with it – just sinks me otherwise because I know I’ll end up spending hours attempting to support it on a windows environment, and I just don’t want that hit. Firefly is ActiveState’s “let’s take and commercialize Trac by hosting it” – and they look like they’ve done a nice job of it. The agile methodology plugins and burn-down charts are pretty darn compelling.

It should be noted that I also took a pretty deep look at Launchpad. Even though it’s open sourced, it’s a damned complex thing to potentially integrate and build up for internal use, and its integration/collaboration features just haven’t been as compelling to me as the machinery that GitHub or Bitbucket provide. The “branch and request a pull” feature set that supports the massive branching and merging for features, bugfixes, etc. is just so much easier in Github or Bitbucket – and the wiki support to the other components is also far superior.

If you’re looking for something yourself, definitely keep an eye on the market and projects though. RIght now it’s just December 2009, and it’s clear there is a lot of effort, momentum, and a ton of good ideas that people are just begging to implement in this space. If I go to make this same decision in 6 months, there’s no guarantee that the landscape won’t have changed dramatically and a different solution would be better.

I’ve been hacking away at a side project for the past three or four weeks – got myself to internal milestone #2 this weekend, for which I’m really pleased. The very tail end of this milestone was deploying the code somewhere and vetting that all the basic, mostly-crappy-still bits actually worked.

Since I’ve got it deployed now onto a little dev server, I was thinking about how I was going to make this work going forward. Database migrations being one of those “how do you upgrade things?” questions when you’re doing rapid development django applications where the database model can get twiddled a bit.

The result of scanning around this afternoon from my favorite coffee shop haunt led me to South. In about 10 minutes I’d enabled and moved to south and had my application up and rolling with it. My previous experience was with Django Evolution – which wasn’t bad, but it was more work to get it into place and running.

The migrations in place, I did some model twiddling and experimented a bit to see how it all worked. I can see how if you’re not being somewhat careful, you could get into some really horrible places that no program could figure out. But basic migrations – moving forward and backward – really do an excellent job of “just working” with this framework out of the box.

The one interesting quirk I ran into – I’ve been doing tests on my codebase with:

coverage manage.py test

which just caught and ran the tests for everything in the application list. Figured it didn’t hurt to test the admin and auth stuff as well. When I added south, which I installed using:

 pip install south

The directory that south’s tests expect to use suddenly becomes a bit less accessible on the build server. Not a bad thing to redirect the tests to just what I’m writing, but an interesting side effect.

Installing Hudson

I start with a basic virtual machine – in this case, I’m using Ubuntu Server 9.10. Once you have a basic machine installed, I recommend you make sure everything’s up to date with patches.

sudo apt-get update
sudo apt-get dist-upgrade


If you haven’t already, consider installing an ssh server for remote login. You don’t need it for hudson, but I find it makes administration easier.

sudo apt-get install openssh-server

To make installing python packages and pieces a bit easier:

sudo apt-get install build-essential python-dev

Start off by getting a copy of hudson.

wget http://hudson-ci.org/latest/hudson.war

Hudson runs fine from that war directly – a darned handy way to make the whole system work. You can also install this in Tomcat, JBoss, or whatever other servlet container you like to run. I find it easiest to run directly from the war file.

If you’re going using Mercurial or Git, you’ll need to install the appropriate version control client onto your machine as well. The plugins use those systems through the command line interface.

sudo apt-get install mercurial
sudo apt-get install git-core
sudo apt-get install subversion


Since hudson is a java application, we need to have java installed… Ubuntu 9.10 comes with OpenJDK which works just fine. You can also download and install the Sun JDK if you want to, but it doesn’t really seem to be required. If you don’t have any JDK already installed, you can generally get OpenJDK using:

sudo apt-get install openjdk-6-jre-headless

To get rolling with Hudson, just fire it up:

java -jar hudson.war

I just run this as the base user – no special privileges. It creates all it’s working directories, workspaces, etc in a “.hudson” directory. You can do the extra work to make this a unix daemon too – I haven’t bothered because it’s been relatively stable. There are some good notes on doing this at http://weblogs.java.net/blog/2009/02/10/hudson-now-good-behaving-unix-daemon and http://wiki.hudson-ci.org/display/HUDSON/Installing+Hudson+as+a+Unix+daemon, so I won’t dive into those details here.

Once you’ve started it, bring up the UI in a browser to make sure it’s all working. My virtual machine is running at 192.168.0.2, so I open up http://192.168.0.2:8080/. I’ll use this same IP address through the course of this walk-through. Of course you can replace it with a hostname.

I generally start off by making sure we have the latest Hudson installed and getting the various client plugins I want to use all installed – click on “Manage Hudson” (http://192.168.0.2:8080/manage)

If there’s a more recent version of Hudson available, it will show at the very top of this page and give you the option to download and install it. You’ll be able to upgrade plugins the same way – Hudson knows to look at it’s own site for the latest plugins and versions, so it’s very easy to maintain. If you’ve been following along with this, the “wget” statement up above will have likely brought down the latest version already, so you won’t see this sort of screen until a new update comes along.

The upgrade automatically mechanism has worked great for me, but a couple of caveats are needed here – the process needs to be running with permissions to the directory in which it’s located to work. The other is once you’ve upgraded the system, you may need to restart the service from the command line to enable future downloads. I’m not sure if I ran into a bug with that, or if that’s expected behavior – but I haven’t run into it often.

Now you should have the latest version – either directly or just upgraded. Lets get to the plugins – because that’s where a lot of the Hudson goodness really lives. You can get to the list of installed plugins so you can see what’s there by default. From the “Manage Hudson” page, click on the link to “Manage Plugins”. The URL for that page is http://192.168.0.2:8080/pluginManager/

There are four tabs here to help you see what’s available – the first tab Updates is the default, and shows any outdated plugins. The second is Available, that shows what’s linked up from the central Hudson plugin list, and Installed, which shows the bits that are already there. Click on the Available tab (url is http://192.168.0.2:8080/pluginManager/available) and you can see the plethora of interesting things ready to use. The ones I like to install for my python projects are:

• Cobertura Plugin
• Git Plugin
• Mercurial Plugin
• Monitoring Plugin
• Python Plugin
• Violations Plugin

All of these plugins will still require you to have whatever local libraries you need to make them work. That’s why we installed Mercurial, Subversion, Git, etc earlier – to make sure those tools were available. Hudson invokes them on the command line on your behalf to make it all work.

Below is a broad set of libraries that cover a lot of pieces – you may want to install more/different libraries to accomplish your aims with Hudson. I’ll be aiming to create some typical build reports with it all – unit test output, coverage, etc.

sudo apt-get install python-setuptools
sudo easy_install coverage
sudo easy_install pylint
sudo easy_install unittest-xml-reporting
sudo easy_install fabric
sudo easy_install nose


Setting up a new Job

Now we just need a project. Let’s start with an open-source project – Pygments is a great project, with lots of diverse things happening within it. And it’s got tests and will be a good general victim for this writeup.

Start out by clicking on “New Job” (http://192.168.0.2:8080/view/All/newJob). A Job in Hudson is a build process. I’m going to make a fairly simple one, but you can cascade multiple jobs together into a far more complex build process if you want to… I’m not going to delve into that here, focusing instead on getting a single python project rolling.

In the new job form:

• set the job name to “pygments”
• choose “Build a free-style software project”
• I also put in a description (it’s not mandatory)
• then choose “Mercurial” from the Source Code management section

• put in “http://dev.pocoo.org/hg/pygments-main” for Repository URL
• put in “trunk” for the Branch
• click the “Save” button down at the bottom

Now you’ve got a job – we can come back and configure this at any time, and we’ll do more tweaking as we go through this. Let’s make sure we have the basics all working first though…

At this point, you should be at the project pygments page in Hudson. If you’re not, you can navigate there from the front page, or go directly to the URL (http://192.168.0.2:8080/view/All/job/pygments/). The page should look something like this:

On the left hand side of the page are the controls for the job, including Configure and Build Now among a list of other options.

• click on “Build Now”

You should see the build start processing with a little rolling progress bar on the left hand side of the window. You can click on that link that looks like #1 Nov 5, 2009 8:35:18 AM’ in the Build History box.

Either immediately, or when the build reports complete, click on that link.

You can see the console output from the build (which is just pulling down the source) with the link Console Output. The output of the build is exactly what happened on the command line. In this case, you’ll see something akin to:

Started by user anonymous
$ hg clone -r trunk http://dev.pocoo.org/hg/pygments-main /home/ubuntu/.hudson/jobs/pygments/workspace
requesting all changes
adding changesets
adding manifests
adding file changes
added 903 changesets with 2247 changes to 269 files
updating working directory
249 files updated, 0 files merged, 0 files removed, 0 files unresolved
[workspace] $ hg log -r . --template {node}
Finished: SUCCESS

Once you’ve done an initial build, the files will also be available for you to look at and download through the Hudson UI. If you click on the link Back to Project or just the project name at the top of the page, you should see a new reference in the project called Workspace. Clicking into that will show you all the files that were downloaded and any other results of the build process. Right now we’re just getting files from source control, but as we add build steps to the process, the results will be available to dig around in at the workspace.

If you want to set up Hudson to check for updates and build when it finds them, you can go back to the project page and click on the Configure link. Once in there, scroll down in the configuration and enable the checkbox labelled “Poll SCM”. As soon as you do, another text field will come available that will let you specify a Cron-like string to identify when the builds should check source control. In the screenshot below, I’ve enabled Hudson to check the mercurial repository every 10 minutes.

…

Any time you change the configuration, go to the bottom of the Configure page and click on “Save” to set the changes.

Adding Build Steps – doing ‘work’

Right now, the build isn’t doing too much – let’s see about getting it doing some more work for us. The Pygments source code is set up with a Makefile to make doing this a bit easier, so we’ll take advantage of that. If you’re not in the Job configuration page – get there. Go to the job an click on the Configure link. Now we’ll add a build step that simple invokes the shell command “make test”.

Scroll down in the configuration page until you see the section labelled Build. Click on the button underneath there labelled Add build step, and choose Execute shell. When the text field for that script appears, put in the text make test.

Scroll to the bottom of the page, save the configuration, and invoke “Build Now” to run the updated build. Looking at the console output for the build will now show the updated console log with the running tests, like the screen shot below.

Should the built in unit tests fail, the system will show a red indicator on the hudson front page and next to the build results link.

Displaying unit test results

What this isn’t doing it making some pretty graphs and charts. Let’s see about getting some of that enabled.

Pygments uses Nose tests for it’s unit tests. That’s pretty nice, because Nose includes a mechanism to output the format of the tests into Java-based JUnit format, and Hudson in turn knows how to make pretty pictures from that format. Since we installed the nose library earlier, we can take advantage of it. Probably the “right” way to do this would be to change the Makefile to invoke the tests that output XML for the unit tests. For right now, we’ll just shim it into place with new build steps.

• Change the content of the Execute shell build step and replace it with
  python tests/run.py --with-xunit
• save the configuration and invoke “Build Now”
• now look in the workspace in you should find a new file called “nosetests.xml” in the tests subdirectory.

Let’s get that graphed…

• Go to the job configuration and enable “Publish JUnit test result report”. It will have a text field so that you can tell it where to find the results. It uses an Ant format for finding filenames in an existing directory structure.
• In this case use **/tests/nosetests.xml
• Save the config
• invoke Build Now

Now you’ll see a new element in the web user interface called “Latest Test Result” which you can dig into. If you invoke “Build Now” again, it will start graphing the results of the tests and making that trend available in the project view. Right now this graph is going to be really darned boring, because there aren’t any changes between builds. Once more code starts rolling in, you’ll see changes with the number of tests being invoked. You can also click on that link and see the tests that were invoked. For python folks, remember that we’re shimming Python unit tests into a JUnit conceptual framework, so there are going to be some “leaky abstractions here”. In particular, test names, classes, and such may not alway match up with expectations. I have more details on how to enable XML output for a Django based test runner that I’ll post in another write up… uh, later.

Add cobertura style test coverage reporting:

Now we get to use Ned Batchelder’s coverage library to see some cool stuff!

(This whole write up is basically a thank you to Ned for writing Coverage)

Go back to the configuration for the job

• change the Execute Shell build step to read:

coverage run tests/run.py --with-xunit
coverage xml


• scroll down and enable  the check box Publish Cobertura Coverage Report
• use the pattern **/coverage.xml

Add pylint reporting

Head back to the job configuration

• add a new Execute shell build step  to read:

#!/bin/bash
pylint --rcfile scripts/pylintrc -f parseable pygments > pylint.txt
echo "pylint complete"


• It’s important to start this with #!/bin/bash, and then end with the echo statement so that the return codes from pylint don’t get interpreted as a build failure (unless you want that… I don’t know about you, but I’m not pythonic enough to have a 100% pylint result.)
• enable the checkbox Report Violations
• under pylint, use XML filename pattern of **/pylint.txt

• save the configuration and invoke “Build Now”

So there you have it – a python project building under hudson with unit tests getting run, cobertura style coverage reports, and pylint style reporting.

A couple of years ago I created the original code behind project Django Queue Service on a bet with myself at the O’Reilly Open Source Convention. A few months later I popped the whole thing up on Google Code hosting and made it an open source project. A couple of fellows joined me in hacking on the code base to move it forward a bit – and one in particular (Rajesh) did an amazing amount of work moving the whole thing forward.

While I had some good original ideas, I didn’t keep the project moving. And the project has been effectively abandoned from my head for the past 9 months. So today I’ve moved ownership of that project to Rajesh. I’ve left myself as a contributor – but I’ve found that unless I’m actively using the code myself, I’m not likely to do anything to contribute into the code base – and right now the Django based work I’m doing is all pretty external to needing any queues to get stuff done.

I think I likely should have handed the project off some time ago – I didn’t do it any favors by just lurking on the code and not promoting it. I didn’t hold up any patches either – I just didn’t make it move forward. I think the project’s in good hands and I’m looking forward to watching the “really simple queue” space to see where it continues to go. There are some interesting alternatives out there too – Gearman, Celery (rides atop RabbitMQ), and beanstalkd.

Congratulations to the entire development team and Django community for making the Django 1.0 release today! It’s a monumental accomplishment, and I’m very pleased to see it here.

I’m not as active in the community these days, but I’m still using the project on a regular basis as one of my “secret weapons” to get things done quickly. It’s worth noting that there are some definite changes in the 1.0 release from even what’s in the various books now available for Django. If you’re heading in to the 1.0 release and can’t find something you expect to be there, make sure you check the online documentation at http://docs.djangoproject.com/ – it’s very complete and very worthwhile.

I signed up with Launchpad today – don’t know why I hadn’t really done it earlier, other than I just didn’t think to or have any projects that drove me there. The project du jour that lured me in was Graphite – a distributed high-performance monitoring solution written in python (and apparently using Django as well!).

(I haven’t tried Graphite yet, but the screenshots looked pretty nice – so I might be doing so to see what it’s all about.)

LaunchPad is Ubuntu/Canonical’s code hosting platform – similar to Google Code or Sourceforge in concept. They use Bzr for their source control, which is a tad odd to me – I’m more in the mercurial camp of any decentralized SCM – but their web site is really first class. They’ve got a nice system and setup – simple, somewhat bold iconography, obvious components to projects, and a clean look. As a place to encourage project and community participation, I admit that I was immediately drawn in. My own project (languishing at the moment) on Google Code (django queue service) has an interface that feels positively stark in contrast.

I’m not going to move anything over, but I suspect I will pay more attention to the LaunchPad setup going into the future.

Word is that the a version of the code review tool that Google uses internally (called Mondrian) on the Google App Engine – called Code Review. I remember first seeing the Google tech talk on Mondrian and thinking that was pretty cool – then later hearing about ReviewBoard on the Django users mailing list.

I’ve since starting using ReviewBoard (and even contributed a few bits back) and have been very pleased with it. I’m curious to dig into what Code Review offers in comparison, but I already know that I’ll continue using ReviewBoard. For one thing, my office wouldn’t take kindly to code and code reviews happening beyond the constrains of the fire wall. But I am seriously jazzed about Code Review in the hopes that it’s very easy to set up. If it’s an easy, low/no-cost solution for open source code… well, that’d be pretty damn cool. ReviewBoard, for all it’s goodness, can take some time and effort to completely install.

UPDATE: Since I wrote about this, I found the Google project that Guido has used to make (and now share) CodeReview – the project is rietveld. Guido mentions that it’s meant to showcase how to do a webapp using Django templating and views with the Google Application Engine.