Pinax does a completely kick ass job of making a combined and base project that pulls together a lot of pretty darn good re-usable django “applications” into a single project. What it doesn’t do well is manage any of the dependencies in that tree, or make it easy to use only a portion of some app. If you want to use Pinax the way Pinax is already developed, you’re rockin’. If not, you’re re-working quite a bit of code. And the worst part is that code is almost all opaque to you and hidden in the pinax framework.

The application I wanted to pull in to my basic “pinax” project was “photos” – some simple photo uploading. Yeah, cool – uses photologue under the covers. What’s not clear is that photos also has dependencies on grouping and tagging, and if you review the dependencies documentation, it looks like there’s circular dependencies embedded in there.

What I ended up doing was to take the “photos” application from the Pinax project directly, fork that, and drop it into my project. Then I had to do a shit-load of renaming and stripping because “photos” is built in to Pinax as a default – so “photos” in my app became “photo” and I was scrambling through code to remove the bindings into commenting, threaded commenting, grouping, and so forth.

I did get it working, and really all told I spent less time stripping out stuff than I would have putting it together from scratch. But the only reason I was able to do it that efficiently is because I’ve already been through the pinax project’s source, had it on my laptop, and was a fairly comfortable and competent Django programmer. Any newcomer hitting this would have been toast, and badly confused.

When I asked around the Seattle Django user’s group related to Pinax, several of them said they’d been shying away from something like Pinax because it was easier for them to simply including the good open-source django apps into their work already. At the time, I thought “Well, maybe that just makes Pinax easier and more effective for newbies into Django.” Now I’m reconsidering.

I don’t have a good answer to the problem, and I’m sure I’ll continue to use Pinax and maybe even dig deeper and see what can be done to resolve this kind of issue. I think the idea of the Pinax project is fantastic, I just wish it was a little less “tightly coupled” under the covers and provided more transparency into it’s depths.

That’s the whole reason I posted gist 284927 yesterday on setting up a remote app server for a pinax project with Fabric.

The biggest hurdle that I have with the fellows I’m bringing up to speed is them understanding what’s IN the project that I’ve already slapped down. Even a basic pinax project has a huge amount of complexity for the newer django developer – it’s pretty overwhelming. To try and mitigate that complexity, I’ve started a side project making Pinax cheat sheets. (http://github.com/heckj/pinax_cheat_sheets).

This is not intended to replace Pinax documentation, and in fact I hope that I can eventually tweak these around and put them into the pinax project. I have a huge respect for it (and I use it), so here’s a little bit of detail for contributing back.

I am very happy to take on any changes, edits, updates, etc – from others. There’s a lot of folks who’ve worked very hard to build all these components, and I’m not writing from the perspective of the creator, but of the “code spelunker”. I’m sure I’ll get some of this stuff wrong – so if you see something, please let me know.

I’d ideally love to receive contributions in the form of pull requests from GitHub. Just fork that project, make your edits, and when you’ve pushed them back up – send me a pull request. I don’t promise that I’ll include everyone’s contributions, but to be honest I don’t expect to be too darn picky.

I’ve written the cheat sheets in markdown, mostly because I know it, and secondarily because it renders nicely as HTML when viewed through GitHub.

The start of the project has the following cheat sheets up:

• django.contrib.auth
• django.contrib.contenttypes
• django.contrib.humanize
• django.contrib.sessions
• django.contrib.sites
• pinax.templatetags
• notification

I have a bunch more external apps to go before I hit the internal Pinax apps. I’m working through everything that’s included in a Pinax “basic_project”.

• django_openid
• emailconfirmation
• mailer
• announcements
• pagination
• timezones
• ajax_validation
• uni_form
• staticfiles

And then the internal apps:

• basic_profiles
• account
• signup_codes
• about
• django.contrib.admin

http://gist.github.com/284927

Please feel free to replicate/clone/use/etc whatever to your heart’s content. It’s built around deploying a Pinax 0.71 project clone that also uses South for schema migrations, and the project is housed in BitBucket.

The other fabric file I found that’s referenced frequently was a part of the inspiration here: http://gist.github.com/212366. It’s a fabfile that’s intended to provide some simple rollback and expects to be used from GitHub – and has some really nice documentation associated with it. Maybe I’ll work on mine to add some of that doc… but mostly I’m just making it available as is.

Trac, of course, is one I’ve used and really enjoy. Trac, however, doesn’t really support multiple projects in any integrated way – and we’ve got way more than one project. From the activity I’ve seen, it’s not about to any time soon. Variations on that theme include DrProject, and more recently Basie. They’re all interesting, but I didn’t see either DrProject or Basie as really ready for prime time and I wanted to be able to quickly grow beyond the basics. Two others that are obviously available are Redmine and Retrospectiva.

Ultimately, I’m heading down the road with Redmine for the following reasons:

1. It’s a decent and clean user interface that has several years of polish now around it
2. The system support plugins, and there’s several plugins that I really want to take advantage of
3. Redmine and Trac appear to have the most active communities around them and driving them

The story cards and burn-down charts from Retrospectiva’s AgilePM addition were darned compelling. But ultimately it has a more limited set of folks focusing on that project, and I think I can achieve the same effects with plugins that are available from the Redmine community. Many of these are hosted on GitHub, and even more are in development or some middlin’ shape at the same site. And finally, I’d much rather deal with a partially implemented plugin than a partially implemented core system.

If I could host our development outside of the corporate firewall, I’d likely be looking at Bitbucket or Firefly for the work. I like Bitbucket partially because it’s all mercurial based, and I need to support some Microsoft based development. Git will work with enough effort, but I don’t want to go down the road of trying to explain all the detail needed there. I love the functionality in GitHub, and frankly think it’s a slightly better and more aggressively updated solution that BitBucket, but the git thing – while I’m happy enough with it – just sinks me otherwise because I know I’ll end up spending hours attempting to support it on a windows environment, and I just don’t want that hit. Firefly is ActiveState’s “let’s take and commercialize Trac by hosting it” – and they look like they’ve done a nice job of it. The agile methodology plugins and burn-down charts are pretty darn compelling.

It should be noted that I also took a pretty deep look at Launchpad. Even though it’s open sourced, it’s a damned complex thing to potentially integrate and build up for internal use, and its integration/collaboration features just haven’t been as compelling to me as the machinery that GitHub or Bitbucket provide. The “branch and request a pull” feature set that supports the massive branching and merging for features, bugfixes, etc. is just so much easier in Github or Bitbucket – and the wiki support to the other components is also far superior.

If you’re looking for something yourself, definitely keep an eye on the market and projects though. RIght now it’s just December 2009, and it’s clear there is a lot of effort, momentum, and a ton of good ideas that people are just begging to implement in this space. If I go to make this same decision in 6 months, there’s no guarantee that the landscape won’t have changed dramatically and a different solution would be better.

Since I’ve got it deployed now onto a little dev server, I was thinking about how I was going to make this work going forward. Database migrations being one of those “how do you upgrade things?” questions when you’re doing rapid development django applications where the database model can get twiddled a bit.

The result of scanning around this afternoon from my favorite coffee shop haunt led me to South. In about 10 minutes I’d enabled and moved to south and had my application up and rolling with it. My previous experience was with Django Evolution – which wasn’t bad, but it was more work to get it into place and running.

The migrations in place, I did some model twiddling and experimented a bit to see how it all worked. I can see how if you’re not being somewhat careful, you could get into some really horrible places that no program could figure out. But basic migrations – moving forward and backward – really do an excellent job of “just working” with this framework out of the box.

The one interesting quirk I ran into – I’ve been doing tests on my codebase with:

coverage manage.py test

which just caught and ran the tests for everything in the application list. Figured it didn’t hurt to test the admin and auth stuff as well. When I added south, which I installed using:

 pip install south

The directory that south’s tests expect to use suddenly becomes a bit less accessible on the build server. Not a bad thing to redirect the tests to just what I’m writing, but an interesting side effect.

I start with a basic virtual machine – in this case, I’m using Ubuntu Server 9.10. Once you have a basic machine installed, I recommend you make sure everything’s up to date with patches.

sudo apt-get update
sudo apt-get dist-upgrade


If you haven’t already, consider installing an ssh server for remote login. You don’t need it for hudson, but I find it makes administration easier.

sudo apt-get install openssh-server

To make installing python packages and pieces a bit easier:

sudo apt-get install build-essential python-dev

Start off by getting a copy of hudson.

wget http://hudson-ci.org/latest/hudson.war

Hudson runs fine from that war directly – a darned handy way to make the whole system work. You can also install this in Tomcat, JBoss, or whatever other servlet container you like to run. I find it easiest to run directly from the war file.

If you’re going using Mercurial or Git, you’ll need to install the appropriate version control client onto your machine as well. The plugins use those systems through the command line interface.

sudo apt-get install mercurial
sudo apt-get install git-core
sudo apt-get install subversion


Since hudson is a java application, we need to have java installed… Ubuntu 9.10 comes with OpenJDK which works just fine. You can also download and install the Sun JDK if you want to, but it doesn’t really seem to be required. If you don’t have any JDK already installed, you can generally get OpenJDK using:

sudo apt-get install openjdk-6-jre-headless

To get rolling with Hudson, just fire it up:

java -jar hudson.war

I just run this as the base user – no special privileges. It creates all it’s working directories, workspaces, etc in a “.hudson” directory. You can do the extra work to make this a unix daemon too – I haven’t bothered because it’s been relatively stable. There are some good notes on doing this at http://weblogs.java.net/blog/2009/02/10/hudson-now-good-behaving-unix-daemon and http://wiki.hudson-ci.org/display/HUDSON/Installing+Hudson+as+a+Unix+daemon, so I won’t dive into those details here.

Once you’ve started it, bring up the UI in a browser to make sure it’s all working. My virtual machine is running at 192.168.0.2, so I open up http://192.168.0.2:8080/. I’ll use this same IP address through the course of this walk-through. Of course you can replace it with a hostname.

I generally start off by making sure we have the latest Hudson installed and getting the various client plugins I want to use all installed – click on “Manage Hudson” (http://192.168.0.2:8080/manage)

If there’s a more recent version of Hudson available, it will show at the very top of this page and give you the option to download and install it. You’ll be able to upgrade plugins the same way – Hudson knows to look at it’s own site for the latest plugins and versions, so it’s very easy to maintain. If you’ve been following along with this, the “wget” statement up above will have likely brought down the latest version already, so you won’t see this sort of screen until a new update comes along.

The upgrade automatically mechanism has worked great for me, but a couple of caveats are needed here – the process needs to be running with permissions to the directory in which it’s located to work. The other is once you’ve upgraded the system, you may need to restart the service from the command line to enable future downloads. I’m not sure if I ran into a bug with that, or if that’s expected behavior – but I haven’t run into it often.

Now you should have the latest version – either directly or just upgraded. Lets get to the plugins – because that’s where a lot of the Hudson goodness really lives. You can get to the list of installed plugins so you can see what’s there by default. From the “Manage Hudson” page, click on the link to “Manage Plugins”. The URL for that page is http://192.168.0.2:8080/pluginManager/

There are four tabs here to help you see what’s available – the first tab Updates is the default, and shows any outdated plugins. The second is Available, that shows what’s linked up from the central Hudson plugin list, and Installed, which shows the bits that are already there. Click on the Available tab (url is http://192.168.0.2:8080/pluginManager/available) and you can see the plethora of interesting things ready to use. The ones I like to install for my python projects are:

• Cobertura Plugin
• Git Plugin
• Mercurial Plugin
• Monitoring Plugin
• Python Plugin
• Violations Plugin

All of these plugins will still require you to have whatever local libraries you need to make them work. That’s why we installed Mercurial, Subversion, Git, etc earlier – to make sure those tools were available. Hudson invokes them on the command line on your behalf to make it all work.

Below is a broad set of libraries that cover a lot of pieces – you may want to install more/different libraries to accomplish your aims with Hudson. I’ll be aiming to create some typical build reports with it all – unit test output, coverage, etc.

sudo apt-get install python-setuptools
sudo easy_install coverage
sudo easy_install pylint
sudo easy_install unittest-xml-reporting
sudo easy_install fabric
sudo easy_install nose


Setting up a new Job

Now we just need a project. Let’s start with an open-source project – Pygments is a great project, with lots of diverse things happening within it. And it’s got tests and will be a good general victim for this writeup.

Start out by clicking on “New Job” (http://192.168.0.2:8080/view/All/newJob). A Job in Hudson is a build process. I’m going to make a fairly simple one, but you can cascade multiple jobs together into a far more complex build process if you want to… I’m not going to delve into that here, focusing instead on getting a single python project rolling.

In the new job form:

• set the job name to “pygments”
• choose “Build a free-style software project”
• I also put in a description (it’s not mandatory)
• then choose “Mercurial” from the Source Code management section

• put in “http://dev.pocoo.org/hg/pygments-main” for Repository URL
• put in “trunk” for the Branch
• click the “Save” button down at the bottom

Now you’ve got a job – we can come back and configure this at any time, and we’ll do more tweaking as we go through this. Let’s make sure we have the basics all working first though…

At this point, you should be at the project pygments page in Hudson. If you’re not, you can navigate there from the front page, or go directly to the URL (http://192.168.0.2:8080/view/All/job/pygments/). The page should look something like this:

On the left hand side of the page are the controls for the job, including Configure and Build Now among a list of other options.

• click on “Build Now”

You should see the build start processing with a little rolling progress bar on the left hand side of the window. You can click on that link that looks like #1 Nov 5, 2009 8:35:18 AM’ in the Build History box.

Either immediately, or when the build reports complete, click on that link.

You can see the console output from the build (which is just pulling down the source) with the link Console Output. The output of the build is exactly what happened on the command line. In this case, you’ll see something akin to:

Started by user anonymous
$ hg clone -r trunk http://dev.pocoo.org/hg/pygments-main /home/ubuntu/.hudson/jobs/pygments/workspace
requesting all changes
adding changesets
adding manifests
adding file changes
added 903 changesets with 2247 changes to 269 files
updating working directory
249 files updated, 0 files merged, 0 files removed, 0 files unresolved
[workspace] $ hg log -r . --template {node}
Finished: SUCCESS

Once you’ve done an initial build, the files will also be available for you to look at and download through the Hudson UI. If you click on the link Back to Project or just the project name at the top of the page, you should see a new reference in the project called Workspace. Clicking into that will show you all the files that were downloaded and any other results of the build process. Right now we’re just getting files from source control, but as we add build steps to the process, the results will be available to dig around in at the workspace.

If you want to set up Hudson to check for updates and build when it finds them, you can go back to the project page and click on the Configure link. Once in there, scroll down in the configuration and enable the checkbox labelled “Poll SCM”. As soon as you do, another text field will come available that will let you specify a Cron-like string to identify when the builds should check source control. In the screenshot below, I’ve enabled Hudson to check the mercurial repository every 10 minutes.

…

Any time you change the configuration, go to the bottom of the Configure page and click on “Save” to set the changes.

Adding Build Steps – doing ‘work’

Right now, the build isn’t doing too much – let’s see about getting it doing some more work for us. The Pygments source code is set up with a Makefile to make doing this a bit easier, so we’ll take advantage of that. If you’re not in the Job configuration page – get there. Go to the job an click on the Configure link. Now we’ll add a build step that simple invokes the shell command “make test”.

Scroll down in the configuration page until you see the section labelled Build. Click on the button underneath there labelled Add build step, and choose Execute shell. When the text field for that script appears, put in the text make test.

Scroll to the bottom of the page, save the configuration, and invoke “Build Now” to run the updated build. Looking at the console output for the build will now show the updated console log with the running tests, like the screen shot below.

Should the built in unit tests fail, the system will show a red indicator on the hudson front page and next to the build results link.

Displaying unit test results

What this isn’t doing it making some pretty graphs and charts. Let’s see about getting some of that enabled.

Pygments uses Nose tests for it’s unit tests. That’s pretty nice, because Nose includes a mechanism to output the format of the tests into Java-based JUnit format, and Hudson in turn knows how to make pretty pictures from that format. Since we installed the nose library earlier, we can take advantage of it. Probably the “right” way to do this would be to change the Makefile to invoke the tests that output XML for the unit tests. For right now, we’ll just shim it into place with new build steps.

• Change the content of the Execute shell build step and replace it with
  python tests/run.py --with-xunit
• save the configuration and invoke “Build Now”
• now look in the workspace in you should find a new file called “nosetests.xml” in the tests subdirectory.

Let’s get that graphed…

• Go to the job configuration and enable “Publish JUnit test result report”. It will have a text field so that you can tell it where to find the results. It uses an Ant format for finding filenames in an existing directory structure.
• In this case use **/tests/nosetests.xml
• Save the config
• invoke Build Now

Now you’ll see a new element in the web user interface called “Latest Test Result” which you can dig into. If you invoke “Build Now” again, it will start graphing the results of the tests and making that trend available in the project view. Right now this graph is going to be really darned boring, because there aren’t any changes between builds. Once more code starts rolling in, you’ll see changes with the number of tests being invoked. You can also click on that link and see the tests that were invoked. For python folks, remember that we’re shimming Python unit tests into a JUnit conceptual framework, so there are going to be some “leaky abstractions here”. In particular, test names, classes, and such may not alway match up with expectations. I have more details on how to enable XML output for a Django based test runner that I’ll post in another write up… uh, later.

Add cobertura style test coverage reporting:

Now we get to use Ned Batchelder’s coverage library to see some cool stuff!

(This whole write up is basically a thank you to Ned for writing Coverage)

Go back to the configuration for the job

• change the Execute Shell build step to read:

coverage run tests/run.py --with-xunit
coverage xml


• scroll down and enable  the check box Publish Cobertura Coverage Report
• use the pattern **/coverage.xml

Add pylint reporting

Head back to the job configuration

• add a new Execute shell build step  to read:

#!/bin/bash
pylint --rcfile scripts/pylintrc -f parseable pygments > pylint.txt
echo "pylint complete"


• It’s important to start this with #!/bin/bash, and then end with the echo statement so that the return codes from pylint don’t get interpreted as a build failure (unless you want that… I don’t know about you, but I’m not pythonic enough to have a 100% pylint result.)
• enable the checkbox Report Violations
• under pylint, use XML filename pattern of **/pylint.txt

• save the configuration and invoke “Build Now”

So there you have it – a python project building under hudson with unit tests getting run, cobertura style coverage reports, and pylint style reporting.

While I had some good original ideas, I didn’t keep the project moving. And the project has been effectively abandoned from my head for the past 9 months. So today I’ve moved ownership of that project to Rajesh. I’ve left myself as a contributor – but I’ve found that unless I’m actively using the code myself, I’m not likely to do anything to contribute into the code base – and right now the Django based work I’m doing is all pretty external to needing any queues to get stuff done.

I think I likely should have handed the project off some time ago – I didn’t do it any favors by just lurking on the code and not promoting it. I didn’t hold up any patches either – I just didn’t make it move forward. I think the project’s in good hands and I’m looking forward to watching the “really simple queue” space to see where it continues to go. There are some interesting alternatives out there too – Gearman, Celery (rides atop RabbitMQ), and beanstalkd.

I’m not as active in the community these days, but I’m still using the project on a regular basis as one of my “secret weapons” to get things done quickly. It’s worth noting that there are some definite changes in the 1.0 release from even what’s in the various books now available for Django. If you’re heading in to the 1.0 release and can’t find something you expect to be there, make sure you check the online documentation at http://docs.djangoproject.com/ – it’s very complete and very worthwhile.

(I haven’t tried Graphite yet, but the screenshots looked pretty nice – so I might be doing so to see what it’s all about.)

LaunchPad is Ubuntu/Canonical’s code hosting platform – similar to Google Code or Sourceforge in concept. They use Bzr for their source control, which is a tad odd to me – I’m more in the mercurial camp of any decentralized SCM – but their web site is really first class. They’ve got a nice system and setup – simple, somewhat bold iconography, obvious components to projects, and a clean look. As a place to encourage project and community participation, I admit that I was immediately drawn in. My own project (languishing at the moment) on Google Code (django queue service) has an interface that feels positively stark in contrast.

I’m not going to move anything over, but I suspect I will pay more attention to the LaunchPad setup going into the future.

I’ve since starting using ReviewBoard (and even contributed a few bits back) and have been very pleased with it. I’m curious to dig into what Code Review offers in comparison, but I already know that I’ll continue using ReviewBoard. For one thing, my office wouldn’t take kindly to code and code reviews happening beyond the constrains of the fire wall. But I am seriously jazzed about Code Review in the hopes that it’s very easy to set up. If it’s an easy, low/no-cost solution for open source code… well, that’d be pretty damn cool. ReviewBoard, for all it’s goodness, can take some time and effort to completely install.

UPDATE: Since I wrote about this, I found the Google project that Guido has used to make (and now share) CodeReview – the project is rietveld. Guido mentions that it’s meant to showcase how to do a webapp using Django templating and views with the Google Application Engine.

Tonight, Google released AppEngine – an elastic computing environment specifically set up with the idea of developing web applications in python. They’ve pushed up a copy of the transcript from their announcement if you’re interested…

It may come as no surprise that the back-end of AppEngine is/was heavily influenced by Django – and in fact the SDK that you can download comes with Django 0.96.1 embedded within it. That’s not to say Django is your only choice, but it definitely it mine.

To that end, there’s an excellent article up at their site on using Django with AppEngine – which basically boils down to: “Don’t use Django’s models, because we’ve got our own back-end datastore thingy, and it works similarly, but from different, than Django’s models and a database. I haven’t looked at the HQL datastore API they’ve got enabled, but I’m going to guess that some Django core developers likely will before too long to see what the potential overlap is for making it more directly usable with Django’s model system. (That guess is PURE speculation, I don’t have any insider knowledge there)

I wasn’t speedy enough on the request to get into the initial pool of folks who can use the beta – but I’ve tacked into the waitling list. It’s not like I really need another cool technology to go spend a lot of time on, but I’m really curious… especially with the idea that hosting some amount of your application might be available for free. Not quite sure how that is going to make sense long term – but I’ll take it while it’s available.

One thing for sure – this is certainly going to bring a boon of new attention to python web development, and likely Django as well.

Review Board includes an automake setup solution, but I didn’t take advantage of it. Partly because I felt comfortable setting up a Django project without it, and partly because I didn’t clue in that it was there and I should use it until well after I’d begun fiddling with all this.

From those basics, here’s some additional tidbits

1. get an instance of reviewboard running:

mysql -u root -p
 (enter password)
create database reviewboard;
GRANT ALL on reviewboard.* TO reviewboard@localhost IDENTIFIED BY "sup3rsekret"

2. set up settings_local.py to match:

# Database backend.  Any supported django database engine should work.
DATABASE_ENGINE = 'mysql'      # 'postgresql', 'mysql', 'sqlite3' or 'ado_mssql'.
DATABASE_NAME = 'reviewboard'  # Or path to database file if using sqlite3.
DATABASE_USER = 'reviewboard'     # Not used with sqlite3.
DATABASE_PASSWORD = 'sup3rsekret' # Not used with sqlite3.
DATABASE_HOST = ''             # Set to empty string for localhost.
DATABASE_PORT = ''             # Set to empty string for default.

3. initialize the database:

cd reviewboard
./manage.py syncdb
 (create an admin account when asked)

4. hook this whole thing up to Apache:

I configured my setup using the provided mod_python template – the Makefile setup system seems to include provisioning it, but I did this all by hand.

Running as mod_python means that the account accessing perforce was “www-data” by default – a user that isn’t normally enabled in our Perforce repository. I worked around that issue by handing down additional perforce specific environment variables:

    SetEnv P4USER readonly-account
    SetEnv P4PORT p4.server.com:1666
    SetEnv P4CLIENT reviewboard
    SetEnv P4PASSWD xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

5. set up the initial data bits to use the service in our environment:
• use “python post-review -d” to find out what perforce is calling your repository.
• Add that repository to the list of repositories in the Admin console of ReviewBoard
• Add at least one “review group” in the Admin form as well
6. Set up a .reviewboardrc for your tree

Place this at the root of your tree. Post-review starts in the current directory and scans upwards to find an instance of this file to know what to do…

# Default Review Board Server
REVIEWBOARD_URL = 'http://10.0.0.25'
 
# Per-repository Review Board servers
# TREES = {
#    '/path/to/local/repo/checkout': {
#         'REVIEWBOARD_URL': 'http://whatever' 
#    }
# }

One other thing of note: post review is sensitive to the Diff Headers in trying to pull out the date. We were using an old version of diff and it had some real trouble with that. We updated to using diffutils 2.8.7 to resolve that.

I pulled down all the parts and pieces. Getting p4python all set up was a bit of a trick, but the wiki had good hints in the right direction to get me rolling. Just spending a few hours with it, I found a few little quirks, and I’ll be adding more back into the project as I find things – might as well, if I’m debugging and fiddling to get things to work anyway, it seems only right to share it back with the community.

So the first thing to note is that getting a ReviewBoard instance set up isn’t really documented on the wiki as yet. GettingStarted, and UserBasics gives some hints, but you’ll still need to read a little code to get it all happening.

Here’s what I’ve got for the first takeaways:

1. To really use this tool (at least with Perforce), you’ll want to use a client like the provided script post-review. The script is pretty straightforward really – it does all the tedious work of creating a diff prior to checkin and shoving it up into the system.
2. post-review supports a config file mechanism, but I don’t yet have that format really working for me. I suspect I’m doing something stupid, but it’ll take a little back-tracing to get a reasonable config set up yet.
3. Getting an instance of this thing rolling requires a few django-specific commands to get the environment seeded up. At a minimum, you’ll need to invoke “./manage.py syncdb” to get the database schemas all in place and rolling.
4. When you get the instance all rolling for the first time, you’ll need to set up an admin account and add a Review Group with an appropriate path. I suspect the subversion folks have this a touch easier than Perforce. With Perforce, the “path” that you enter needs to match the details that you’d otherwise get from running “p4 info“. When I first set things up, I didn’t realize that. The server name we use internally is different than the data presented in “p4 info“, so that took me a bit by surprise.
5. post-review also appears to be sensitive to the format of the output of running a diff. If your diff tool, for example, outputs it’s header data in the form of “Fri, Feb 8 14:23:52 2008″ instead of “08/18/2008 14:23:52″, you’re going to get a very confusing error.

There’s more, I’m sure – but I had to deal with other tasks this afternoon, so I haven’t yet got the basics up and functional. I’ve at least submitted a bug that I was sure wasn’t just a quirk of my somewhat odd development environment. Something that looked like a re-factoring edge case that just hadn’t been caught yet. I included a patch to fix it – although to be honest, I’m wasn’t sure if I should be submitting that to a ReviewBoard instance someone or not, just I just shoved it into the bug.

I think one of the most impressive things about ReviewBoard is that it supports a nice mechanism, and some example scripts, for doing a pre-checkin review. I had naively assumed that I could only get the discussion/review aspects in a post-commit mechanism – without ever even really looking at the project. While I haven’t read the post-review code in depth, it’s pretty clearly set up with the ReviewBoard server as something where a client could significantly help out with this work.

So it’s not smooth and super easy. Yet. To be fair, the devs don’t say it either, and they’ve got a wiki, issue tracker, and all the code available… so really its just a matter of time before I get it working to suit me. I’m really glad they’ve made this available at all, and I’m looking forward to seeing if I can really make it sing.

Update: The answer is yes – you should submit updates and patches to the reviewboard at http://reviews.review-board.org/.

apt-cache search django

… and sure enough, it came back with a hit.

I had a development environment set up in minutes with the following:

apt-get update
apt-get dist-upgrade
# dev environment
apt-get install python-django python-pysqlite2 
# running with MySQL
apt-get install python-mysqldb mysql-server-5.0 mysql-server python-egenix-mxdatetime

Nice!

The results are all from my laptop – 2GHz Intel Core Duo, 2GB RAM. Running python 2.5 and the current trunk of Django (0.97-pre-SVN-6668).

The test ran through the timing for adding messages into the queue and getting/deleting messages from the queue, scaling the queue size from 0 to 25,000 items. The client that ran these tests was put together with httplib2, release 0.40 – running in python on the same machine. Yeah – so pretty much factor out network latency on this one…

The adding stayed a fairly consistent 8 ms, deviating around a bit – but mostly there.

The combination of getting a message and deleting it (in our API, the functions are separate – so it means two calls back to the service) rose linearly with the size of the database. At a base level, it was running about 20 ms and by the time the database size was at 25,000 items the “get and delete” operations were taking roughly 165 ms.

For SQLite-on-disk database backend, I’m pretty pleased with those results.

Rajesh apparently picked up a copy of the book RESTful Web Services (O’Reilly), and made a number of suggestions about future direction after reading the book. It really sounded like he got a lot out of the book, so I ordered a copy (through Queen Anne Ave Books cause they’re cool and local). I should have picked it up Thursday evening, but I was being a slacker and only went by and got it this morning. Anyway, I was starting to read it walking home from the bookstore and I immediately spotted a small chapter/segment in there on doing REST services with Django! Apparently Jacob helped out with a chapter in there, so there’s some interesting notes tucked away in Chapter 12 that I’m interested in getting to.

My knowledge is so scattershot (the trouble with being a technology generalist), that I think I’m going to take this book through from the start to make sure I catch up on those niggling little details that seem to slip the the cracks. I’m currently taking a break before seriously digging in to Chapter 2 (writing REST clients), and I’m glad to see a nice level of detail upcoming on how to actually get the various (what I’ll call obscure) HTTP verbs functioning from the clients. They have details on several languages, including libraries to use and example code – Ruby, Python, Java, JavaScript, C#, and so forth. It looks to be a good chapter.

For a long time, my reluctance to really get down to the “PUT” and “DELETE” verbs was because the libraries just didn’t easily support them – and I wasn’t sure what the “right way” of making them work should be. I’ve been pretty horrifically burned trying to learn the details needed to make some of the WS* bullshit specs work, and that translated pretty directly to a reluctance to learn the details of programmatically creating HTTP requests with the appropriate “verbing”. A blank spot, for sure – and one that I’m fixing now.

This project (the Django Queue Service) is mostly a “because I can” project. I’m not heavily using it in anything currently, although Rajesh and Marcus are – and I’ve heard a few other folks have apparently picked it up and run with implementations from it. I’m firmly of the opinion that the desktop and web-space cloud will continue to mix things up. A concept that Brent outlined in his Hybrid Apps Presentation at C4 over the summer of 2006. (Better have a Mac if you want to see that presentation by the way – it’s a VoodooPad document). I’ll be the very last to claim that desktop applications are dead – with the advent of libraries like Sparkle the distribution benefit of the web is just as easy made available to desktop apps as to web applications. And from my own experiences with browsers recently, coming up with fantastic UI functionality that works across browsers is an intense pain in the ass. Its getting better all the time but there are just some things you can’t do there in terms of interactivity and feedback that I’d really like to see and want to take advantage of.

All props to Matt for the initial deployment and use during the current sprint. There’s still some bugs and kinks to work out of the system – and we’re going to be hammering those out as soon as the sprint it complete. Since Matt got it functional for the sprint, we didn’t want to mess with the configurations and start throwing false alarms while there was such dedicated forward progress going on.

Right now, we’ve got some base builds and tests running against Python 2.4 and 2.5 with Postgres and SQLite to pretty good effect. Still remaining are stable runs against MySQL, anything with Python 2.3 support, and more effective implementations to make it easier for us to support remote build slaves (Oracle? Windows? anyone?) and enable branches to get built a little more easily.

There is plenty to go, and we’re just getting kicking. I’m sure you’ll hear more as we get things nailed down and continue moving forward.

Of course, this is assuming you’ve got a decent handle on how to check out a specific revision from Subversion. I’m not sure that always applies to folks looking for a from-the-start tutorial, but the idea is really cool.

At OSCON 2007, I was hunting around for something to use to deal with background processing initiated from my web application. Not finding anything that I could immediately use and implement, I took it as a challenge to knock out something in the time I was there. The result is the django queue service.

It was later in the week (at the Django BOF) that I learned about TheSchwartz and Gearman, which Brad Fitzpatrick is hacking away it to make a queue mechanism available for fellow Python and Ruby geeks.

At the time, I thought “I wonder if I can hack this out in a week…” Well, yes I could. I modeled the service after the REST API from the Amazon Simple Queue Service, because that seemed both reasonable and effective. And I didn’t mind the idea of having my own Queue that (mostly) matched that API that I didn’t have to pay anything to use.

The project also contains from code from the CherryPy project. Mark Ramm (a really nice fellow, by the way) gave a session at OSCON about WSGI servers that was really intriguing. I’ve been looking for a “pure python” service to run Django with for a while, and after chatting a bit with him, thought I could probably make it happen with the CherryPy WSGI server. So I did. I have a little piece of linkage code that fires up the CherryPy WSGI server code and links it right on through into Django. I didn’t want to use the rest of CherryPy, so I nicked that code and dropped it into this project as a nice reference and a handy way to run the queue.

I suppose this project could easily end here, and of course there was the “but I want to…” list of things that I had in mind to do before I released it into the world. Well – as they saying goes: “Now is better than never.” so out it goes.

If you are interested in getting involved with the project, sending me patches, or whatever – let me know. You can leave a comment here or use that spiffy Google Code Hosting issue tracker. The project is functional, but not brilliant – an not entirely cleaned up in terms of its API. But enough excuses – its there, feel free to use it or not as you like, contribute or not as you like.

I also got a chance to meet Paul Bissex, who wrote briefly about the first day of OSCO. He’s in the Python WSGI session with me now.

Speaking of which, I’ve now learned about FAPWS from William (downloads) which was one of the WSGI servers mentioned, and apparently tested with Paul’s simple wiki code. It uses PyEvent under the covers, and apparently achieves some very nice performance characteristics.