Rhonabwy

I signed up with Launchpad today - don’t know why I hadn’t really done it earlier, other than I just didn’t think to or have any projects that drove me there. The project du jour that lured me in was Graphite - a distributed high-performance monitoring solution written in python (and apparently using Django as well!).

(I haven’t tried Graphite yet, but the screenshots looked pretty nice - so I might be doing so to see what it’s all about.)

LaunchPad is Ubuntu/Canonical’s code hosting platform - similar to Google Code or Sourceforge in concept. They use Bzr for their source control, which is a tad odd to me - I’m more in the mercurial camp of any decentralized SCM - but their web site is really first class. They’ve got a nice system and setup - simple, somewhat bold iconography, obvious components to projects, and a clean look. As a place to encourage project and community participation, I admit that I was immediately drawn in. My own project (languishing at the moment) on Google Code (django queue service) has an interface that feels positively stark in contrast.

I’m not going to move anything over, but I suspect I will pay more attention to the LaunchPad setup going into the future.

Word is that the a version of the code review tool that Google uses internally (called Mondrian) on the Google App Engine - called Code Review. I remember first seeing the Google tech talk on Mondrian and thinking that was pretty cool - then later hearing about ReviewBoard on the Django users mailing list.

I’ve since starting using ReviewBoard (and even contributed a few bits back) and have been very pleased with it. I’m curious to dig into what Code Review offers in comparison, but I already know that I’ll continue using ReviewBoard. For one thing, my office wouldn’t take kindly to code and code reviews happening beyond the constrains of the fire wall. But I am seriously jazzed about Code Review in the hopes that it’s very easy to set up. If it’s an easy, low/no-cost solution for open source code… well, that’d be pretty damn cool. ReviewBoard, for all it’s goodness, can take some time and effort to completely install.

UPDATE: Since I wrote about this, I found the Google project that Guido has used to make (and now share) CodeReview - the project is rietveld. Guido mentions that it’s meant to showcase how to do a webapp using Django templating and views with the Google Application Engine.

I’m sure that anyone into Django will either see or hear about this before they catch my blog post, but it’s too neat a story not to write about a comment on.

Tonight, Google released AppEngine - an elastic computing environment specifically set up with the idea of developing web applications in python. They’ve pushed up a copy of the transcript from their announcement if you’re interested…

It may come as no surprise that the back-end of AppEngine is/was heavily influenced by Django - and in fact the SDK that you can download comes with Django 0.96.1 embedded within it. That’s not to say Django is your only choice, but it definitely it mine.

To that end, there’s an excellent article up at their site on using Django with AppEngine - which basically boils down to: “Don’t use Django’s models, because we’ve got our own back-end datastore thingy, and it works similarly, but from different, than Django’s models and a database. I haven’t looked at the HQL datastore API they’ve got enabled, but I’m going to guess that some Django core developers likely will before too long to see what the potential overlap is for making it more directly usable with Django’s model system. (That guess is PURE speculation, I don’t have any insider knowledge there)

I wasn’t speedy enough on the request to get into the initial pool of folks who can use the beta - but I’ve tacked into the waitling list. It’s not like I really need another cool technology to go spend a lot of time on, but I’m really curious… especially with the idea that hosting some amount of your application might be available for free. Not quite sure how that is going to make sense long term - but I’ll take it while it’s available.

One thing for sure - this is certainly going to bring a boon of new attention to python web development, and likely Django as well.

I made some notes on getting Review Board up and running. I thought they might be useful for someone else wishing to do the same… I implemented review board on a virtual machine, with the VM running Ubuntu Gutsy. I highly recommend getting the basic setup from the GettingStarted page on the reviewboard project wiki.

Review Board includes an automake setup solution, but I didn’t take advantage of it. Partly because I felt comfortable setting up a Django project without it, and partly because I didn’t clue in that it was there and I should use it until well after I’d begun fiddling with all this.

From those basics, here’s some additional tidbits

1. get an instance of reviewboard running:

mysql -u root -p
 (enter password)
create database reviewboard;
GRANT ALL on reviewboard.* TO reviewboard@localhost IDENTIFIED BY "sup3rsekret"

2. set up settings_local.py to match:

# Database backend.  Any supported django database engine should work.
DATABASE_ENGINE = 'mysql'      # 'postgresql', 'mysql', 'sqlite3' or 'ado_mssql'.
DATABASE_NAME = 'reviewboard'  # Or path to database file if using sqlite3.
DATABASE_USER = 'reviewboard'     # Not used with sqlite3.
DATABASE_PASSWORD = 'sup3rsekret' # Not used with sqlite3.
DATABASE_HOST = ''             # Set to empty string for localhost.
DATABASE_PORT = ''             # Set to empty string for default.

3. initialize the database:

cd reviewboard
./manage.py syncdb
 (create an admin account when asked)

4. hook this whole thing up to Apache:

I configured my setup using the provided mod_python template - the Makefile setup system seems to include provisioning it, but I did this all by hand.

Running as mod_python means that the account accessing perforce was “www-data” by default - a user that isn’t normally enabled in our Perforce repository. I worked around that issue by handing down additional perforce specific environment variables:

    SetEnv P4USER readonly-account
    SetEnv P4PORT p4.server.com:1666
    SetEnv P4CLIENT reviewboard
    SetEnv P4PASSWD xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

5. set up the initial data bits to use the service in our environment:
• use “python post-review -d” to find out what perforce is calling your repository.
• Add that repository to the list of repositories in the Admin console of ReviewBoard
• Add at least one “review group” in the Admin form as well
6. Set up a .reviewboardrc for your tree

Place this at the root of your tree. Post-review starts in the current directory and scans upwards to find an instance of this file to know what to do…

# Default Review Board Server
REVIEWBOARD_URL = 'http://10.0.0.25'
 
# Per-repository Review Board servers
# TREES = {
#    '/path/to/local/repo/checkout': {
#         'REVIEWBOARD_URL': 'http://whatever' 
#    }
# }

One other thing of note: post review is sensitive to the Diff Headers in trying to pull out the date. We were using an old version of diff and it had some real trouble with that. We updated to using diffutils 2.8.7 to resolve that.

I’ve been meaning to dive into the project ReviewBoard since I first heard about it, many months ago. I had first heard of a similar critter - Google’s internal tool called Mondrian. I was pretty darn excited about that tool when I heard about it, but Google’s not making any of it available. Instead some of the fine staff at VMware (fine products that I use, even if they do seem a tad overly ‘proud’ price wise of their enterprise server products…) made it available. More than made it available, actually - as they’re hosting it, developing it, working it, and making it real with Google’s informal help with code hosting.

I pulled down all the parts and pieces. Getting p4python all set up was a bit of a trick, but the wiki had good hints in the right direction to get me rolling. Just spending a few hours with it, I found a few little quirks, and I’ll be adding more back into the project as I find things - might as well, if I’m debugging and fiddling to get things to work anyway, it seems only right to share it back with the community.

So the first thing to note is that getting a ReviewBoard instance set up isn’t really documented on the wiki as yet. GettingStarted, and UserBasics gives some hints, but you’ll still need to read a little code to get it all happening.

Here’s what I’ve got for the first takeaways:

1. To really use this tool (at least with Perforce), you’ll want to use a client like the provided script post-review. The script is pretty straightforward really - it does all the tedious work of creating a diff prior to checkin and shoving it up into the system.
2. post-review supports a config file mechanism, but I don’t yet have that format really working for me. I suspect I’m doing something stupid, but it’ll take a little back-tracing to get a reasonable config set up yet.
3. Getting an instance of this thing rolling requires a few django-specific commands to get the environment seeded up. At a minimum, you’ll need to invoke “./manage.py syncdb” to get the database schemas all in place and rolling.
4. When you get the instance all rolling for the first time, you’ll need to set up an admin account and add a Review Group with an appropriate path. I suspect the subversion folks have this a touch easier than Perforce. With Perforce, the “path” that you enter needs to match the details that you’d otherwise get from running “p4 info“. When I first set things up, I didn’t realize that. The server name we use internally is different than the data presented in “p4 info“, so that took me a bit by surprise.
5. post-review also appears to be sensitive to the format of the output of running a diff. If your diff tool, for example, outputs it’s header data in the form of “Fri, Feb 8 14:23:52 2008″ instead of “08/18/2008 14:23:52″, you’re going to get a very confusing error.

There’s more, I’m sure - but I had to deal with other tasks this afternoon, so I haven’t yet got the basics up and functional. I’ve at least submitted a bug that I was sure wasn’t just a quirk of my somewhat odd development environment. Something that looked like a re-factoring edge case that just hadn’t been caught yet. I included a patch to fix it - although to be honest, I’m wasn’t sure if I should be submitting that to a ReviewBoard instance someone or not, just I just shoved it into the bug.

I think one of the most impressive things about ReviewBoard is that it supports a nice mechanism, and some example scripts, for doing a pre-checkin review. I had naively assumed that I could only get the discussion/review aspects in a post-commit mechanism - without ever even really looking at the project. While I haven’t read the post-review code in depth, it’s pretty clearly set up with the ReviewBoard server as something where a client could significantly help out with this work.

So it’s not smooth and super easy. Yet. To be fair, the devs don’t say it either, and they’ve got a wiki, issue tracker, and all the code available… so really its just a matter of time before I get it working to suit me. I’m really glad they’ve made this available at all, and I’m looking forward to seeing if I can really make it sing.

Update: The answer is yes - you should submit updates and patches to the reviewboard at http://reviews.review-board.org/.

I slapped together a virtual machine to try out an instance of ReviewBoard. It’s been a little while since I fiddled with Ubuntu, so I grabbed the latest server release(7.10) and set it up. Then I settled in to get all the various python pieces I’d need for django… On a lark, I tried

apt-cache search django

… and sure enough, it came back with a hit.

I had a development environment set up in minutes with the following:

apt-get update
apt-get dist-upgrade
# dev environment
apt-get install python-django python-pysqlite2 
# running with MySQL
apt-get install python-mysqldb mysql-server-5.0 mysql-server python-egenix-mxdatetime

Nice!

I took some time this afternoon to benchmark out the work we’ve been doing with the Django Queue Service. We haven’t done anything to optimize it, and we’re starting to push in some code to make it fully support REST verbs (as opposed to its current REST/RPC mixed style). Seemed like a good time to get some numbers down.

The results are all from my laptop - 2GHz Intel Core Duo, 2GB RAM. Running python 2.5 and the current trunk of Django (0.97-pre-SVN-6668).

The test ran through the timing for adding messages into the queue and getting/deleting messages from the queue, scaling the queue size from 0 to 25,000 items. The client that ran these tests was put together with httplib2, release 0.40 - running in python on the same machine. Yeah - so pretty much factor out network latency on this one…

The adding stayed a fairly consistent 8 ms, deviating around a bit - but mostly there.

The combination of getting a message and deleting it (in our API, the functions are separate - so it means two calls back to the service) rose linearly with the size of the database. At a base level, it was running about 20 ms and by the time the database size was at 25,000 items the “get and delete” operations were taking roughly 165 ms.

For SQLite-on-disk database backend, I’m pretty pleased with those results.

Work has been continuing apace on the Django Queue Service, mostly through the efforts of Rajesh and Marcus who have really done an outstanding job of filling in the holes in testing and structure for the application to make it viable over the long term. We had some good conversations over email the other day (enough so that I set up a list for it: django-queue-service@googlegroups.com). I’ve been terrible about arranging anything on IRC - I know I should to really get us all talking on the same page.

Rajesh apparently picked up a copy of the book RESTful Web Services (O’Reilly), and made a number of suggestions about future direction after reading the book. It really sounded like he got a lot out of the book, so I ordered a copy (through Queen Anne Ave Books cause they’re cool and local). I should have picked it up Thursday evening, but I was being a slacker and only went by and got it this morning. Anyway, I was starting to read it walking home from the bookstore and I immediately spotted a small chapter/segment in there on doing REST services with Django! Apparently Jacob helped out with a chapter in there, so there’s some interesting notes tucked away in Chapter 12 that I’m interested in getting to.

My knowledge is so scattershot (the trouble with being a technology generalist), that I think I’m going to take this book through from the start to make sure I catch up on those niggling little details that seem to slip the the cracks. I’m currently taking a break before seriously digging in to Chapter 2 (writing REST clients), and I’m glad to see a nice level of detail upcoming on how to actually get the various (what I’ll call obscure) HTTP verbs functioning from the clients. They have details on several languages, including libraries to use and example code - Ruby, Python, Java, JavaScript, C#, and so forth. It looks to be a good chapter.

For a long time, my reluctance to really get down to the “PUT” and “DELETE” verbs was because the libraries just didn’t easily support them - and I wasn’t sure what the “right way” of making them work should be. I’ve been pretty horrifically burned trying to learn the details needed to make some of the WS* bullshit specs work, and that translated pretty directly to a reluctance to learn the details of programmatically creating HTTP requests with the appropriate “verbing”. A blank spot, for sure - and one that I’m fixing now.

This project (the Django Queue Service) is mostly a “because I can” project. I’m not heavily using it in anything currently, although Rajesh and Marcus are - and I’ve heard a few other folks have apparently picked it up and run with implementations from it. I’m firmly of the opinion that the desktop and web-space cloud will continue to mix things up. A concept that Brent outlined in his Hybrid Apps Presentation at C4 over the summer of 2006. (Better have a Mac if you want to see that presentation by the way - it’s a VoodooPad document). I’ll be the very last to claim that desktop applications are dead - with the advent of libraries like Sparkle the distribution benefit of the web is just as easy made available to desktop apps as to web applications. And from my own experiences with browsers recently, coming up with fantastic UI functionality that works across browsers is an intense pain in the ass. Its getting better all the time but there are just some things you can’t do there in terms of interactivity and feedback that I’d really like to see and want to take advantage of.

Some very cool news - Django now has a public build site up (http://buildbot.djangoproject.com/)and running, matching the trunk and one of the branches against an assortment of versions of python and databases. Matt Croydon and I are going to be working together to work out the kinks and continue to drive the build environment forward.

All props to Matt for the initial deployment and use during the current sprint. There’s still some bugs and kinks to work out of the system - and we’re going to be hammering those out as soon as the sprint it complete. Since Matt got it functional for the sprint, we didn’t want to mess with the configurations and start throwing false alarms while there was such dedicated forward progress going on.

Right now, we’ve got some base builds and tests running against Python 2.4 and 2.5 with Postgres and SQLite to pretty good effect. Still remaining are stable runs against MySQL, anything with Python 2.3 support, and more effective implementations to make it easier for us to support remote build slaves (Oracle? Windows? anyone?) and enable branches to get built a little more easily.

There is plenty to go, and we’re just getting kicking. I’m sure you’ll hear more as we get things nailed down and continue moving forward.

While out perusing bookmarks and randomly following links, I stumbled across a nice “from the beginning” Django tutorial that includes a little bit of Ajax with the Dojo Toolkit. There’s also a Part 2 to the example, and I think it’s particularly cool that the author went to the trouble of saving the steps as specific revisions within google code so you can “follow along” by checking out revisions.

Of course, this is assuming you’ve got a decent handle on how to check out a specific revision from Subversion. I’m not sure that always applies to folks looking for a from-the-start tutorial, but the idea is really cool.