I’m not as active in the community these days, but I’m still using the project on a regular basis as one of my “secret weapons” to get things done quickly. It’s worth noting that there are some definite changes in the 1.0 release from even what’s in the various books now available for Django. If you’re heading in to the 1.0 release and can’t find something you expect to be there, make sure you check the online documentation at http://docs.djangoproject.com/ - it’s very complete and very worthwhile.

(I haven’t tried Graphite yet, but the screenshots looked pretty nice - so I might be doing so to see what it’s all about.)

LaunchPad is Ubuntu/Canonical’s code hosting platform - similar to Google Code or Sourceforge in concept. They use Bzr for their source control, which is a tad odd to me - I’m more in the mercurial camp of any decentralized SCM - but their web site is really first class. They’ve got a nice system and setup - simple, somewhat bold iconography, obvious components to projects, and a clean look. As a place to encourage project and community participation, I admit that I was immediately drawn in. My own project (languishing at the moment) on Google Code (django queue service) has an interface that feels positively stark in contrast.

I’m not going to move anything over, but I suspect I will pay more attention to the LaunchPad setup going into the future.

I’ve since starting using ReviewBoard (and even contributed a few bits back) and have been very pleased with it. I’m curious to dig into what Code Review offers in comparison, but I already know that I’ll continue using ReviewBoard. For one thing, my office wouldn’t take kindly to code and code reviews happening beyond the constrains of the fire wall. But I am seriously jazzed about Code Review in the hopes that it’s very easy to set up. If it’s an easy, low/no-cost solution for open source code… well, that’d be pretty damn cool. ReviewBoard, for all it’s goodness, can take some time and effort to completely install.

UPDATE: Since I wrote about this, I found the Google project that Guido has used to make (and now share) CodeReview - the project is rietveld. Guido mentions that it’s meant to showcase how to do a webapp using Django templating and views with the Google Application Engine.

Tonight, Google released AppEngine - an elastic computing environment specifically set up with the idea of developing web applications in python. They’ve pushed up a copy of the transcript from their announcement if you’re interested…

It may come as no surprise that the back-end of AppEngine is/was heavily influenced by Django - and in fact the SDK that you can download comes with Django 0.96.1 embedded within it. That’s not to say Django is your only choice, but it definitely it mine.

To that end, there’s an excellent article up at their site on using Django with AppEngine - which basically boils down to: “Don’t use Django’s models, because we’ve got our own back-end datastore thingy, and it works similarly, but from different, than Django’s models and a database. I haven’t looked at the HQL datastore API they’ve got enabled, but I’m going to guess that some Django core developers likely will before too long to see what the potential overlap is for making it more directly usable with Django’s model system. (That guess is PURE speculation, I don’t have any insider knowledge there)

I wasn’t speedy enough on the request to get into the initial pool of folks who can use the beta - but I’ve tacked into the waitling list. It’s not like I really need another cool technology to go spend a lot of time on, but I’m really curious… especially with the idea that hosting some amount of your application might be available for free. Not quite sure how that is going to make sense long term - but I’ll take it while it’s available.

One thing for sure - this is certainly going to bring a boon of new attention to python web development, and likely Django as well.

Review Board includes an automake setup solution, but I didn’t take advantage of it. Partly because I felt comfortable setting up a Django project without it, and partly because I didn’t clue in that it was there and I should use it until well after I’d begun fiddling with all this.

From those basics, here’s some additional tidbits

1. get an instance of reviewboard running:

mysql -u root -p
 (enter password)
create database reviewboard;
GRANT ALL on reviewboard.* TO reviewboard@localhost IDENTIFIED BY "sup3rsekret"

2. set up settings_local.py to match:

# Database backend.  Any supported django database engine should work.
DATABASE_ENGINE = 'mysql'      # 'postgresql', 'mysql', 'sqlite3' or 'ado_mssql'.
DATABASE_NAME = 'reviewboard'  # Or path to database file if using sqlite3.
DATABASE_USER = 'reviewboard'     # Not used with sqlite3.
DATABASE_PASSWORD = 'sup3rsekret' # Not used with sqlite3.
DATABASE_HOST = ''             # Set to empty string for localhost.
DATABASE_PORT = ''             # Set to empty string for default.

3. initialize the database:

cd reviewboard
./manage.py syncdb
 (create an admin account when asked)

4. hook this whole thing up to Apache:

I configured my setup using the provided mod_python template - the Makefile setup system seems to include provisioning it, but I did this all by hand.

Running as mod_python means that the account accessing perforce was “www-data” by default - a user that isn’t normally enabled in our Perforce repository. I worked around that issue by handing down additional perforce specific environment variables:

    SetEnv P4USER readonly-account
    SetEnv P4PORT p4.server.com:1666
    SetEnv P4CLIENT reviewboard
    SetEnv P4PASSWD xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

5. set up the initial data bits to use the service in our environment:
• use “python post-review -d” to find out what perforce is calling your repository.
• Add that repository to the list of repositories in the Admin console of ReviewBoard
• Add at least one “review group” in the Admin form as well
6. Set up a .reviewboardrc for your tree

Place this at the root of your tree. Post-review starts in the current directory and scans upwards to find an instance of this file to know what to do…

# Default Review Board Server
REVIEWBOARD_URL = 'http://10.0.0.25'
 
# Per-repository Review Board servers
# TREES = {
#    '/path/to/local/repo/checkout': {
#         'REVIEWBOARD_URL': 'http://whatever' 
#    }
# }

One other thing of note: post review is sensitive to the Diff Headers in trying to pull out the date. We were using an old version of diff and it had some real trouble with that. We updated to using diffutils 2.8.7 to resolve that.

I pulled down all the parts and pieces. Getting p4python all set up was a bit of a trick, but the wiki had good hints in the right direction to get me rolling. Just spending a few hours with it, I found a few little quirks, and I’ll be adding more back into the project as I find things - might as well, if I’m debugging and fiddling to get things to work anyway, it seems only right to share it back with the community.

So the first thing to note is that getting a ReviewBoard instance set up isn’t really documented on the wiki as yet. GettingStarted, and UserBasics gives some hints, but you’ll still need to read a little code to get it all happening.

Here’s what I’ve got for the first takeaways:

1. To really use this tool (at least with Perforce), you’ll want to use a client like the provided script post-review. The script is pretty straightforward really - it does all the tedious work of creating a diff prior to checkin and shoving it up into the system.
2. post-review supports a config file mechanism, but I don’t yet have that format really working for me. I suspect I’m doing something stupid, but it’ll take a little back-tracing to get a reasonable config set up yet.
3. Getting an instance of this thing rolling requires a few django-specific commands to get the environment seeded up. At a minimum, you’ll need to invoke “./manage.py syncdb” to get the database schemas all in place and rolling.
4. When you get the instance all rolling for the first time, you’ll need to set up an admin account and add a Review Group with an appropriate path. I suspect the subversion folks have this a touch easier than Perforce. With Perforce, the “path” that you enter needs to match the details that you’d otherwise get from running “p4 info“. When I first set things up, I didn’t realize that. The server name we use internally is different than the data presented in “p4 info“, so that took me a bit by surprise.
5. post-review also appears to be sensitive to the format of the output of running a diff. If your diff tool, for example, outputs it’s header data in the form of “Fri, Feb 8 14:23:52 2008″ instead of “08/18/2008 14:23:52″, you’re going to get a very confusing error.

There’s more, I’m sure - but I had to deal with other tasks this afternoon, so I haven’t yet got the basics up and functional. I’ve at least submitted a bug that I was sure wasn’t just a quirk of my somewhat odd development environment. Something that looked like a re-factoring edge case that just hadn’t been caught yet. I included a patch to fix it - although to be honest, I’m wasn’t sure if I should be submitting that to a ReviewBoard instance someone or not, just I just shoved it into the bug.

I think one of the most impressive things about ReviewBoard is that it supports a nice mechanism, and some example scripts, for doing a pre-checkin review. I had naively assumed that I could only get the discussion/review aspects in a post-commit mechanism - without ever even really looking at the project. While I haven’t read the post-review code in depth, it’s pretty clearly set up with the ReviewBoard server as something where a client could significantly help out with this work.

So it’s not smooth and super easy. Yet. To be fair, the devs don’t say it either, and they’ve got a wiki, issue tracker, and all the code available… so really its just a matter of time before I get it working to suit me. I’m really glad they’ve made this available at all, and I’m looking forward to seeing if I can really make it sing.

Update: The answer is yes - you should submit updates and patches to the reviewboard at http://reviews.review-board.org/.

apt-cache search django

… and sure enough, it came back with a hit.

I had a development environment set up in minutes with the following:

apt-get update
apt-get dist-upgrade
# dev environment
apt-get install python-django python-pysqlite2 
# running with MySQL
apt-get install python-mysqldb mysql-server-5.0 mysql-server python-egenix-mxdatetime

Nice!

The results are all from my laptop - 2GHz Intel Core Duo, 2GB RAM. Running python 2.5 and the current trunk of Django (0.97-pre-SVN-6668).

The test ran through the timing for adding messages into the queue and getting/deleting messages from the queue, scaling the queue size from 0 to 25,000 items. The client that ran these tests was put together with httplib2, release 0.40 - running in python on the same machine. Yeah - so pretty much factor out network latency on this one…

The adding stayed a fairly consistent 8 ms, deviating around a bit - but mostly there.

The combination of getting a message and deleting it (in our API, the functions are separate - so it means two calls back to the service) rose linearly with the size of the database. At a base level, it was running about 20 ms and by the time the database size was at 25,000 items the “get and delete” operations were taking roughly 165 ms.

For SQLite-on-disk database backend, I’m pretty pleased with those results.

Rajesh apparently picked up a copy of the book RESTful Web Services (O’Reilly), and made a number of suggestions about future direction after reading the book. It really sounded like he got a lot out of the book, so I ordered a copy (through Queen Anne Ave Books cause they’re cool and local). I should have picked it up Thursday evening, but I was being a slacker and only went by and got it this morning. Anyway, I was starting to read it walking home from the bookstore and I immediately spotted a small chapter/segment in there on doing REST services with Django! Apparently Jacob helped out with a chapter in there, so there’s some interesting notes tucked away in Chapter 12 that I’m interested in getting to.

My knowledge is so scattershot (the trouble with being a technology generalist), that I think I’m going to take this book through from the start to make sure I catch up on those niggling little details that seem to slip the the cracks. I’m currently taking a break before seriously digging in to Chapter 2 (writing REST clients), and I’m glad to see a nice level of detail upcoming on how to actually get the various (what I’ll call obscure) HTTP verbs functioning from the clients. They have details on several languages, including libraries to use and example code - Ruby, Python, Java, JavaScript, C#, and so forth. It looks to be a good chapter.

For a long time, my reluctance to really get down to the “PUT” and “DELETE” verbs was because the libraries just didn’t easily support them - and I wasn’t sure what the “right way” of making them work should be. I’ve been pretty horrifically burned trying to learn the details needed to make some of the WS* bullshit specs work, and that translated pretty directly to a reluctance to learn the details of programmatically creating HTTP requests with the appropriate “verbing”. A blank spot, for sure - and one that I’m fixing now.

This project (the Django Queue Service) is mostly a “because I can” project. I’m not heavily using it in anything currently, although Rajesh and Marcus are - and I’ve heard a few other folks have apparently picked it up and run with implementations from it. I’m firmly of the opinion that the desktop and web-space cloud will continue to mix things up. A concept that Brent outlined in his Hybrid Apps Presentation at C4 over the summer of 2006. (Better have a Mac if you want to see that presentation by the way - it’s a VoodooPad document). I’ll be the very last to claim that desktop applications are dead - with the advent of libraries like Sparkle the distribution benefit of the web is just as easy made available to desktop apps as to web applications. And from my own experiences with browsers recently, coming up with fantastic UI functionality that works across browsers is an intense pain in the ass. Its getting better all the time but there are just some things you can’t do there in terms of interactivity and feedback that I’d really like to see and want to take advantage of.

All props to Matt for the initial deployment and use during the current sprint. There’s still some bugs and kinks to work out of the system - and we’re going to be hammering those out as soon as the sprint it complete. Since Matt got it functional for the sprint, we didn’t want to mess with the configurations and start throwing false alarms while there was such dedicated forward progress going on.

Right now, we’ve got some base builds and tests running against Python 2.4 and 2.5 with Postgres and SQLite to pretty good effect. Still remaining are stable runs against MySQL, anything with Python 2.3 support, and more effective implementations to make it easier for us to support remote build slaves (Oracle? Windows? anyone?) and enable branches to get built a little more easily.

There is plenty to go, and we’re just getting kicking. I’m sure you’ll hear more as we get things nailed down and continue moving forward.

Of course, this is assuming you’ve got a decent handle on how to check out a specific revision from Subversion. I’m not sure that always applies to folks looking for a from-the-start tutorial, but the idea is really cool.

At OSCON 2007, I was hunting around for something to use to deal with background processing initiated from my web application. Not finding anything that I could immediately use and implement, I took it as a challenge to knock out something in the time I was there. The result is the django queue service.

It was later in the week (at the Django BOF) that I learned about TheSchwartz and Gearman, which Brad Fitzpatrick is hacking away it to make a queue mechanism available for fellow Python and Ruby geeks.

At the time, I thought “I wonder if I can hack this out in a week…” Well, yes I could. I modeled the service after the REST API from the Amazon Simple Queue Service, because that seemed both reasonable and effective. And I didn’t mind the idea of having my own Queue that (mostly) matched that API that I didn’t have to pay anything to use.

The project also contains from code from the CherryPy project. Mark Ramm (a really nice fellow, by the way) gave a session at OSCON about WSGI servers that was really intriguing. I’ve been looking for a “pure python” service to run Django with for a while, and after chatting a bit with him, thought I could probably make it happen with the CherryPy WSGI server. So I did. I have a little piece of linkage code that fires up the CherryPy WSGI server code and links it right on through into Django. I didn’t want to use the rest of CherryPy, so I nicked that code and dropped it into this project as a nice reference and a handy way to run the queue.

I suppose this project could easily end here, and of course there was the “but I want to…” list of things that I had in mind to do before I released it into the world. Well - as they saying goes: “Now is better than never.” so out it goes.

If you are interested in getting involved with the project, sending me patches, or whatever - let me know. You can leave a comment here or use that spiffy Google Code Hosting issue tracker. The project is functional, but not brilliant - an not entirely cleaned up in terms of its API. But enough excuses - its there, feel free to use it or not as you like, contribute or not as you like.

I also got a chance to meet Paul Bissex, who wrote briefly about the first day of OSCO. He’s in the Python WSGI session with me now.

Speaking of which, I’ve now learned about FAPWS from William (downloads) which was one of the WSGI servers mentioned, and apparently tested with Paul’s simple wiki code. It uses PyEvent under the covers, and apparently achieves some very nice performance characteristics.

I haven’t heard how the initial lunches turned out, but the concept is really interesting and I think I’ll have to sign up for a lunch or two in the future when I’m not feeling so drained of the ability to be an extrovert.

Not surprisingly, I downloaded the source and immediately tried to see if I could get it to work on my “day” desktop. Unfortunately, they’re using the python class POpen3 inside the code somewhere, and that’s not available on Windows - so I’ll have to wait until I can get to my desktop at home or a build a quick VM to try this out.

Just the fact that it supported Perforce out of the box damn near landed me on the floor. Of course, these guys are from VMWare, and I knew they used Perforce - but that connection didn’t click in my brain immediately. Since I’m using Perforce at work, I’m really hoping to see if I can cobble something together and do a little “check it out!” show and tell with the groups around me. Regardless of their uptake, it’s a great tool from the brief look I’ve taken from it - and open source to boot. (What was I saying about not having a lot of time recently? damnit…)

Well, hopefully I can just use it out of the box, and maybe even give a quizical eye to some friends who are using Perforce to see if they are interested in trying it out for their internal use.

Now I wonder what it would take to have it pull diffs from Trac bugs in a certain well known Trac instance…

He took a really interesting tack - instead of tieing OpenID into the authentication system that comes as a part of Django (contrib.auth), he overlaid the session tidbits with a middleware component that passes information into your views (and potentially your templates) as an extension on the request object. So instead of using the Django auth mechanisms for your OpenID users, you can let it ride without permissions and really even heavy tracking of those users just through their OpenID.

Obviously that may not be quite what you’re looking for, but in taking advantage of enabling some light level of authentication and identity to a public site, it’s really in a sweet spot.

I was really impressed with is how he packaged the whole kit for someone to take a look at, learn, and use. First off, there’s a great “single-page” documentation page right in the source, along with a .txt counterpart. The in addition to the core code itself, he packaged up a super-light django “example” application that most folks will be able to pull down and immediately fire up to get a sense of what it is doing and how to use it.

In working with Fritz to implement TrenchMice’s OpenID consumer support, we ran into a number of problems when we bound OpenID into the heavier weight contrib.auth setup. “Required” fields were not always returned from the OpenID provided when requested through the registration extensions, and he ended up iterating on the process a couple times to finally route the OpenID registration process into their normal registration process when there wasn’t enough information to fully process things.

Clearly, if you don’t need the heavier weight (contrib.auth) setup, Simon’s contribution will be a godsend to your application.

For one, the AddManipulators (being deprecated as things move to newforms instead of the Manipulators setup) don’t play nicely with the removal of LazyDate and auto_add_now functionality from the older models setup. My terribly hack workaround, since I don’t use too many of the AddManipulators, is to tweak them when I run into it.

In particular I’m storing “creation_time” for some objects (which had been set to auto_add_now), and the AddManipulator decided that it was required, regardless of “null=True” or other goodies. I was overriding the save() method in the model - so I’m getting the data… but the Manipulator was being a pain. My tweak for the Manipulator went something like this:

book_manipulator = Book.AddManipulator()
for field in topic_manipulator.fields:
  if field.field_name.startswith('creation_time'):
    field.is_required=False

Which appears to have resolved the whole “creation_time_time” and “creation_time_date” fields being required and stuffing up my otherwise nicely functioning forms. Hopefully nobody else will run into this and need this hack (at least until I get rid of the AddManipulators in favor of the newforms setup) - but just in case, I thought I’d make it nicely available for Google.

Being a fan of that particular framework, I find it pretty interesting. It’s not quite in the realm of the hybrid application that Brent was talking about, but it’s definitely another take on how you can get some functionality on the desktop in a consistent way pretty darn quickly. I still think straight Desktop style apps that do the cool whizzy feedback will keep their place, but the possibilities for something like DjangoKit to make a quick appliance wrapper interface is really appealing to me. (DjangoKit on Apple TV?)

I haven’t tried out DjangoKit, but I’ve got all the prereqs (that would be a copy of PyObjC on my laptop). It’ll be interesting to see where it goes.

Deploying TrenchMice has a really nice bit of detail on how they’ve set up Django for development in a team with multiple environments sharing the same code base.

The Enabling OpenID article details out the rough sketch of the code they used to tack on OpenID authentication to a Django web application.