Rhonabwy

I really like the concept of Pinax, but I’m beginning to see the darker sides of the project. Here’s the fundamental issue:

Pinax does a completely kick ass job of making a combined and base project that pulls together a lot of pretty darn good re-usable django “applications” into a single project. What it doesn’t do well is manage any of the dependencies in that tree, or make it easy to use only a portion of some app. If you want to use Pinax the way Pinax is already developed, you’re rockin’. If not, you’re re-working quite a bit of code. And the worst part is that code is almost all opaque to you and hidden in the pinax framework.

The application I wanted to pull in to my basic “pinax” project was “photos” – some simple photo uploading. Yeah, cool – uses photologue under the covers. What’s not clear is that photos also has dependencies on grouping and tagging, and if you review the dependencies documentation, it looks like there’s circular dependencies embedded in there.

What I ended up doing was to take the “photos” application from the Pinax project directly, fork that, and drop it into my project. Then I had to do a shit-load of renaming and stripping because “photos” is built in to Pinax as a default – so “photos” in my app became “photo” and I was scrambling through code to remove the bindings into commenting, threaded commenting, grouping, and so forth.

I did get it working, and really all told I spent less time stripping out stuff than I would have putting it together from scratch. But the only reason I was able to do it that efficiently is because I’ve already been through the pinax project’s source, had it on my laptop, and was a fairly comfortable and competent Django programmer. Any newcomer hitting this would have been toast, and badly confused.

When I asked around the Seattle Django user’s group related to Pinax, several of them said they’d been shying away from something like Pinax because it was easier for them to simply including the good open-source django apps into their work already. At the time, I thought “Well, maybe that just makes Pinax easier and more effective for newbies into Django.” Now I’m reconsidering.

I don’t have a good answer to the problem, and I’m sure I’ll continue to use Pinax and maybe even dig deeper and see what can be done to resolve this kind of issue. I think the idea of the Pinax project is fantastic, I just wish it was a little less “tightly coupled” under the covers and provided more transparency into it’s depths.

No secret, I prefer Mercurial, and I think pretty highly of BitBucket too, but a whole lotta folks are using GitHub, and the platform is pretty damn good for general project collaboration, especially for open source projects.

The only problem, in my opinion, with GitHub is having to use git! I find it darned confusing, and I’m pretty confident and effective with source control tools. Not that I don’t appreciate it’s power, I’m just not in favor of that particular level of complexity/ease-of-use tradeoff.

Anyway, while I’m not great with it, I can figure it out. And since I did, I wrote it up. The example is available for the GitHub project letters if you’re interested. It should be really easy to translate into any other project on GitHub, and has the basics for collaborating, including getting code from anyone’s fork of the project and pull it into your own.

http://github.com/ccgus/letters/blob/master/README-CONTRIB.markdown

Hope it’s useful… I’ll be doing another one of these at some point for BitBucket and mercurial…

I was reading through back-links that I’ve shoved into Delicious over the past few years, doing some “where was that…?” sorts of things. I realized that I had about 1500 entries stashed into my delicious links, so I thought I’d see what the tag-cloud looked like. Pretty darn representative of my interests in computing. Probably more so than my blog actually.

I work, live, and play in technology. My job today is in technology – doing software to run software (a bit cyclical, yes)  - operations, in short. Almost everyone surrounding me at work is technical, smart, detailed, and very, very into computing and what it can do. To be honest, I’m mostly surrounded at work by people who happily use and understand Windows as a common operating system.

They’ve all been buzzing about the iPad. Shit, anyone in technology has been buzzing about it. Microsoft and HP (or laptop/tablet vendor of choice) had a decade to break this code and didn’t manage it. I’ve been hearing almost exclusively around why Apple fucked up, blew it, and why the Kindle and netbooks will continue to rule the roost. I won’t even go near the whole “Flash thing”.

I think they’re all dead fuckin’ wrong. Pardon the vuglarity, or not, as you please. It’s needed on this one. The iPad is a sign post. A marker of where we’re heading, and a sign of the change to come. I saw the preso, reading a few IRC channels and frantically refreshing web pages – and my thought was “Holy shit! I’m seeing something just like the first copy of NCSA Mosaic running on a Mac. This is going to change the world…”

And I found that I was completely unable to articulate this in any way that I could present to my coworkers. I didn’t bother demanding they listen, or that I was right, or any of that nonsense. I see it coming – they will too, eventually. In the past day or two, however, some very articulate fellows have done an amazing job putting down into text what I couldn’t. If you haven’t read these – do.

• Future Shock (Fraser Speirs)

• New Worlders (Steven Frank)

Thanks gents – I appreciate the words and thoughts – especially since I couldn’t seem to articulate them.

I have a few friends that I’m bringing up to speed pretty quickly on Django. The basics are going well, and I’m providing the foundations for the effort. To take advantage of all the good stuff out there, I’ve started diving deeply into Pinax, and then wrapping some automation around it with Fabric to make it really easy to work on new apps and projects.

That’s the whole reason I posted gist 284927 yesterday on setting up a remote app server for a pinax project with Fabric.

The biggest hurdle that I have with the fellows I’m bringing up to speed is them understanding what’s IN the project that I’ve already slapped down. Even a basic pinax project has a huge amount of complexity for the newer django developer – it’s pretty overwhelming. To try and mitigate that complexity, I’ve started a side project making Pinax cheat sheets. (http://github.com/heckj/pinax_cheat_sheets).

This is not intended to replace Pinax documentation, and in fact I hope that I can eventually tweak these around and put them into the pinax project. I have a huge respect for it (and I use it), so here’s a little bit of detail for contributing back.

I am very happy to take on any changes, edits, updates, etc – from others. There’s a lot of folks who’ve worked very hard to build all these components, and I’m not writing from the perspective of the creator, but of the “code spelunker”. I’m sure I’ll get some of this stuff wrong – so if you see something, please let me know.

I’d ideally love to receive contributions in the form of pull requests from GitHub. Just fork that project, make your edits, and when you’ve pushed them back up – send me a pull request. I don’t promise that I’ll include everyone’s contributions, but to be honest I don’t expect to be too darn picky.

I’ve written the cheat sheets in markdown, mostly because I know it, and secondarily because it renders nicely as HTML when viewed through GitHub.

The start of the project has the following cheat sheets up:

• django.contrib.auth
• django.contrib.contenttypes
• django.contrib.humanize
• django.contrib.sessions
• django.contrib.sites
• pinax.templatetags
• notification

I have a bunch more external apps to go before I hit the internal Pinax apps. I’m working through everything that’s included in a Pinax “basic_project”.

• django_openid
• emailconfirmation
• mailer
• announcements
• pagination
• timezones
• ajax_validation
• uni_form
• staticfiles

And then the internal apps:

• basic_profiles
• account
• signup_codes
• about
• django.contrib.admin

I spent the better part of this morning screwing around with Fabric to get even a basic deployment working correctly with Pinax. I may be doing this horrifically backwards… Since I finally nailed something that worked for me and I couldn’t find much out there on the web, I wanted to post up a result so that other folks could use/riff etc.

http://gist.github.com/284927

Please feel free to replicate/clone/use/etc whatever to your heart’s content. It’s built around deploying a Pinax 0.71 project clone that also uses South for schema migrations, and the project is housed in BitBucket.

The other fabric file I found that’s referenced frequently was a part of the inspiration here: http://gist.github.com/212366. It’s a fabfile that’s intended to provide some simple rollback and expects to be used from GitHub – and has some really nice documentation associated with it. Maybe I’ll work on mine to add some of that doc… but mostly I’m just making it available as is.

In my Redmine investigations that I’ve been writing about recently, I ran across an issue between a plugin and the latest version of Redmine. I really don’t know the first thing about about rails – I can read it, but I can’t always make sense of it, and I certainly don’t know the “expected” bits (some might call constraints, others I’ve heard call it “the magic”). So when I took the fix posted to the forums, slapped it up in a Github fork, and submitted a fork request – I had no idea if it would be wanted.

I’ve got to say that I’m incredibly gratified that it was accepted back into the main branch. All I really did was make it darned easy to pull in a patch someone else created, but even doing something simple like that I think made a difference.

I’m a bit more proud of the patches that I submitted to pyrrd (a real fix and some better doctests). Those were, honestly, harder – but really only because I found the Launchpad interface and Bzr to be harder to navigate and use than Git and Github. (I still assert that the best thing about Git is Github). Duncan (maintainer of pyrrd) prefers that system, so I created a branch there and submitted the patches. Didn’t quite do it 100% for him – I bundled two fixes into the same branch instead of a branch/fork per fix – but in the end he took them in and I think appreciated the help.

A little side light was helping out Ned with my writeup of using Hudson for python/django projects. Chris Heisel took it that one step farther and walked through using Hudson for Django projects as well. So I ended up contributing and seeding a bigger conversation with that too.

There have really been few projects that I’ve provided back to this year – it’s been a busy year at work, but much of it has been more people and process oriented rather than making code happen. Still, contributing back in whatever areas gives me a boost – and at the end of the year I like to look back and think that even in some small ways, I made something available for others.

What’s this coming year going to bring? Heh – who knows. I’ve got a few things rolling (at work and outside of work) that should prove to be really interesting. One is an open source monitoring project, although at the moment in the very, very early stages. The others I’ll have to leave off for talking about another time.

After a few days of playing and working with Redmine and some plugins, here’s where I’ve ended up:

• Redmine 0.8.7 (latest stable branch)
• Mercurial repository
• redmine charts plugin (http://github.com/mszczytowski/redmine_charts)
• redmine backlog plugin (http://github.com/relaxdiego/backlogs)

I tried the redmine scrumdashboard plugin (http://github.com/thus/redmine-scrumdashboard-plugin), but was never able to get it even functional. The whole thing stacktraced under 0.8.7 and I’m just too ill informed on rails to be able to effectively debug it. I did log it as an issue (http://github.com/thus/redmine-scrumdashboard-plugin/issues#issue/6) though.

While I was working with the backlog plugins, I found a few quirks. In one case, it wasn’t editing anything from the backlog tab in Redmine. That turned out to be an issue with a recent Redmine feature related to cross site forgery protection. There were a couple of posts in the plugin forum and some associated bugs at the bug tracker. On the forums,  Eric Doughty-Papassideris suggested a solution, so I forked up the github project and posted that fix for anyone to use. Even made a pull request back to the master project – don’t know if it’s 100% the right thing to do for the solution, but it does do the trick.

The backlog plugin, while working, did have a quirk that I never quite identified. I enabled the plugin and slapped in a bunch of issues, and one of them caused the plugin to render in a really odd way. I suspect it wasn’t defending well against stupid parameters, but I wasn’t ever able to figure out why the thing was rendering all whacked. I ended up deleting that example project, but on another it appeared to work correctly.

Of the two, the scrumdashboard plugin appears to be in a “quiet phase” – little commit activity and I suspect it’s running afoul of recent Redmine updates and features. Backlog seems to be keeping up better, so I expect that’s where we’ll focus our usage.

Update:

One thing that I found surprising – If you’re using Redmine with Mercurial, the redmine instance wants to have a mercurial repository *local* on that machine – you can’t point it at a remote mercurial repository and have it function correctly. It only wants to read off the local filesystem to do it’s work. That surprised me – as I was planning on having a separate machine for the repository originally to separate the concerns.

Ironically, the choices for an in-house integrated project management toolchain are, to my mind, significantly more limited than the straight up open-source market. I have been looking and watching several projects over the past months for something that I could use for my development team at work (i.e. behind a corporate firewall and not publicly available) to support a more open, transparent view of what was happening.

Trac, of course, is one I’ve used and really enjoy. Trac, however, doesn’t really support multiple projects in any integrated way – and we’ve got way more than one project. From the activity I’ve seen, it’s not about to any time soon. Variations on that theme include DrProject, and more recently Basie. They’re all interesting, but I didn’t see either DrProject or Basie as really ready for prime time and I wanted to be able to quickly grow beyond the basics. Two others that are obviously available are Redmine and Retrospectiva.

Ultimately, I’m heading down the road with Redmine for the following reasons:

1. It’s a decent and clean user interface that has several years of polish now around it
2. The system support plugins, and there’s several plugins that I really want to take advantage of
3. Redmine and Trac appear to have the most active communities around them and driving them

The story cards and burn-down charts from Retrospectiva’s AgilePM addition were darned compelling. But ultimately it has a more limited set of folks focusing on that project, and I think I can achieve the same effects with plugins that are available from the Redmine community. Many of these are hosted on GitHub, and even more are in development or some middlin’ shape at the same site. And finally, I’d much rather deal with a partially implemented plugin than a partially implemented core system.

If I could host our development outside of the corporate firewall, I’d likely be looking at Bitbucket or Firefly for the work. I like Bitbucket partially because it’s all mercurial based, and I need to support some Microsoft based development. Git will work with enough effort, but I don’t want to go down the road of trying to explain all the detail needed there. I love the functionality in GitHub, and frankly think it’s a slightly better and more aggressively updated solution that BitBucket, but the git thing – while I’m happy enough with it – just sinks me otherwise because I know I’ll end up spending hours attempting to support it on a windows environment, and I just don’t want that hit. Firefly is ActiveState’s “let’s take and commercialize Trac by hosting it” – and they look like they’ve done a nice job of it. The agile methodology plugins and burn-down charts are pretty darn compelling.

It should be noted that I also took a pretty deep look at Launchpad. Even though it’s open sourced, it’s a damned complex thing to potentially integrate and build up for internal use, and its integration/collaboration features just haven’t been as compelling to me as the machinery that GitHub or Bitbucket provide. The “branch and request a pull” feature set that supports the massive branching and merging for features, bugfixes, etc. is just so much easier in Github or Bitbucket – and the wiki support to the other components is also far superior.

If you’re looking for something yourself, definitely keep an eye on the market and projects though. RIght now it’s just December 2009, and it’s clear there is a lot of effort, momentum, and a ton of good ideas that people are just begging to implement in this space. If I go to make this same decision in 6 months, there’s no guarantee that the landscape won’t have changed dramatically and a different solution would be better.

What else does a geek do on Christmas eve? How about some ruby on rails debugging, especially when I don’t really know anything other than the basics of Ruby on Rails.

So here’s what I’ve done and what I’ve learned, for the great search engine in the sky and the next time I find myself in this pickle.

I’ve installed Redmine on a CentOS box, which worked out really well. I cribbed the install notes from http://blog.itsmine.co.uk/2009/01/22/howto-install-subversion-and-redmine-on-centos5-rhel5/, which is a fantastic walk-through to the process.

So first is getting authentication working with Redmine. It includes a nice LDAPS:// based built-in mechanism, but there’s little detail around configuring it. The best is the wiki page RedmineLDAP on the Redmine wiki. If you’re doing LDAP authentication, you probably want LDAPS – otherwise you’re shipping passwords in clear over the wire. The default port of LDAPS on Active Directory is 636.

The other key thing you need to know is that even though you configure an LDAP connection in the redmine application (under http://YourRedmineHost/auth_sources/list), when you test it – it’ll test positive even if it isn’t binding into Active Directory and returning a result. Why? Now that another story – and one I don’t have clearly nailed.

To start, I enabled debugging in Redmine by editing /var/www/rails/redmine/config/environments/production.rb and inserting the line:

1

config.log_level = :debug

(the default, if you’re curious, is config.log_level = :info)

Restarted the service and now I’m getting a lot more detail. Turns out the LDAP authentication pieces really only trigger useful logging if the search mechanism that looks for the ID is successful. I found the ruby code doing the lifting at redmine/app/models/auth_source_ldap.rb – and starting adding in additional debugging methods. What I found was that the system wasn’t throwing any exceptions, but the result of a call to ldap_con.search() wasn’t returning any values.

What I haven’t been able to solve is why that search wasn’t returning any values.

Since I’m not super conversant on hacking on ruby (yet – I think I’m about to learn the hard way), I switched to a different tactic to see what was happening: tcpdump.

The command line to watch the LDAP protocol stream in plain-text as it flows is:

1

tcpdump -l -s 1024 -A -vv port 389

(you’ll need to run that as root, and turn of LDAPS so you can actually see the darned protocol). Wireshark would be a hell of a lot easier, but I’m doing this on a remote RHEL server and didn’t have a GUI to wrap into the darn thing. What I saw from that was a string embedded in the protocol noise:

LdapErr: DSID-0C090334, comment: AcceptSecurityContext error, data 525, vece.

Why this wasn’t propagating an exception or error in Rails is not clear to me (which is why I think I’m about to learn more rails hacking, the hard way). I’ve tweaked things around to get different exceptions, but what it all amounts to is that I’m not yet getting a good/successful bind on LDAP from which I can do searches.

I’ve managed to get LDAP authentication working from python (and Django) within my environment, so I’m familiar with the proper base DN and some of the attributes that are available for use from our Active Directory – but so far, no love in the ruby world.

When you’re configuring the LDAP, the code is written so that Redmine has to have an account and password that will bind with LDAPS properly and be able to return search results. If you’re doing this against active directory, make sure you include the domain in the account you’re binding (i.e. YOURDOMAIN\youraccount). That made the difference in getting a successful bind to LDAP. Using the correct domain is what got me past the bind issue. The fact that it wasn’t successfully binding and not propogating an error seems like a bug – that’s been logged today as issue 4483

(by the way – the time when I want more screen real estate is when I have 5 terminal consoles and two different web browsers open debugging something like the redmine application)

More later… still debugging, but now I’m diving into using ruby script/console to interactively figure out this LDAP code in ruby.

UPDATE: So the final solution was all about having the right configuration entries in the fields to make it work. I ended up using ruby script/console to manually drive values until I had a success, and then the results were really clear as to what I needed to use.

For anyone else coming along and trying to same thing, here’s what worked for me (minus my password…) to learn what attributes were being passed back from LDAP:

1

ruby script/console

1
2
3
4
5
6
7
8

require 'net/ldap'
options = { :host => 'my.adhost.com', :port => 389, :encryption => nil, :method => :simple, :username => 'DOMAIN\jheck', :password => 'mypassword' }
ldap_conn = Net::LDAP.new options
ldap_conn.open { }
ldap_conn.auth('DOMAIN\jheck','mypassword')
ldap_conn.bind
filter = Net::LDAP::Filter.eq('sAMAccountName','jheck')
ldap_conn.search(:base => 'DC=MYDOMAIN1,DC=MYDOMAIN2', :filter => filter)

A related update/patch submitted to Redmine a few months ago is issue 4283 which solves a quandry I have about needing a separate account to authenticate through LDAP. I’ll be trying out that patch next monday.