Nov 27
generator based concurrent programming - redux
A little earlier in November, I wrote up a little concurrent programming exercise using Kamaelia. Michael Sparks commented a bit on it, and suggested some improvements - which was really cool. He’s the guy that make Kamaelia, and helped me a few months back when I was first investigating “hey, will this work?”
So I took used his code updates and I wanted to share the difference in execution speed:
10 concurrent objects, 1000 loops:
python timeit.py -s “import hackysack” “hackysack.runit(10,1000)”
Originally: 10 loops, best of 3: 127 msec per loop
Updated Code: 10 loops, best of 3: 142 msec per loop
100 concurrent objects, 1000 loops
python timeit.py -s “import hackysack” “hackysack.runit(100,1000)”
Originally: 10 loops, best of 3: 587 msec per loop
Updated Code: 10 loops, best of 3: 180 msec per loop
1000 concurrent objects, 1000 loops
python timeit.py -s “import hackysack” “hackysack.runit(1000,1000)”
Originally: 10 loops, best of 3: 6.05 sec per loop
Updated Code: 10 loops, best of 3: 550 msec per loop
10000 concurrent objects, 1000 loops
python timeit.py -s “import hackysack” “hackysack.runit(10000,1000)”
Originally: 10 loops, best of 3: 60.4 sec per loop
Updated Code: 10 loops, best of 3: 4.26 sec per loop
So what’s this all mean? Well - mostly that using generators for this sort of shared in-process concurrent style execution is possible and effective. Stackless Python still kicks it’s ass hands down - but the benefit you get right now is you can use a stock Python distribution and get the benefit of the tasklet coding style that makes some things really effective.
Michael even has some recent fiddling that uses forked python processes, talking between them to get multiple cores into the game.
December 1st, 2007 at 2:25 pm
[...] I poked around in Eventlet this morning - between shopping runs and helping Karen move things around in the basement. I’d hoped to knock out a quick hackysack experiment with that stuff - since it sounded about the same - and see how it did. I over-estimated by ability to quickly grok Eventlet, and it’s underpinnings of greenlet as well. [...]
December 9th, 2007 at 8:08 pm
[...] So at a brief reading of the API and description, it looks Michael has implemented the critter. So far, all the fiddling I’ve done has taken advantage of just a single core - in short, the little tasklets aren’t running concurrently - they’re explicitly sharing back and forth. Michael’s clearly thought further down the road there and determined that it would be nice to get all 8 cores of an Octo-Mac into the action. (Well, I expect that’s not QUITE what he though, but that was my immediate translation of it) [...]