I expected there were some optimizations in there that I could make, but hadn’t yet teased out. In fact, I knew about pause() keeping things off the run loop until it was “woken” – but I hadn’t gone back and read through the code to see if the way I was injecting messages into an “inbox” would properly awake the target.

It is more memory hungry than Stackless – to be expected since they ripped out the whole stack frame setup, their smallest tasklets are just tiny amounts of memory. Even still – it was the scaling aspect that I really wanted to dig into and see.

I fully expect I’ll keep fiddling and teasing with this setup for a while. I’ll post more on my blog as I roll it forward.

To
….def main(self):
……..yield 1
……..while 1:
…………while not self.anyReady():
…………….self.pause()
…………….yield 1
…………while self.dataReady(‘inbox’): # Note change from “if”

Oh, the reason why the stackless version would be interesting is because it’d give a relatively simple optimisation path for code if needed

Yes the project is still going Interesting benchmark.

First of all, your code, on my machine:

/media/usbdisk/Kamaelia> python /usr/lib/python2.5/timeit.py “import hackysack” “hackysack.runit(10,1000)”
10 loops, best of 3: 182 msec per loop

/media/usbdisk/Kamaelia> python /usr/lib/python2.5/timeit.py “import hackysack” “hackysack.runit(100,1000)”
10 loops, best of 3: 820 msec per loop

/media/usbdisk/Kamaelia> python /usr/lib/python2.5/timeit.py “import hackysack” “hackysack.runit(1000,1000)”
10 loops, best of 3: 9.23 sec per loop

I kinda lost patience at this point – so I made a simple change:

Changing this part of your code:
def main(self):
yield 1
while 1:
if self.dataReady(‘inbox’):

To
def main(self):
yield 1
while 1:
while not self.anyReady():
self.pause()
yield 1
while self.dataReady(‘inbox’): # Note change from “if”

Yields the following change:

I suppose the difference here from threads is that when we pause, we do actually
pause and get removed from the run queue.

/media/usbdisk/Kamaelia> python /usr/lib/python2.5/timeit.py “import hackysack” “hackysack.runit(10,1000)”
10 loops, best of 3: 206 msec per loop

/media/usbdisk/Kamaelia> python /usr/lib/python2.5/timeit.py “import hackysack” “hackysack.runit(100,1000)”
10 loops, best of 3: 267 msec per loop

/media/usbdisk/Kamaelia> python /usr/lib/python2.5/timeit.py “import hackysack” “hackysack.runit(1000,1000)”
10 loops, best of 3: 816 msec per loop

I can’t reliably test above that because I don’t have enough memory. However, even with that caveat:

/media/usbdisk/Kamaelia> python /usr/lib/python2.5/timeit.py “import hackysack” “hackysack.runit(5000,1000)”
10 loops, best of 3: 2.77 sec per loop

That figure includes my machine swapping. Despite that, I think that’s somewhat better

I’ve got a feeling that this is much more memory hungry – probably need to look into that. The optimisations to allow this were aimed at allowing the system to use next to zero cpu when it can (since we were doing the changes to allow shelling out to transcoders to utlise as much CPU as possible). As a result memory wasn’t a consideration at the time.

(these are running on an Open SuSE 10.3 laptop : Intel(R) Pentium(R) M processor 1.60GHz, 496 MB RAM, Python 2.5.1 & firefox sitting in the background eating up 372MB RAM….)

Looking at the Why Stackless link, this is still slower than stackless (which makes sense, stackless *does* remove a layer of complexity), but it appears to have similar scaling properties. Also, I suspect part of the issue is the fact that our scheduler is implemented in python rather than C.

It’s tempting to see what would happen with the same code running under stackless, and then changed such the the components were tasklets rather than generators. (inboxes/outboxes could then be implemented using channels after all)