Working with Stackless Python and Microthreads
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
If you got the chance to improve the performance and capacity of your favorite programming language for free—by just dropping in a replacement interpreter and a few additional libraries—wouldn't you jump at it? Such a chance exists for Python. By installing the so-called "stackless" variant of Python instead of the vanilla flavor, you get access to microthreads, a version of threading that allows you an order of magnitude or two more threads on your machine than standard Python. Apart from that, there is the very advanced and interesting possibility of using co-routines and continuations.
Stackless Python was written by Christian Tismer and is available from www.stackless.com. The main difference from standard Python, as its name implies, is that Stackless Python doesn't use the C stack. Standard Python, which is written in C, uses this rather limited memory area to store the call stack (the stack of Python functions that are being called from each other) and the function frames (the representation of the state a certain function has reached) between thread switches. In Stackless Python, these vital data structures are stored on the heap—the potentially limitless area of memory in which all objects are created.
Essentially (and much simplified), this makes it possible to treat the state that a function is in when calling another (the execution frame) as a Python object, and this is immensely powerful. Continuations, co-routines, and generators are within arms' reach, even if using them is a bit hairy. To make it easier to benefit from the advantages of Stackless Python, Will Ware, Just van Rossum, and Christian Tismer have written the uthread Python module, which builds microthreads on top of continuations. This module is just as easy to use as standard Python threads—perhaps even a little easier.
Note that Stackless Python is not compatible with Jython—the version of Python that runs in a Java Virtual Machine (JVM). In fact, it's impossible to implement microthreads efficiently in Java. The basis of Stackless Python is the removal of the use of the C stack in the Python Virtual Machine. If you want to have the same kind of power in Java, you'd have to patch the Java Virtual Machine to avoid the C stack. Not impossible, but extremely difficult because of the wide variety of JVM implementations. The only way of giving applications access to more than a few threads on Java is to write your own cooperative scheduler in Java, and let your "threads" run on top of that. This scheme, however, is incompatible with Jython. You can run microthread-based applications on Jython, since it includes a small compatibility mode with system-level threading, but don't expect to get too much performance out of it.