The arrow of time

Ivan Voras' blog

Python WSGI performance, or - is Python dieing?

I'm looking into starting an interesting little project and of course, one of the most fun phases of it all is choosing your technology stack. I'm going to do something a bit unusual, just because I can, but for starts, I've chosen Python/WSGI combination and within this palette of tools I've come across Werkzeug. It is a light-weight framework which I think will be perfect for the type of code I write - with an eye to what happens behind the scenes.

I've tried Werkzeug with Mako templates and I'm pleased that on current hardware, such a combination which is still very very much high-level and as remote from "bare metal" as I feel comfortable working with, can yield simple dynamic page performance of around 1000 pages per second per CPU. I'm sure this is more of a testament of how fast current hardware is than of Python's performance, which is, frankly, atrocious for today's standards. And this is even before we talk about its GIL which is killing its adoption for high-performance multithreaded server applications.

I'm very saddened to have found out that Psyco, the Python's JIT compiler and the only thing that made Python usable performance-wise outside simple scripting, is abandoned and, what's more, that it doesn't and probably never will work on 64-bit machines (mainly, amd64) - again, this means server workloads are affected the most.

Sigh. Maybe project Unladen Swallow will help move things along eventually if the base implementation is stagnating.

But it will probably be a good project.

(to answer my title question - yes, Python is dieing, but only for the types of server projects I'm doing - no JIT and the GIL are killing it here)

#1 Re: Python WSGI performance, or - is Python dieing?

Added on 2010-07-14T08:26 by brandnewmath

You may want to check out PyPy (http://pypy.org); it's actively developed and I believe some of the principles from the Psycho project are involved.

#2 Re: Python WSGI performance, or - is Python dieing?

Added on 2010-07-14T11:55 by Ivan Voras

I think projects like PyPy and other noncanonical Python implementations are doomed to obscurity because of two obvious reasons: 1) They will always lag behind the canonical one (cpython) and 2) They will never get the support from third party library authors that cpython has. The second reason is the more critical one - I see PyPy doesn't even have all the regular cpython modules implemented, nevermind business-critical ones like database libraries, etc.

#3 Re: Python WSGI performance, or - is Python dieing?

Added on 2010-07-14T11:58 by Ivan Voras

Psyco was the way to go - a plugin to the central implementation which "grew" as the implementation grew and provided a very useful feature - large speedups. It would be particularly useful for environments like web applications which are full of repeatable string processing.

#4 Re: Python WSGI performance, or - is Python dieing?

Added on 2010-07-14T18:32 by Dan Smith

Iron Python does not have the GIL, and is very well supported.

http://wiki.python.org/moin/IronPython

#5 Re: Python WSGI performance, or - is Python dieing?

Added on 2010-07-21T01:06 by Graham Dumpleton

The GIL is not as bad as you make out as main stream hosting options use multiple processes. Read:

http://blog.dscpl.com.au/2007/07/web-hosting-landscape-and-modwsgi.html

http://blog.dscpl.com.au/2007/09/parallel-python-discussion-and-modwsgi.html

#6 Re: Python WSGI performance, or - is Python dieing?

Added on 2010-07-21T02:24 by Ivan Voras

Yes, WSGI will be deployed as multi-process, I'm more seriously lamenting the demise of Psyco.

#7 Re: Python WSGI performance, or - is Python dieing?

Added on 2010-07-29T23:38 by Ivan Voras

Yeah, FastCGI is great, but what if you need to do something non-HTTP? Or anything else that is stateful and needs state keeping and/or inter-client communication through the server? GIL is definitely killing Python.

#8 Re: Python WSGI performance, or - is Python dieing?

Added on 2010-08-09T02:20 by James William Pye

Nah, IMO, pypy is *nearly* poised to replace cpython if it doesn't come through with unladen swallow. *many* might call BS there, but with VPS based deployments, cycles can often be directly mapped into $$ quantities. http://speed.pypy.org/comparison/

If you only need half the app boxes, you can operate at half the price(well, give me some margin of error =).

AFA library support is concerned, the only real problem with pypy is the C-API, which a number of individuals are working to improve. Much is already supported, AFAIK. Problem may not even exist anymore. =)

Also, GIL is primarily an issue on CPU intensive tasks, which you may not actually want to do in Python anyways--well, it's slow. Sounds a bit strange coming from a Python user? Well, this particular user chose Python not only for its clean language, but for its clean and reasonable C-API. Think punting CPU intensive tasks to C code that distributes the work to regular pthreads. Python providing easy access to such functionality often makes an attractive solution.


cheers, jwp

Post your comment here!

Your name:
Comment title:
Text:
Type "xxx" here:

Comments are subject to moderation and will be deleted if deemed inappropriate. All content is © Ivan Voras. Comments are owned by their authors... who agree to basically surrender all rights by publishing them here :)