|
FAQ
Frequently Asked Questions
Q: Why base Unladen Swallow on 2.6.1 instead of 3.x?A: We're based on the 2.6.1 branch because of internal considerations. Google's currently running 2.4 internally, and we want to offer our internal users an easy and stable target for porting. Python 2.7 is still in flux, and will be through the entire life of this project, and 3.0 would be too big a jump for our existing client applications. We want to make it as easy as possible for large applications to obtain the performance benefits we're offering. That said, we will port our changes to 3.0 for inclusion in mainline CPython. Whether our changes land in CPython 2.x or 3.y is a question for the CPython community -- we have to play by their rules on this. Q: Why branch CPython? Why not use Jython, IronPython or PyPy?A: We looked at that. We wanted to offer Google Python applications better performance, and we obviously don't want to do more work than we have to. However, Google has a massive body of C++ code exposed to Python applications via SWIG, none of which would work with any of the other Python implementations. Google has many of these systems implemented in Java, too, but the people on the hook for these applications were uncomfortable with that much change in their code base; porting to Jython and Java infrastructure would not have been a small change, and the app owners don't want to be paged at 3AM if they can help it. Ultimately, the reason for working off of CPython is compatibility: compatibility with pure-Python code, and also compatibility with the vast array of C extension modules in production. Q: What is your relationship with Google?A: Unladen Swallow is Google-sponsored, but not Google-owned. The engineers on the project are full-time Google engineers, but ultimately this an open-source project, not really that different from Chrome or Google Web Toolkit. We are pushing patches back into mainline CPython as quickly as we can. These patches are all code-reviewed by the CPython community, and we cannot merge the patches without their consent. Q: Do I need to sign a Contributor License Agreement?A: Yes. You should sign both Google's CLA and the Python Software Foundation's CLA, since all the code that goes into Unladen Swallow is intended to end up back in CPython. At the very minimum, though, you would need to sign Google's CLA (linked above) and just be aware that we will probably ship your code in our patches back to the PSF-licensed CPython. If you are a Google employee, your contributions are automatically covered by Google's blanket CLA with the PSF. |
Sign in to add a comment
Since you are working with CPython, it would be very interesting to hear your thoughts about the relevance or non-relevance of Cython ( http://www.cython.org derived from Pyrex).
Cython would let you dramatically speed up libraries by allowing compilation and unboxing of types (early binding of types) for speed on an incremental basis where you just touch the most performance-critical loops.
It sounds like you are pragmatic and focused on webapp performance and this is a very quick and easy way to provide that for "hot" functions.
See http://docs.cython.org/docs/early_binding_for_speed.html and http://wiki.cython.org/EarlyBindingForSpeed for pages in docs and wiki relevant to speed.
It is a very cheap way to get manifold speedups for hot spots in my experience.
I'm confussed... you said your goal is to make it at least 5x faster, but that seems very little compared to other projects. For example, psyco can speed up algorithmic code up to 100x, v8 is up to 56x faster than common javascript and so on... Is it perhaps a mistake? Sometimes talking about percentages of times lead to confussion. What exactly are your performance expectations compared to compiled languages such as java or c++, for example?
Q: I tried to check out the tests directory on my Mac but it failed. Why?
A: The version of Subversion in /usr/bin/svn on Mac OS X 10.5.x is only 1.4.4. Checking out the tests requires Subversion 1.5 or later.
Another question: Seeing the recent success of Google's V8, the speedy javascript VM, why don't you just base your work on it? It's a Google project, it's succesfull and it's very fast. Why reinventing the wheel?
luismgz: translating Python to Javascript would be easy to implement for about 80% of the language, but you'd hit a wall in implementing that last 20%. Just ask the Jython, PyPy? and IronPython? teams how hard 100% compatibility is. They've done some really heroic work to implement every dark and musty corner of the language, and I think they'd be the first to tell you that it's easy to get something like the Fibonacci function working, but things like metaclasses are a different story. We hope to side-step that by reusing as much of CPython as possible.
Psyco's claimed benefits of 100x speed-up on algorithmic code is rarely seen in real applications. It can certainly be used to optimize hotspots that fit Psyco's profile, but in examining the performance of some Google applications that use Psyco, we found that they see only a ~10% improvement in overall CPU usage. While that might be a valuable savings for some applications, it's not 100x, nor even the 2-4x low-end estimate that I've seen in the Psyco docs.
Are our performance goals too modest? We don't think so. Our team is small -- only two full-time engineers -- and we want to allow for unexpected surprises along the way. We feel that 5x is a good goal for the time being, especially given that we may need to make changes to LLVM along the way. If things go astoundingly well, we may raise those numbers, but for now, we're comfortable with our stated goals.
I do wonder if the merits of Parrot have been fully evaluated. As far as VMs go LLVM is of course a more mature project, and definitely gets a lot more exposure, but I would argue that Parrot has just reached a place where it is mature enough for this sort of project, and it is far better suited to the stated goals for Unladen-Swallow, because it is a high-level register-based VM specifically built for optimizing dynamic languages.
I firmly believe that a serious, marquee-name-backed Python port to Parrot's VM would greatly accelerate and benefit both projects.
I need not explain to Google what Parrot is and isn't, but I would like to say that if it was dismissed early on because it's new, or because of "Perl6 Stigma" that's just unfortunate for both Parrot and Unladen Swallow - akin to Sir Galahad's fate at the Bridge of Death!
Lefsha: The purpose of the project is 100% current python compatibility, AND a performance increase. If you want to use a statically typed language, then by all means go do so. If you like Boo, use Boo.
This project wants to enable significant performance increases for existing code, and to get merged into the existing CPython codebase.
Please ignore uneducated comments like those from lefsha. The wrong-headed concepts in that post are too numerous to address individually. Python's dynamic typing is fundamental to what it is. Improving performance isn't "stupid". Anyone who understands the nature of the GIL problem welcomes the effort being put forth here, as it is precisely the thing that will allow a "better computer" to exhibit improved performance.
An overall speedup factor 5 is awesome and would be very welcome indeed. It seems that Psyco will not be ported to 64-bit, so a speedup in the main interpreter is required to compensate for this. I hope you can address Windows CPython users as well – a large code base means that we’ll have to stay with CPython for a very long while.
Thank you for your efforts, and good luck! Martin
A very interesting talk on the "true horrors" of the GIL: http://blip.tv/file/2232410
Is it possible to use the python shared library to embed it into other programs? When I try this, I get: configure:16724: gcc -o conftest -g -O2 -D_THREAD_SAFE -pthread -I/usr/local/include/python2.6 conftest.c -L/usr/local/lib/python2.6 -L/usr/local/lib -lpython2.6 >&5 /usr/local/lib/libpython2.6.so: undefined reference to `llvm::ExecutionEngine?::addGlobalMapping(llvm::GlobalValue? const, void)' /usr/local/lib/libpython2.6.so: undefined reference to `llvm::Module::getGlobalVariable(llvm::StringRef? const&, bool) const' /usr/local/lib/libpython2.6.so: undefined reference to `llvm::ConstantArray?::get(llvm::LLVMContext&, llvm::StringRef? const&, bool)' /usr/local/lib/libpython2.6.so: undefined reference to `llvm::ModuleProvider?::ModuleProvider?()' /usr/local/lib/libpython2.6.so: undefined reference to `llvm::Module::addTypeName(llvm::StringRef? const&, llvm::Type const)' /usr/local/lib/libpython2.6.so: undefined reference to `llvm::TruncInst?::TruncInst?(llvm::Value, llvm::Type const, llvm::Twine const&, llvm::BasicBlock?)' /usr/local/lib/libpython2.6.so: undefined reference to `llvm::createLICMPass()' /usr/local/lib/libpython2.6.so: undefined reference to `llvm::ConstantInt?::get(llvm::Type const, unsigned long, bool)' /usr/local/lib/libpython2.6.so: undefined reference to `llvm::DerivedType?::refineAbstractTypeTo(llvm::Type const)' /usr/local/lib/libpython2.6.so: undefined reference to `llvm::AttrListPtr?::operator=(llvm::AttrListPtr? const&)' /usr/local/lib/libpython2.6.so: undefined reference to `llvm::ConstantExpr?::getSDiv(llvm::Constant, llvm::Constant)' /usr/local/lib/libpython2.6.so: undefined reference to `llvm::StringMapImpl?::RehashTable?()' /usr/local/lib/libpython2.6.so: undefined reference to `llvm::getGlobalContext()' /usr/local/lib/libpython2.6.so: undefined reference to `llvm::StructLayout?::getElementContainingOffset(unsigned long) const' /usr/local/lib/libpython2.6.so: undefined reference to `llvm::CallInst?::CallInst?(llvm::Value, llvm::Value, llvm::Twine const&, llvm::Instruction)' /usr/local/lib/libpython2.6.so: undefined reference to `llvm::Argument::Argument(llvm::Type const, llvm::Twine const&, llvm::Function)' /usr/local/lib/libpython2.6.so: undefined reference to `llvm::GlobalVariable?::GlobalVariable?(llvm::Module&, llvm::Type const, bool, llvm::GlobalValue?::LinkageTypes?, llvm::Constant, llvm::Twine const&, llvm::GlobalVariable?, bool, unsigned int)' ...?
What library should I add? Or isn't it so simple?