I do, however, have a question about threads, and how (if you do) handle them? For instance, I have a python application that I would like to create multiple times for multiple C++ class instances. Looking around, it seems using multiple embedded interpreters is not the way to go. neither is boost::python (or really python for that matter) thread safe as far as I can tell. So, I am resorting to using a global thread lock for the embedded python interpreter. I do not predict this will really be much of a bottle neck, as at most, there will only be a few real threads operating on the python object at a time (so, not too much locking). Probably, at most 3 possible concurrent threads - handling less than a hundred separate python class instances.
You can create multiple interpretors for each thread, you can also create threads internally in python and let it do all the setup for you. Python uses a global interpretor lock, so only one interpretor is going through python bytecode at a time, hence if you wish to use threads then make sure to release the lock when entering your code (a simple macro exposed by the python header) and relock it (another simple macro) when exiting back to python code, this will make sure the python interpretor can run while inside your own code. I did this for a bit at first, and it is doable, but my later designs forgo python across threads (instead preferring physics, pathfinding, etc...) and I use StacklessPython now, so I use micro-threads inside python to simulate full threads (but far far faster), though not truly concurrent, and more difficult to setup (not too much, just need about two wrapper functions).
PyPy will be far better for threads as it will have no global interpretor lock later (the threading model in it is not complete yet, so it still uses CPython's design), and it is version 1.0 (meaning it passes all python's core tests), but it is still rather slow right now (about 10 times slower then cpython at worst, about 20 times faster at best, usually worst though as the only main speedups are in heavy math code, averages about 2 to 4 times slower in my tests). As soon as PyPy is optimised, then I will be dropping the CPython and stackless implementations completely and only for it.
When PyPy is complete, I'm sure you would love it as well. It is an implementation of Python written in Python (python running in python) as they short describe it, but in reality it is vastly different. It is actually a multi-level compiler. The top level can take any code that has a layer written for it (Python is the main one right now, obviously, but others are being written, a C++ layer, etc... etc... Python will no doubt be the purpse of it though for the language layer) and compiles it down to a set of RPython (short for Restricted Python). RPython is python, but with a more limited command set and some minor typing (faster then normal python, but you cannot make programs for it as it is essentially a program form of an AST, but far far more powerful). RPython is powerful enough that any language in existance (there is also a Prolog layer to it being made) can have a layer to compile to it, whether they define functions at runtime to any manner of oddities you can think of. RPython is then sent to the bottom layer, of which there are multiple's of. The bottom layer right now has a Python interpretor layer (written in Python), a C++ layer (to export RPython to C++ to compile in your favorate compiler, which it does automagically, visual studio or mingw on windows, gcc on *nix, etc...) with all the optimizations they are capable of (RPython will also heavilly optimize the code as well), a javascript layer (yes, they even have examples of a webserver on their demo site, showing pypy running as a webserver, compiling the code in real-time to javascript which the page then loads, it is pretty amazing, and rather complete, I may never touch javascript itself again when I need to write it once I learn this), to directly to machine code, to .NET (the .NET layer is pretty complete, but due to .NET lack of capabilities, some of the high-end features cannot be optimized and run rather slow, like stack fiddling, but only if you use those features, and they are not too slow, just slower then any other language bytecode), and a partially made Java bytecode creator (also has many of the issues as the .NET code).
Basically, PyPy is being made so you can code in Python (or other languages once the top layer has more out) and can save it out as (the things currently made are:) .NET bytecode (which can call and be called by other .NET code), Java Bytecode (ditto as with .NET), Javascript (which was suprisingly well done, if not verbose, designed with AJAX in mind in the current incarnation), LLVM (Low Level Virtual Machine, a popular 'interpretor' for processor independent x86-like machine code), C++, C, and machine code (x86 currently) (I think there is one more for Lisp as well or some-such that has been started). Features it has (if the top language supports it, which a C++ layer, the Python layer, and a couple others, would have no issue with) are things like stackless python (real stackless code, can have infinite recursion exceeding the stack limits of the system if you wish, micro-threads as in stackless python, and many other things that stackless python cannot support, etc...), thunk object space (lazy evaluation in a fully transparent manner, objects morph to the full value once anything requires the value for the first time), logic (real logic programming, proper binding of values and all), Aspect Oriented Programming (supports AOP natively, not a wierd wrapper like C++ ones are, or languages like .NET and Java cannot support it at all, the best example of AOP is LISP's macro language, if you know it, you've known AOP all along), taint object space (you can mark anything as 'taint'ed, thus any non-accepted code is unable to access, query, or do anything with it), and so on.
Taint may require an example, so from the docs:
>>>> from __pypy__ import taint
>>>> x = taint(6)
# x is hidden from now on. We can pass it around and
# even operate on it, but not inspect it. Taintness
# is propagated to operation results.
>>>> x
TaintError
>>>> if x > 5: y = 2 # see below
TaintError
>>>> y = x + 5 # ok
>>>> lst = [x, y]
>>>> z = lst.pop()
>>>> t = type(z) # type() works too, tainted answer
>>>> t
TaintError
>>>> u = t is int # even 'is' works
>>>> u
TaintError
Notice that using a tainted boolean like x > 5 in an if statement is forbidden. This is because knowing which path is followed would give away a hint about x; in the example above, if the statement if x > 5: y = 2 was allowed to run, we would know something about the value of x by looking at the (untainted) value in the variable y.
It continues on telling about error conditions and so on and so forth, for example if you did "x/0" above, it will still return a taint'ed object, but as soon as it is untainted by trusted code, 'that' code gets the divide by zero exception, so something cannot test if something is an integer by trying a divide by zero or anything else like that and so forth.
As stated though, it is status 1.0 because the toolchain is 'complete' as well as many aspects, but the plugins for the chain (like better threading, it still uses python's method if you have threading enabled) are not all yet complete and are being worked on now. And not all the languages are usable either, the most working one is Python, but C++ is well supported at the moment though as well; for saving code out, C is the main output path, the .NET and Java bytecode is working well enough for most things, and the javascript browser AJAX is working well.
One other nice new feature is that all the RPython commands are available inside your app as well, and if you use RPython translator (the part that converts a language to RPython) then it will embed the translator in your app, so you can have the user's of your app (for example) write some code in something like LUA or Python or C# or what-not (any language the translator supports in your build of the translator) and compile the code inside your app as it runs. The translated RPython is returned as a Translator class (which you can play with directly in the PyPy shell if you wish) which you can call the normal operations on, such as you can either run it directly (in a variety of ways, basically as an interpretor, debug into it, etc... etc...), save it out, or compile it (either compiling into a source file, such as so you can save the C version or what-not, or compile it into operational code. An example of compiling a function to fully compiled (not source) C code, you would do something like this:
>>> def myfunc(a, b): return a+b
...
>>> t = Translation(myfunc)
>>> t.annotate([int, int])
>>> f = t.compile_c()
>>> f(4, 5)
9
>>> f(-2,3)
1"f" is the compiled function myfunc and is running at full native speed, not interpreted. First the function pointer is given to the translator, which returns the Translation class. You then mark the arguments (two integers in this case, could be any native value, or classes, or function pointers, etc...). Then it is compiled, which returns a pointer to the fully compiled function, which is then executed a couple times.
So, for now, with Python, use the global interpretor lock when you enter C code that will be sizable (at least 100 or so opcodes, so quite small, or anything that can block, or calls other functions, etc...). The best case is to make it so that any threads that python will need to exist in, are created by python itself, don't create other interpretors as it forces you to create a ton of work to handle it all, whereas just eval'ing a short python string from your C side to setup all your threads that python will need to exist in is far better.
PyPy in the future will be incredably nice, and you may want to watch it, but it is not fast enough to use yet, so don't bother. (It is actually growing amazingly fast, it has corporate backing behind it so it has a money-flow to create it, it will always be free so don't worry about that, just some companies have a vested interest to get it working so they can use it; as stated, keep an eye on it.)
Er.. dreadfully sorry about starting to ramble, just been reading up on PyPy recently and it is on my mind, only reached version 1.0 a few weeks ago (March 27, 2007).
