Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[mysterycoder]

Ah ok, so I suppose something with soft threads will be fine. 
I think I'm going to give stackless a shot. If it doesn't work, it'll be easy to fall back to normal python as the binding and everything is the same.

No problems with that then?   

[OvermindDL1]

With Stackless Python you would send a message to a tasklet (not a thread) and it would handle it, sending a message back whenever it is done; in such a system you pretend threads do not exist, tasklets clean it up (that is why my new language is kind of like erlang in power, but can actually be easily bound, and will be *fast*, something I am focusing on, it actually compiles, no interpreting).

[mysterycoder]

From what I understand, you can send messages while the tasklets are running. I considered them threads even though they aren't "real" threads because the stackless website classifies them as microthreads and they effectively run concurrently using a multitasking scheduler.

Btw do you know how Psyco would work in this instance? Would it pre-compile the modules before releasing the game? or would it compile the modules on the fly?

Edit: Ah I suppose I wasn't clear, I meant that we would use functions wrapped in tasklets to effectively become microthreads.

[OvermindDL1]

Psyco works fine with stackless (heck, one of the main creators of stackless made psyco).  And Psyco always works by compiling on the fly (as sometimes the interpreted can be faster then the compiled in rare circumstances, like heavy string manipulation, psyco is most useful on heavy math work, but is still useful everywhere, it automatically handles it all correctly by doing on-the-fly profiling like Java and such does).

[mysterycoder]

I've got all the binding and initialisation working with standard python and WGE can run scripts. It auto imports the WGE module and the various singletons are already available as objects.   (btw I'm not that fast, I just had a bunch of this written before).

As far as I know, all I have to do to move to stackless python is to recompile boost python against stackless.

The script does not need to do anything with threads imho, but the scripting engine should allow multiple scripts to be run concurrently by the game engine.

I've been looking at how to run scripts concurrently using python. The standard approach I think is to create multiple interpreters using the Python C API. However, I'm a little worried about boost: http://www.boost.org/doc/libs/1_36_0/libs/python/doc/v2/faq.html#threadsupport.

So I was thinking another way to do it would be to pass stackless tasklets to WGE and have it run them concurrently using the stackless scheduler. But this might kind of defeat the purpose of tasklets.

I see three possibilities:

• Use multiple interpretors and run multiple scripts at the engine level and then allow microthreads within the scripts.
• Run multiple scripts at engine level simulating multiple interpretors using stackless functionality.
• Allow one script to run at a time and use microthreads within it.
  

If I were to only let one script run at a time, I suppose a possibility would be to make it non-blocking by sticking all of the scripting in a separate thread.

[mysterycoder]

Huh. 
I was under the impression that there was a method for running multiple python scripts at once, but it appears to do that I need to create a thread from C++ for each script to run in.
Do you all know if that is correct? I've been searching through documentation and mailing lists, but the answer is not clear.

[pjcast]

Not sure. But it makes sense that each threaded python script execution would be started from its own C++ thread. Ie, if somehow the engine was threaded and updating a bunch of Objects at once, which was also calling their action scripts.

[OvermindDL1]

Even if you started up multiple interpreters and such for such things, it would still not 'scale' as you are thinking.  Python by itself is threadsafe, but it is designed for the scripts to be run in one and only one thread, but they can safely interact with other threads just fine.

[mysterycoder]

I know that it will only use only one processor, but I wanted to get the ability to run scripts "concurrently," even if it is not true concurrency.
Would you address this at the interpretor level? or would you try to emulate this functionality?

Or perhaps I'm approaching this from the wrong way.

[OvermindDL1]

Stackless does that all fine and native, nothing special you need to do, just treat it like a standard Actor system.

[mysterycoder]

Cool, I've got it building on linux and windows with stackless python 2.6. Unfortunately it requires a source build of boost::python that links to stackless.

I did it all in two months too, a speed demon I tell ya 

Edit:
Right now WGE uses ptypes for threading the audio. How do you feel about the move to boost::threads?
I think Ogre requires it for background loading anyway.

[pjcast]

Yeah, ptypes is pretty much a dead fish now. The only thing I don't like about boost::threads (I've used it for parts of OIS as well as other things) is the lack of Aborting a miss behaving thread... But, that situation is pretty rare. Oh, and a few missing threading primitives last I used it. I'd be more inclined to simply create some wrapping for a bit of Win32/Posix threads - perhaps using ptypes as the code base (I think it's license permits that).

[mysterycoder]

Do you remember which threading primitives were missing in particular?
From what I've read, boost::threads got updated recently in boost 1.36, with several missing features added.

Right now I'm also thinking about how python should interface. It's a bit strange thinking with stackless python in terms of what is non blocking and what is. Currently, the C++ side controls the main loop. I'm thinking there are a few different ways to approach it.

• (In C++ main loop)
  C++ calls the python code of game objects and python performs actions as per the scripts.
  Python object wants to interact with another object, it sends a message to C++, which receives and routes message to that the requested python object. This doesn't seem very stackless-like, a good straight python approach?
  
• (In C++ main loop)
  C++ calls the python code of game objects but the python code runs in its own thread(somehow) and python performs the actions as per the scripts.
  Python object wants to interact with another game object. It queries the object through the C++ side and establishes a channel with it. It uses that channel to send the message.
  
• (In python main loop) - Not a very good idea?
  Python handles the rendering loop, but most things are done C++ side, e.g. Game Object management, rendering.
  
• (In python and C++ main loops)
  The python and C++ main loops run in different threads. This makes it easy for stackless objects to always be running. Cleanest separation. C++ has to lock the GIL and send commands using the Stackless API. However game objects have direct access to one another. Synchronization issues? Performance issues? Pain in the butt?
  

Any better ideas than these ? (I'm sure there are)
Any foresight into some problems I may have?
Should I do some strange combination of the above methods? 

[mysterycoder]

I just read EVE Online's presentation on their Stackless Python usage. They have a very cool idea, they combine stackless python's main tasklet and the actual C++ message loop by emulating a python function  by having the main message loop function take PyObjects as parameters and returning a PyObject.

Code: [Select]

PyObject *myApp = new EveApp();
PyStackless_CallMethod_Main(myApp, “WinMain”, 0);

This seems like a great solution to me. There should be no thread or deadlock issues.

The only thing I wonder about is if there is a performance benefit from having the logic and engine running on separate threads. But I imagine the methods needed to communicate between python and wge, each on separate threads, would induce quite a bit of overhead. Also with the max amount of cores commonly available is about two, I think threading different parts of WGE would provide better performance benefits. Already the sound system is threaded. Once physics are added, I believe bullet should take advantage of quite a few threads. Ogre background resource loading too. Also, if anyone wants to implement networking on their game, another thread. So already it seems like enough is parallel to take advantage of today's processors.

[mysterycoder]

Better stackless support is on its way. 
By using this and the eve online presentation, I've got some rudimentary support in.

This is in the main function, this snippet binds an already defined C++ function and injects it into the main namespace, it then proceeds to use that function as stackless' main tasklet:

Code: [Select]

py::handle<> mainHandle(py::borrowed(PyImport_AddModule("__main__")));
py::object mainModule(mainHandle);
py::def("mainLoop", &mainLoop);

PyStackless_CallMethod_Main(mainModule.ptr(), "mainLoop", 0);
This is the function referenced in the snippet above:

Code: [Select]

void mainLoop()
{
     while(WGE::Manager::getSingleton().update())
     {
          //Only needed to update the debug stats
          WGE::OgreManager::getSingleton().updateStats();
     }
}

The main loop is very basic right now, but things like scheduling tasklets will be added in soon.
It doesn't do much, aside from acting as the old main loop did, but it's a start. 

Edit: Fixed a link
Edit2: Fixed formatting.
Thinking about it, the function should really be put into the wge namespace that gets created in python, will do later.