clogged pipe

What if? Say you want to execute something in a python sub-process and you want to read and resend all output data, no matter how big they are?

The simplest solution to use pipe is BAD idea - pipe has it's own buffer and what's worse size of this buffer is constant and cannot be changed by using some smart fnctl call any more. Let's run a simple test:



On my laptop I'm not able to send more than 64kB at a time, moreover its completely hanging:



Hmmm... So how can you send more than that? Let's say this way:



Use a simple manager, man! More info here.

There is also possibility  to use shared memory objects (read this), but those are rather for storing not so huge amount of data.

Disclaimer: Before use, read the contents of the package insert or consult your physician or pharmacist because each drug used improperly threatens your life or health.

python multiprocessing handy template with callbacks

Here is a handy implementation of process pool I wrote a few days ago. All workers are created in pool and are running in daemonic mode, reading and executing ProcessTasks, which is made up using any python callable together with callback and exception callback definition.

The process pool is using two queues: first for tasks to be processed and second holding task results for callbacks. It could be executed in daemon mode too, processing results in a separate thread. Once you decide the job is done, you need to call ProcessPool::finalize (or ProcessPool::processAllResults), which is sending dummy assassins (in fact just a True value) to the workers (so they could return from their run method), terminating them once they are not alive any more and closing tasks and results queues. Blah, blah, blah... Code that was published below was kinda buggy. Error-free version can be found here.