Python multiprocessing.Queue is using internally shelve module, which again internally is using cPickle for speed up. But if something cannot be pickled, when calling Queue.put, you will see some cryptic error message:
File "/opt/dirac/pro/Linux_x86_64_glibc-2.5/lib/python2.6/multiprocessing/queues.py", line 242, in _feed
- send(obj)
TypeError: expected string or Unicode object, NoneType found
Ugh! Not helping at all, but what to do except start debugging? Taking the advantage of the opportunity I had fixed also some problems here and there (un-shelvable locks or instance methods in several classes), but at the very, very end I've got this:
pickle.PicklingError: Can't pickle
Huh? What the heck? Where is this one? Find and grep fellows stayed completely muted, so I've started to dump symbols from *.so under PYTHONPATH, and guess what? I HAVE FOUND IT!
[volhcb13] /opt/dirac/pro > nm Linux_x86_64_glibc-2.5/lib/python2.6/lib-dynload/_elementtree.so | grep copyelement
0000000000209dd8 b elementtree_copyelement_obj
This little bastard was hiding in cElementTree! I've swapped this one with its python twin (ElementTree). Taa-daam! Queue.put is working again, not a big deal for me, XML fragments I'm dealing with are rather small.
5 hours! Five! FIVE HOURS lost...
So, dear children, please remember: when going multiprocessing forget about cElementTree (and perhaps several other modules rewritten in C for speed up) and use pure python implementations.
