grep.ro

I got the parallelization library to the point where it can do most of what I want. It can launch multiple threads, feed them with data, pull data from them, buffer results and pickle messages to temporary files. Most importantly, all of the library's interface is a single decorator with clean, obvious syntax.

@async('input1', 'input2', workers=10, buffer=15, temptile_output=True)

input1 and input2 are named parameters of the decorated function that are going to be piped to the worker threads. workers=10 means that the library will launch 10 child processes. buffer=15 means that results will be pulled from the workers before they are requested downstream (this is especially useful if the results are pulled by a single worker thread). tempfile_output=True tells the library to pass results via temporary files because sending large amounts of data through pipes is very slow on some platforms.

There's a few caveats, like the fact that workers exist in separate processes and don't have access to the parent's globals – nothing too dangerous, but I'll have to document them anyway. The next step is to use the library to parallelize the Wide Finder program. Now that there's proper buffering going on, total running time should be a little over 15 minutes.

Given enough time, I want to add some profiling capabilities to the library: it would log all message-passing events with worker processes, which can then be drawn as a timeline, to visualize how workers start and stop, when they request data and when the output is requested, the size of buffers, etc.