Python/ThreadFunc provides tmap, treduce, tfilter functions that are compatible with built-in functions map, reduce and filter, respectively. Provided functions process data in multi-thread internally, they become much faster if they requries heavy I/O.

This module also provides function "parallel" which recieve functions to process them in parallel easily.

Example

def test_tmap(numThreads):
  def double(x):
    time.sleep(random.random()) #simulation of something time-consuming procedure
    return x*2
  return tmap(double,range(1,32),numThreads)

### testing tmap() ###
result: 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62
1 thread(s), 13.9450819492 sec

result: 2 4 6 8 12 10 14 16 18 20 24 22 26 28 30 34 36 32 38 42 40 44 46 50 52 48 54 56 58 62 60
2 thread(s), 8.19523096085 sec

result: 2 8 10 4 6 12 16 14 18 24 22 20 26 32 28 38 30 36 40 42 34 48 44 46 56 50 52 62 54 60 58
4 thread(s), 4.12061500549 sec

result: 8 12 32 18 4 10 30 26 22 6 36 24 34 48 40 58 2 20 28 14 54 16 46 52 44 38 42 56 60 50 62
16 thread(s), 1.59210181236 sec

result: 16 12 54 2 4 28 60 22 24 36 38 50 48 34 8 14 20 52 32 58 10 40 6 44 62 56 26 18 30 46 42
32 thread(s), 0.983188152313 sec

result: 20 56 6 46 32 60 4 42 62 38 28 26 30 36 8 12 40 2 10 34 22 58 52 18 48 44 54 16 50 24 14
64 thread(s), 0.977565050125 sec

result: 34 28 38 54 36 24 40 30 14 56 48 4 58 12 2 52 16 10 22 26 6 18 44 46 50 42 60 8 20 62 32
128 thread(s), 1.06019997597 sec

Download

threadfunc.py