Monday, April 18, 2011

Parallel processing on bash shell

Straight to the idea:
suppose you are working on a machine which has some 16 cores. And you have a heavy program which generates particle events and outputs to a file. You can make the whole thing parallel process on the shell.

Here we go.
The program name toyMC
the program is run with an argumen n_events. The program reads this number and generates "n_evetns" number of MonteCarlo evetns and writes it into and output file evGen.txt

How do we parallelize this

Say we run 100 parallel instances of this program:

first creat 100 directories and then "cd" to these directories and run the code.

It is safe to do it this way.. otherwise multiple instances of the program might access the file and it will be a great mess.

The Full command on bash:

for i in `seq 1 1 100; do (mkdir job${i}; cd job{$i}; ../toMC 100 & ); done

thats it...

warning: the macro is not tested.

2 comments:

  1. Say you only want to run one job per CPU but still have 100 jobs. With GNU Parallel you can do:

    seq 100 | parallel mkdir job{}\; cd job{}\; ../toMC 100

    Watch the intro video to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ

    ReplyDelete
  2. hi Tange

    Thanks for the suggestion. GNU parallel is something that I wasn't aware of.

    ReplyDelete