DVgrab doesn't automatically pluck off DV from a DV video cam to Mpeg. This means that to save a modicum of space I have to convert the DV files into mpegs. This isn't really a problem with the most excellent video tool transcode, but when you've got a dozen or so 2-3 minute blurbs it can be a pain since each takes several minutes to convert running on my primary workstation (a 2.4ghz PIV, .5 GB ram). To make it simple I wrote a shell script that would convert every DV file in a generic mpeg (note that the call avi2mpg is yet another shell script I wrote.
Since I have two other boxes on the LAN each with two processors, and I recently got ssh to do a login without a password, I got the idea to set up a poorman's cluster and run each file in parallel on a different box.
here is the shell script
The basic algorithm is:
1) set up a series of files that give the number of files being processed on each node
2) get the a filename to process
3) check that number of current process (listed in the files from part 1) don't exceed some specified total number of processes (i.e. the number of processes on each node) for a particular node.
4) if a given node's 'load' hasn't been exceeded, increment the number of processes, then start a thread to process it on that node
5) go to 3, repeat until there are no more files.
NOTE: the thread that gets started in 4 is here, and when the avi2mpg script compeletes it decriments the number of processes for that node.
The program sort of works.
For the box with a single processor the number of processes incriments/decriments perfectly.
For the boxes with dual processors, the number of processes increments well up to 2, and will decriment down to 1, but for some reason won't decriment that final number when the second process finishes...
I have a sneaking suspicion that it has something to do with the way the shell script variables are held in memory when said script is running twice on the same box...
I've been thinking of rewriting in Perl, but as I've only read an O'Reily book on Perl, and am convinced it has can handle multithreading far better than the bash scripting, but as I've never done any serious scriping in perl, I'm a little intimidated.
It should also be noted that I've never taken other than a basic course in C-programming, never taken a class in how to impliment multi-threading apps...
any comments?
Since I have two other boxes on the LAN each with two processors, and I recently got ssh to do a login without a password, I got the idea to set up a poorman's cluster and run each file in parallel on a different box.
here is the shell script
The basic algorithm is:
1) set up a series of files that give the number of files being processed on each node
2) get the a filename to process
3) check that number of current process (listed in the files from part 1) don't exceed some specified total number of processes (i.e. the number of processes on each node) for a particular node.
4) if a given node's 'load' hasn't been exceeded, increment the number of processes, then start a thread to process it on that node
5) go to 3, repeat until there are no more files.
NOTE: the thread that gets started in 4 is here, and when the avi2mpg script compeletes it decriments the number of processes for that node.
The program sort of works.
For the box with a single processor the number of processes incriments/decriments perfectly.
For the boxes with dual processors, the number of processes increments well up to 2, and will decriment down to 1, but for some reason won't decriment that final number when the second process finishes...
I have a sneaking suspicion that it has something to do with the way the shell script variables are held in memory when said script is running twice on the same box...
I've been thinking of rewriting in Perl, but as I've only read an O'Reily book on Perl, and am convinced it has can handle multithreading far better than the bash scripting, but as I've never done any serious scriping in perl, I'm a little intimidated.
It should also be noted that I've never taken other than a basic course in C-programming, never taken a class in how to impliment multi-threading apps...
any comments?
