JobQueue - Let run your jobs in parallel¶ ↑
This package contains jobQueue:
* A simple class to parallalize your work on a user definend number of threads * A ruby script, which uses the above class to run each line of a shell script
jobQueue can do the following things:
-
Run blocks, Procs and Lambdas
-
Run instance and class methods
-
Run shell commands
-
Respect user definded locks
I started a python2 implementation of this, which can be installed via pip (pypi.python.org/pypi/jobqueue). When its maature enough, I will use the ruby version numbers for it.
Installation¶ ↑
Gem Installation¶ ↑
Download and install jobQueue with the following.
gem install jobQueue
Requirements¶ ↑
JobQueue requires Ruby only, but versions 1.9.x are needed to make use of system threads.
Usage¶ ↑
Parallelize Ruby’s blocks, procs, lambdas and things¶ ↑
Create a JobQueue with nThreads worker with:
jq = JobQueue.new(nThreads)
Use its push method to put in something to do
-
For blocks:
jq.push do myObject.method0(...) myObject.method1(...) myObject.method3(...) end
-
For procs and lambdas:
jp.push(myProc,arg0,arg1,...)
-
For object methods:
jq.push([myObject,[:method,arg0,arg1,...])
-
Same code can be used for class methods:
jq.push(myClass,[:myClassMethod,arg0,arg1,...])
To start the workers, call
jq.run
That’s it. You might have look at tests, that come with the jobQueue gem.
Parallelize system commands¶ ↑
Use a separate class for this and push string to it:
jq = SystemJobs.new(nThreads) jq.push('find ./src -name "*.rb"','find ./downloads -name "*.rb"')
will start 2 parallel searches for ruby files. This is implemented in the prun.rb script, which comes with the jobQueue gem. To run each line of a shell script on 10 threads, use:
prun.rb -j 10 jobs.sh
The ‘-j’ switch is optional. Default is the maximum number of cores. The script accepts multiple files and processes one after another. Try ‘-h’ for documentation.
Support, Issues, Bugs, …¶ ↑
please use personal mail, ruby-lang mailing list or github
Changelog¶ ↑
-
1.0.11: prun.rb now ignores empty lines
-
1.0.10: more flexible logging control (new switches ‘-l’ and ‘-b’)
-
1.0.9: print out stdout and stderr from the jobs given to prun.rb, use ‘-D’ to avoid printing
-
1.0.8: support AIX for getting the maximum number of processors, improve processor count for jruby and rbx
Credits¶ ↑
- Robert Klemme
-
For the first hints: www.ruby-forum.com/topic/68001#86298
License¶ ↑
jobQueue use the BSD License
:include LICENSE
Other stuff¶ ↑
- Author
-
Ralf Mueller <stark.dreamdetective@gmail.com>
- Requires
-
Ruby 1.9 or later
- License
-
Copyright 2011-2016 by Ralf Mueller Released under BSD-style license. See the LICENSE file included in the distribution.