pathos.helpers module documentation¶
mp_helper module¶
map helper functions
- random_seed(s=None)¶
sets the seed for calls to ‘random()’
- random_state(module='random', new=False, seed='!')¶
return a (optionally manually seeded) random generator
For a given module, return an object that has random number generation (RNG) methods available. If new=False, use the global copy of the RNG object. If seed=’!’, do not reseed the RNG (using seed=None ‘removes’ any seeding). If seed=’*’, use a seed that depends on the process id (PID); this is useful for building RNGs that are different across multiple threads or processes.
- starargs(f)¶
decorator to convert a many-arg function to a single-arg function
pp_helper module¶
- class ApplyResult(task)¶
Bases:
ppft._pp._Task
result object for an ‘apply’ method in parallelpython
enables a pp._Task to mimic the multiprocessing.pool.ApplyResult interface
Initializes the task
- __call__(raw_result=False)¶
Retrieves result of the task
- __init__(task)¶
Initializes the task
- __module__ = 'pathos.helpers.pp_helper'¶
- __unpickle()¶
Unpickles the result of the task
- property callback¶
- property callbackargs¶
- finalize(sresult)¶
Finalizes the task *internal use only*
- property finished¶
- get(timeout=None)¶
Retrieves result of the task
- property group¶
- property lock¶
- ready()¶
Checks if the result is ready
- property server¶
- successful()¶
Measures whether result is ready and loaded w/o printing
- property tid¶
- wait(timeout=None)¶
Waits for the task
- class MapResult(size, callback=None, callbackargs=(), group='default')¶
Bases:
object
- __call__()¶
Retrieve the results of the tasks
- __dict__ = mappingproxy({'__module__': 'pathos.helpers.pp_helper', '__init__': <function MapResult.__init__>, 'finalize': <function MapResult.finalize>, '_MapResult__unpickle': <function MapResult.__unpickle>, 'queue': <function MapResult.queue>, '__call__': <function MapResult.__call__>, 'wait': <function MapResult.wait>, 'get': <function MapResult.get>, 'ready': <function MapResult.ready>, 'successful': <function MapResult.successful>, '_set': <function MapResult._set>, '__dict__': <attribute '__dict__' of 'MapResult' objects>, '__weakref__': <attribute '__weakref__' of 'MapResult' objects>, '__doc__': None, '__annotations__': {}})¶
- __init__(size, callback=None, callbackargs=(), group='default')¶
- __module__ = 'pathos.helpers.pp_helper'¶
- __unpickle()¶
Unpickles the results of the tasks
- __weakref__¶
list of weak references to the object (if defined)
- _set(i, task)¶
- finalize(*results)¶
finalize the tasks *internal use only*
- get(timeout=None)¶
Retrieves results of the tasks
- queue(*tasks)¶
Fill the MapResult with ApplyResult objects
- ready()¶
Checks if the result is ready
- successful()¶
Measures whether result is ready and loaded w/o printing
- wait(timeout=None)¶
Wait for the tasks
- class Server(ncpus='autodetect', ppservers=(), secret=None, restart=False, proto=2, socket_timeout=3600)¶
Bases:
object
Parallel Python SMP execution server class
Creates Server instance
ncpus - the number of worker processes to start on the local computer, if parameter is omitted it will be set to the number of processors in the system ppservers - list of active parallel python execution servers to connect with secret - passphrase for network connections, if omitted a default passphrase will be used. It’s highly recommended to use a custom passphrase for all network connections. restart - restart the worker process after each task completion proto - protocol number for pickle module socket_timeout - socket timeout in seconds, which is the maximum time a remote job could be executed. Increase this value if you have long running jobs or decrease if connectivity to remote ppservers is often lost.
With ncpus = 1 all tasks are executed consequently. For the best performance either use the default “autodetect” value or set ncpus to the total number of processors in the system.
- __add_to_active_tasks(num)¶
Updates the number of active tasks
- __connect()¶
Connects to all remote ppservers
- __del__()¶
- __detect_ncpus()¶
Detects the number of effective CPUs in the system
- __dict__ = mappingproxy({'__module__': 'ppft._pp', '__doc__': 'Parallel Python SMP execution server class\n ', 'default_port': 60000, 'default_secret': 'epo20pdosl;dksldkmm', '__init__': <function Server.__init__>, 'submit': <function Server.submit>, 'wait': <function Server.wait>, 'get_ncpus': <function Server.get_ncpus>, 'set_ncpus': <function Server.set_ncpus>, 'get_active_nodes': <function Server.get_active_nodes>, 'get_stats': <function Server.get_stats>, 'print_stats': <function Server.print_stats>, 'insert': <function Server.insert>, 'connect1': <function Server.connect1>, '_Server__connect': <function Server.__connect>, '_Server__detect_ncpus': <function Server.__detect_ncpus>, '_Server__dumpsfunc': <function Server.__dumpsfunc>, '_Server__find_modules': <function Server.__find_modules>, '_Server__scheduler': <function Server.__scheduler>, '_Server__get_source': <function Server.__get_source>, '_run_local': <function Server._run_local>, '_run_remote': <function Server._run_remote>, '_Server__add_to_active_tasks': <function Server.__add_to_active_tasks>, '_Server__stat_add_time': <function Server.__stat_add_time>, '_Server__stat_add_job': <function Server.__stat_add_job>, '_Server__update_active_rworkers': <function Server.__update_active_rworkers>, '_Server__gentid': <function Server.__gentid>, '__del__': <function Server.__del__>, 'destroy': <function Server.destroy>, '__dict__': <attribute '__dict__' of 'Server' objects>, '__weakref__': <attribute '__weakref__' of 'Server' objects>, '__annotations__': {}})¶
- __dumpsfunc(funcs, modules)¶
Serializes functions and modules
- __find_modules(prefix, dict)¶
recursively finds all the modules in dict
- __gentid()¶
Generates a unique job ID number
- __get_source(func)¶
Fetches source of the function
- __init__(ncpus='autodetect', ppservers=(), secret=None, restart=False, proto=2, socket_timeout=3600)¶
Creates Server instance
ncpus - the number of worker processes to start on the local computer, if parameter is omitted it will be set to the number of processors in the system ppservers - list of active parallel python execution servers to connect with secret - passphrase for network connections, if omitted a default passphrase will be used. It’s highly recommended to use a custom passphrase for all network connections. restart - restart the worker process after each task completion proto - protocol number for pickle module socket_timeout - socket timeout in seconds, which is the maximum time a remote job could be executed. Increase this value if you have long running jobs or decrease if connectivity to remote ppservers is often lost.
With ncpus = 1 all tasks are executed consequently. For the best performance either use the default “autodetect” value or set ncpus to the total number of processors in the system.
- __module__ = 'ppft._pp'¶
- __scheduler()¶
Schedules jobs for execution
- __stat_add_job(node)¶
Increments job count on the node
- __stat_add_time(node, time_add)¶
Updates total runtime on the node
- __update_active_rworkers(id, count)¶
Updates list of active rworkers
- __weakref__¶
list of weak references to the object (if defined)
- _run_local(job, sfunc, sargs, worker)¶
Runs a job locally
- _run_remote(job, sfunc, sargs, rworker)¶
Runs a job remotelly
- connect1(host, port, persistent=True)¶
Conects to a remote ppserver specified by host and port
- default_port = 60000¶
- default_secret = 'epo20pdosl;dksldkmm'¶
- destroy()¶
Kills ppworkers and closes open files
- get_active_nodes()¶
Returns active nodes as a dictionary [keys - nodes, values - ncpus]
- get_ncpus()¶
Returns the number of local worker processes (ppworkers)
- get_stats()¶
Returns job execution statistics as a dictionary
- insert(sfunc, sargs, task=None)¶
Inserts function into the execution queue. It’s intended for internal use only (in ppserver).
- print_stats()¶
Prints job execution statistics. Useful for benchmarking on clusters
- set_ncpus(ncpus='autodetect')¶
Sets the number of local worker processes (ppworkers)
- ncpus - the number of worker processes, if parammeter is omitted
it will be set to the number of processors in the system
- submit(func, args=(), depfuncs=(), modules=(), callback=None, callbackargs=(), group='default', globals=None)¶
Submits function to the execution queue
func - function to be executed args - tuple with arguments of the ‘func’ depfuncs - tuple with functions which might be called from ‘func’ modules - tuple with module names to import callback - callback function which will be called with argument list equal to callbackargs+(result,) as soon as calculation is done callbackargs - additional arguments for callback function group - job group, is used when wait(group) is called to wait for jobs in a given group to finish globals - dict from which all modules, functions, and classes will be imported, for instance: globals=globals()
- wait(group=None)¶
Waits for all jobs in a given group to finish. If group is omitted waits for all jobs to finish
- exception TimeoutError¶
Bases:
multiprocess.context.ProcessError
- __module__ = 'multiprocess.context'¶
- _ApplyResult¶
alias of
multiprocess.pool.ApplyResult
- _MapResult¶
alias of
multiprocess.pool.MapResult
- class _Task(server, tid, callback=None, callbackargs=(), group='default')¶
Bases:
object
Class describing single task (job)
Initializes the task
- _Task__unpickle()¶
Unpickles the result of the task
- __annotations__ = {}¶
- __call__(raw_result=False)¶
Retrieves result of the task
- __dict__ = mappingproxy({'__module__': 'ppft._pp', '__doc__': 'Class describing single task (job)\n ', '__init__': <function _Task.__init__>, 'finalize': <function _Task.finalize>, '__call__': <function _Task.__call__>, 'wait': <function _Task.wait>, '_Task__unpickle': <function _Task.__unpickle>, '__dict__': <attribute '__dict__' of '_Task' objects>, '__weakref__': <attribute '__weakref__' of '_Task' objects>, '__annotations__': {}})¶
- __init__(server, tid, callback=None, callbackargs=(), group='default')¶
Initializes the task
- __module__ = 'ppft._pp'¶
- __weakref__¶
list of weak references to the object (if defined)
- finalize(sresult)¶
Finalizes the task.
For internal use only
- wait()¶
Waits for the task