cctools
|
Public Member Functions | |
def | __init__ (self, batch_type="local", manager=None, manager_host_port=None, manager_name=None, factory_binary=None, worker_binary=None, log_file=os.devnull) |
Create a factory for the given batch_type and manager name. More... | |
def | start (self) |
Start a factory process. More... | |
def | stop (self) |
Stop the factory process. More... | |
Launch a Work Queue factory.
The command line arguments for work_queue_factory
can be set for a factory object (with dashes replaced with underscores). Creating a factory object does not immediately launch it, so this is a good time to configure the resources, number of workers, etc. Factory objects function as Python context managers, so to indicate that a set of commands should be run with a factory running, wrap them in a with
statement. The factory will be cleaned up automatically at the end of the block. You can also make config changes to the factory while it is running. As an example,
# normal WQ setup stuff workers = work_queue.Factory("sge", "myproject") workers.cores = 4 with workers: # submit some tasks workers.max_workers = 300 # got a pile of tasks, allow more workers # any additional cleanup steps on the manager
def ndcctools.work_queue.Factory.__init__ | ( | self, | |
batch_type = "local" , |
|||
manager = None , |
|||
manager_host_port = None , |
|||
manager_name = None , |
|||
factory_binary = None , |
|||
worker_binary = None , |
|||
log_file = os.devnull |
|||
) |
Create a factory for the given batch_type and manager name.
One of manager_name
, manager_host_port
, or manager
should be specified. If factory_binary or worker_binary is not specified, $PATH will be searched.
def ndcctools.work_queue.Factory.start | ( | self | ) |
Start a factory process.
It's best to use a context manager (with
statement) to automatically handle factory startup and tear-down. If another mechanism will ensure cleanup (e.g. running inside a container), manually starting the factory may be useful to provision workers from inside a Jupyter notebook.
def ndcctools.work_queue.Factory.stop | ( | self | ) |
Stop the factory process.