ndcctools.taskvine.task.PythonTask Class Reference

Inheritance diagram for ndcctools.taskvine.task.PythonTask:

Public Member Functions
def	__init__ (self, func, *args, **kwargs)
	Creates a new python task. More...
def	submit_finalize (self)
	Finalizes the task definition once the manager that will execute is run. More...
def	enable_temp_output (self)
	Marks the output of this task to stay at the worker. More...
def	set_output_cache (self, cache=False)
	Set the cache behavior for the output of the task. More...
def	output_file (self)
	Returns the ndcctools.taskvine.file.File object that represents the output of this task. More...
def	output (self)
	returns the result of a python task as a python variable More...
def	disable_output_serialization (self)
	Disables serialization of results to disk when writing to a file for transmission. More...
Public Member Functions inherited from ndcctools.taskvine.task.Task
def	__init__ (self, command, **task_info)
	Create a new task specification. More...
def	clone (self)
	Return a copy of this task. More...
def	set_command (self, command)
	Set the command to be executed by the task. More...
def	needs_library (self, library_name)
	Set the name of the library at the worker that should execute the task's command. More...
def	provides_library (self, library_name)
	Set the library name provided by this task. More...
def	set_function_slots (self, nslots)
	Set the number of concurrent functions a library can run. More...
def	set_scheduler (self, scheduler)
	Set the worker selection scheduler for task. More...
def	set_tag (self, tag)
	Attach a user defined logical name to the task. More...
def	set_category (self, name)
	Label the task with the given category. More...
def	add_feature (self, name)
	Label the task with the given user-defined feature. More...
def	add_input (self, file, remote_name, strict_input=False, mount_symlink=False)
	Add any input object to a task. More...
def	add_output (self, file, remote_name, watch=False, failure_only=None, success_only=None)
	Add any output object to a task. More...
def	set_snapshot_file (self, filename)
	When monitoring, indicates a json-encoded file that instructs the monitor to take a snapshot of the task resources. More...
def	add_starch_package (self, file)
	Add a Starch package as an execution context. More...
def	add_poncho_package (self, file)
	Add a Poncho package as an execution context. More...
def	add_execution_context (self, f)
	Adds an execution context to the task. More...
def	set_retries (self, max_retries)
	Indicate the number of times the task should be retried. More...
def	set_max_forsaken (self, max_forsaken)
	Indicate the number of times the task can be returned to the manager without being executed. More...
def	set_cores (self, cores)
	Indicate the number of cores required by this task. More...
def	set_memory (self, memory)
	Indicate the memory (in MB) required by this task. More...
def	set_disk (self, disk)
	Indicate the disk space (in MB) required by this task. More...
def	set_gpus (self, gpus)
	Indicate the number of GPUs required by this task. More...
def	set_priority (self, priority)
	Indicate the the priority of this task (larger means better priority, default is 0). More...
def	set_env_var (self, name, value=None)
	Set this environment variable before running the task. More...
def	set_monitor_output (self, directory)
	Set a name for the resource summary output directory from the monitor. More...
def	tag (self)
	Get the user-defined logical name for the task. More...
def	category (self)
	Get the category name for the task. More...
def	command (self)
	Get the shell command executed by the task. More...
def	std_output (self)
	Get the standard output of the task. More...
def	id (self)
	Get the task id number. More...
def	exit_code (self)
	Get the exit code of the command executed by the task. More...
def	result (self)
	Return a string that explains the result of a task. More...
def	completed (self)
	Return True if task executed and its command terminated normally. More...
def	successful (self)
	Return True if task executed successfully, (i.e. More...
def	get_metric (self, name)
	Return various integer performance metrics about a completed task. More...
def	addrport (self)
	Get the address and port of the host on which the task ran. More...
def	hostname (self)
	Get the address and port of the host on which the task ran. More...
def	resources_measured (self)
	Get the resources measured for the task execution if resource monitoring is enabled. More...
def	limits_exceeded (self)
	Get the resources the task exceeded. More...
def	resources_requested (self)
	Get the resources the task requested to run. More...
def	resources_allocated (self)
	Get the resources allocated to the task in its latest attempt. More...
def	add_nopen (self, manager)
	Adds inputs for nopen library and rules file and sets LD_PRELOAD. More...

Constructor & Destructor Documentation

__init__()

def ndcctools.taskvine.task.PythonTask.__init__	(		self,
			func,
		*	args,
		**	kwargs
	)

Creates a new python task.

Parameters

self	Reference to the current python task object
func	python function to be executed by task
args	arguments used in function to be executed by task
kwargs	keyword arguments used in function to be executed by task

Member Function Documentation

submit_finalize()

def ndcctools.taskvine.task.PythonTask.submit_finalize

(

self

)

Finalizes the task definition once the manager that will execute is run.

This function is run by the manager before registering the task for execution.

Parameters

self	Reference to the current python task object
manager	Manager to which the task was submitted

Reimplemented from ndcctools.taskvine.task.Task.

Reimplemented in ndcctools.taskvine.task.FunctionCall, ndcctools.taskvine.futures.FuturePythonTask, and ndcctools.taskvine.futures.FutureFunctionCall.

enable_temp_output()

def ndcctools.taskvine.task.PythonTask.enable_temp_output

(

self

)

Marks the output of this task to stay at the worker.

Functions that consume the output of this tasks have to add self.output_file as an input, and cloudpickle.load() it.

E.g.:

ta = PythonTask(fn, ...)
ta.enable_temp_output()
tid = m.submit(ta)
 
t = m.wait(...)
if t.id == tid:
  tb = PythonTask(fn_with_tmp, "ta_output.file")
  tb.add_input(ta.output_file, "ta_output.file")
  m.submit(tb)

where fn_with_tmp may look something like this:

def fn_with_tmp(filename):
  import cloudpickle
  with open(filename) as f:
    data = cloudpickle.load(f)

Parameters

self	Reference to the current python task object

set_output_cache()

def ndcctools.taskvine.task.PythonTask.set_output_cache	(		self,
			cache = False
	)

Set the cache behavior for the output of the task.

Parameters

cache

If True or 'workflow', cache the file at workers for reuse until the end of the workflow. If 'always', the file is cache until the end-of-life of the worker. Default is False (file is not cache).

output_file()

def ndcctools.taskvine.task.PythonTask.output_file

(

self

)

Returns the ndcctools.taskvine.file.File object that represents the output of this task.

output()

def ndcctools.taskvine.task.PythonTask.output

(

self

)

returns the result of a python task as a python variable

Parameters

self	reference to the current python task object

Reimplemented from ndcctools.taskvine.task.Task.

Reimplemented in ndcctools.taskvine.task.FunctionCall.

disable_output_serialization()

def ndcctools.taskvine.task.PythonTask.disable_output_serialization

(

self

)

Disables serialization of results to disk when writing to a file for transmission.

WARNING: Only do this if the function itself encodes the output in a way amenable for serialization.

Parameters

self	Reference to the current python task object

---

The documentation for this class was generated from the following file:

• task.py