Cooperative Computing Tools Documentation

Getting Started

Software Components

  • TaskVine is our third-generation workflow system for building data-intensive workflow applications. TaskVine applications consist of many chained tasks that pull in external data into a cluster, where computed results can be cached and re-used by later tasks, even in successive workflows.

  • Work Queue is a system and library for creating and managing scalable manager-worker style programs that scale up to thousands of machines on clusters, clouds, and grids. Work Queue programs are easy to write in Python (example|api) or C (example|api) .

  • Makeflow is a workflow system for parallel and distributed computing using either the classic Make syntax or the more advanced JX Workflow Language. Using Makeflow, you can write simple scripts that easily execute on hundreds or thousands of machines.

  • Resource Monitor is a tool to monitors the cpu, memory, io, and disk usage of applications running in distributed systems, and can optionally enforce limits on each resource. The monitor can be compiled to a single executable that is easily deployed to track executable file, or it can be used as a library to track the execution of Python functions.

  • Parrot is a transparent user-level virtual filesystem that allows any ordinary program to be attached to many different remote storage systems, including HDFS, iRODS, Chirp, and FTP.

  • Chirp is a personal user-level distributed filesystem that allows unprivileged users to share space securely, efficiently, and conveniently. When combined with Parrot, Chirp allows users to create custom wide-area distributed filesystems.

  • Catalog Server is a common facility used to monitor running services, workflows, and tasks. It provides real-time status and historical data on all components of the CCTools.

Research Prototypes

  • Accelerated Weighted Ensemble (AWE) is an ensemble molecular dynamics applications that uses Work Queue to scale out molecular simulations to thousands of GPUs on multipel clusters.

  • Confuga is an active storage cluster file system designed for executing DAG-structured scientific workflows. It is used as a collaborative distributed file system and as a platform for execution of scientific workflows with full data locality for all job dependencies.

  • Umbrella is a tool for specifying and materializing execution environments, from the hardware all the way up to software and data. Umbrella parses a task specification and determines the minimum mechanism necessary to run it. It downloads missing dependencies, and executes the application through the available minimal mechanism, which may be direct execution, a system container, a virtual machine, or submissions to a cloud and cluster environments.

  • Prune Prune is a system for executing and precisely preserving scientific workflows to ensure reproducibility. Every task to be executed in a workflow is wrapped in a functional interface and coupled with a strictly defined environment.

Reference Information