Installing the Cooperative Computing Tools

Overview

The Cooperative Computing Tools (CCTools) are a collection of programs that enable large scale distributed computing on systems such as clusters, clouds, and grids. These tools are commonly used in fields of science and engineering that rely on large amounts of computing.

CCTools can be installed on Linux and Mac.

Install from Conda

If you need a personal installation of CCTools, say for your laptop, or an account in your campus cluster, the easiest way to install CCTools is using the conda package manager. To check that you have conda install, in a terminal try:

$ conda list

If it fails, then you need to install either miniconda or anaconda. Miniconda is a light version of anaconda, and we recommend it as it is much faster to install. We also recommend installing the versions for Python 3.7

Warning

On Mac, the available from conda CCTools does not work with Python 2.7 or with perl. For such case, please compile CCTools from source.

With conda command available, install CCTools with:

$ conda install -y -c conda-forge ndcctools

And that's it! You can test your setup following the instructions here.

Install from Spack

Alternatively, you can install CCTools using the spack.io package manager. Spack will compile CCTools for you, and it is recommended for HPC sites for which a conda installation is not available, or which have special software stack requirements (such as specially compiled python versions).

First you need to check that the spack command is available. In a terminal type:

$ spack help

If this command fails, then please install spack following the instructions here.

Once spack is installed, install CCTools with:

$ spack install cctools

To use CCTools, you need to load it into the spack environment with:

$ spack load cctools

You only need to do spack install once, but you will need spack load cctools everytime you want to use CCTools.

Once this command finished, you can test your installation following the instructions here.

Install From Source

To install from source, please follow the instructions here.

You will only need to install CCTools from source if you want to track our latest developments on our github repository, to propose a bug fix, or if you need to add a driver support to parrot not normally included in the conda installation, such as CVMFS, or iRODS.

Install from Binary Tarball

Binary packages are available for several operating systems. Please follow the instructions here.

Testing Your Installation

You can test that the python and perl modules are available with:

$ python -c 'import work_queue; print(work_queue.WORK_QUEUE_DEFAULT_PORT)'
9123

$ perl -MWork_Queue -e 'print("$Work_Queue::WORK_QUEUE_DEFAULT_PORT\n")'
9123

If the above commands fail, please make sure that you follow one (and only one!) of the methods above. For example, if you are using a conda installation, make sure that your PYTHONPATH is unset.

You can test the availability of CCTools* commands with:

$ makeflow --version

makeflow version X.Y.Z...
...(output trimmed)...

Warning

Remember that for installations from source you need swig at compile time, and to set the environment variables PATH, PYTHONPATH and PERL5LIB appropriately, as explained here.

For conda and spack installation you should not need to manually set any of these variables, and in fact setting them may produce errors.

License

The Cooperative Computing Tools are Copyright (C) 2003-2004 Douglas Thain and Copyright (C) 2005- The University of Notre Dame.

All rights reserved.

This software is distributed under the GNU General Public License.