Random task graph

Random task graph example demonstrates the capabilities of wfGenes in task coupling and code generation for different workflow management systems. We will see how the GUI provides necessary functionalities to model your workflow and use various executors to perform computation, scaling from personal to supercomputer centers. For the sake of simplicity, as the first example, we consider a simple sleep function that resembles computation in real life applications. As it is explained in the setup section, wfGenes’s GUI can be run on top Jupyter notebook either on your personal computers or on JupyterHUB.

Remote Engine

Alternatively, we can use wfGene remote executor and setup the same workflow using local machine and perform computation on remote machine. Overall, working on a local machine and submitting jobs on remote machines can provide a flexible and efficient way to perform computational tasks while removing the burden on local hardware resources. However, in order to work with remote machines using your local machine, you need to complete certain steps beforehand. These steps are as follows:

1. Generate ssh key with strong pass phrase

2. Registering your ssh key on bwIDM/bwServces .

3. Start an SSH agent and add one or more SSH keys to it.

eval $(ssh-agent); ssh-add

Note

Interactive SSH Keys are not valid all the time, but only for one hour after the last 2-factor login. They have to be “unlocked” by entering the OTP and service password.

Once you have completed these steps, you will be able to log in to the cluster using a passwordless SSH key and proceed with the example demonstrated in the tutorial below.

The example of a random task graph suggests how wfGenes is capable of coupling tasks and parallelizing them by translating the graph, which consists of thousands of nodes, to various WMSs that run on thousands of cores. However, for simple benchmarking we also modeled random task graph with ten nodes presented in the tutorial videos.