Jobman

v 1.0, 2015-09-07, by Heng Sun

This page gives a brief introduction to Jobman. For a full manual in PDF, click here.

Introduction

Computation Job Management (jobman) is a program to call executables according a given program flow. Each executable is run in a separate process.

Jobman addresses the following situation. A project needs to carry out a series calculations that are performed by software written in different languages or supplied by third parties, according to certain program flow. The program may run in a machine with multiple processors. Writing multi-processed program is not your interest. However, invoking each individual program manually is error prone and hard to manage. This is especially the case in some scientific computing, quantitative finance, and prototyped programming. The various individual programs don't communicate with each other except via persistent storage like databases and flat files, where inputs are read and outputs are written. In short, Jobman treats individual executables like Lego blocks. It lets a user to build a program from those blocks in a flexible way.

Jobman has the following features:

Glue individual executables into a program according to a given program flow.
Run individual executables in multiple processes.
It supports some dynamic alterations of execution flows. Certain execution paths and user preferences can be modified when the program is running.
It supports resumed run. The program can start from where it was left last time.
The user can prioritize jobs when the number of jobs to be run in parallel exceeds the maximum number of parallels allowed.

In jobman, the program flow is expressed in certain configuration files. Each collection of tasks to be performed by individual executables is called a 'job'. The basic communications between individual jobs are via log files and parameter files.

Jobman is implemented in Java. To use the Jobman, Java Runtime Environment (JRE) 8 or above must be installed, which can be downloaded from Javasoft. However, no Java knowledge is required to use Jobman.

The project is hosted at SourceForge. The distribution of this program consists of a single JAR file jobman.jar. This JAR file, together with the source code, documentations, and demos, can be downloaded there.

Installation/Operation Overview

Download and unpack the program jobman at http://sourceforge.net/projects/jobman.
Copy the jar file dist/jobman.jar into some place where you want to run the program. Let's call this place jobman home directory. All directories and files mentioned in this document are relative to the jobman home directory.
In jobman home directory, create a directory (folder) called 'cfg', which will store all user configurations. In the directory cfg, create a files job_list.txt, and optional files opt_static.txt, opt_dynamic.txt, and/or job_exec.txt. (It is actually better to copy the files from a demo.) Edit the file according to the instructions in later sections.
Go to jobman home directory and execute the file jobman.jar: java -jar jobman.jar <job_name>

If you want to re-build jobman.jar, install Apache Ant first. Then execute ant in the jobman home directory. A subdirectory dist will contain the Java jar file jobman.jar.

For Javadoc APIs, please visit API Documentation.

Program Execution Flow Overview

Jobman controls program flow using the relationships between jobs. A parent job can contain child jobs. A job without a child is a terminal job, which is an executable to be run in a separate process or thread. So all the jobs to be run is like a hierarchical tree of jobs, with possible further dependencies among branches and leafs. When jobman executes, it starts from the top most job, and walks through certain jobs in the tree. In this process, it collects all the executables (terminal jobs) to be run. At the end of a walk, jobman starts executing the collected leaf jobs in separate processes/threads. In the meantime, it performs another round of walk of the tree of jobs. This process continues until all jobs are completed or some job fails.

Quick Look

Before running Jobman, you need to specify the flows by job_list.txt, The following section gives quick look at the expressions that Jobman builds job relationships.

We use Unix file path separator '/' for illustration. For Windows users, replace '/' with '\', all else the same.

Job Types and Specifications

You need to define the program flow according to the format specified by jobman. Jobs are building blocks of the program. Job specifications specify how jobs are glued together in the whole program. There are eight job specifications. A parent job consists of other jobs. A terminal job does not contain any other jobs.

loop: It is a parent job consisting of child jobs that run in a sequential order.
fork: It is a parent job consisting of child jobs that run in parallel.
plop: It is a parent job consisting of a sequence of parameterized jobs. Each child job uses the same set of executables with different parameters.
pfrk: It is a parent job consisting of a set of parameterized jobs that run in parallel. Each child job uses the same set of executables with a different parameters.
exec: It is an executable that perform an individual computation. An exec job is always run in a separate process.

A program flow may be expressed in terms of the above job specifications.

Job List File

You tell jobman the flow by editing cfg/job_list.txt. Except comments and blank lines, each line in the file tells how a job is performed. The format of each line is

[job_type] [job name] [child jobs or action]

The first element is the job type as explained in the previous section. The second element is the name of the job that you give. A job name can only use alphanumeric characters and underscore '_'. Special characters like spaces and tabs are not allowed in a job name. The last element depends on job type, which will be explained in details in the subsequent paragraphs. The syntax for each job type is given below.

loop [job name] [child job name 1] [child job name 2] ...
fork [job name] [child job name 1] [child job name 2] ...
plop [job name] [parameterization job name] [child job name]
pfrk [job name] [parameterization job name] [child job name]
exec [job name] [executable] [argument 1] [argument 2] ...

Formally, a job of type plop or pfrk can only contain two child jobs. The first child, which is the parameterization job in a plop or pfrk job, gives the parameters to be used by the other child job. Each parameter is an argument to the second child job. The parameters are written into a file [job_name].par in a subdirectory of logs in jobman home, one parameter per line by the parameterization job. A parameter line can contain the space and other special characters.

An example job_list.txt may look like this: (Line numbers at the beginning of each line are not part of the file.)

loop job1 job2 job3 job4 job5
exec job2 a/my_exe1 arg1 arg2
fork job3 job6 job7
exec job6 /b/c/my_exe2
plop job7 job8 job9
exec job8 script1 $par
exec job9 my_exe3 $job9
pfrk job4 job10 job11
exec job10 script2 $par
exec job11 my_exe4 $job11
exec job5 my_exe5

The first line in the configuration tells the top most job is job1, which will loop over job2, job3, job4, and job5.

The second line tells that job2 is an exec job, which is performed by executing a/my_exe1 in jobman home directory, with two arguments arg1 and arg2.

The third line tells that job3 is a fork job, which will run job6 and job7 in parallel.

According to the fourth line, job6 is performed by executing /b/c/my_exe2.

According to the fifth line, job7 will run job8 first, which will provide parameters for job9. Jobman will run job9 using the parameters from job8, in loop.

The sixth line tells that job8 gives parameters by calling script1. It writes the parameters into the last argument $par, which will be expanded into its parameter file by Jobman. A word starting with the dollar sign “$” is a variable.

Line 7 tells job9 is an exec job. Its last argument $job9 is a variable to be expanded by Jobman into the parameter passing to job9.

Line 8 tells that job4 is a pfrk job. It runs job11 with different parameters in parallel. The parameters are supplied by job10.

The above flow can be represented as follows:

Description: Description: Description: Example Flow Chart

A job inside a rounded box is a leaf. A job in rectangular box is a parent job. A wide horizontal bar represents a synchronization bar.

More sophisticated relationships among jobs can be specified. Please refer to the PDF manual.

Contents of Release

After unpack the released file jobman_<release>.tar.gz, you will see

Directory src contains all Java source code.
Directory dist contains the jar file jobman.jar.
Directory docs contains the manual PDF.
Directory docs/api contains JavaDoc HTML files.
Directories demo and examples contain demo programs.
File build.xml is the Ant buildfile.
File LICENSE.txt and release notes.

Legal

Computation Job Management (jobman)
        Managing the executions of programs like Lego blocks.
        Version 1.0
        http://sourceforge.net/projects/jobman
        Copyright © 2006-2015 Heng Sun <sunheng at hotmail dot com>

This program is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. You can see a copy of license here.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA