Grid[Way] Job Template Manager

The Grid[Way] Job Template Manager is an extension to the GridWay Metascheduler especially useful for parameter sweeping. It provides a lots of features in order to create, delete, submit, purge, kill and get information about a set of job templates with similar structure.

It is possible to define two levels of features:

• Job Template manager: Create and Delete job templates. These two subcommands are always available. Their mission is, respectivly, to generate and remove the files containing the job description. An argument file with the description of how to span the parameter space is compulsory for the creation.
• GridWay Job control: Submit, Purge, Kill, and Info subcommands which interact with the GridWay instance in order to perform control and monitoring over the jobs related to the job templates.

The project development is hosted as a subspace of The GridWay Metascheduler. There you may find an activity log as well as the official repository and downloads.

• http://dev.gridway.org/projects/gwjobtemplatemanager

The tool is an effort of academic research. If you used it or obtained benefit from it, it would be very nice to hear from you and get recognition through the following cite:

Alejandro Lorca, Eduardo Huedo, Ignacio M. Llorente
The Grid[Way] Job Template Manager, a tool for parameter sweeping
Computer Physics Communications, 182 (2011) pp. 1047-1060
ISSN 0010-4655, DOI: 10.1016/j.cpc.2010.12.041.

• The perl script
  
• The gzipped tar archive with README, LICENSE and examples

• 1.0 gamma (30 Oct 2009)
  
• 1.0 beta (19 Sep 2009)

The GridWay Job Template Manager does not require explicit configuration. It should work out of the box. In order to modify some parameters, use the --config option.

Usage: gw_job_template_manager [OPTION] SUBCOMMAND ARG(S)

 SUBCOMMANDS:
  -c, --create            Create templates
    ARG:  <PARAMETER_FILE>
         PARAMETER_FILE   File with parameter description,
    OPTION[S]: --worker WORKER_FILE --template TEMPLATE_FILE
         WORKER_FILE      Executable file to be run remotely,
         TEMPLATE_FILE    Append extra variables from template file.

  -d, --delete            Delete templates
    ARG:  <all|[un]submitted|[un]finished|[un]successful|FROM-TO>
         all              all the templates are deleted,
         [un]submitted    only those which were [not] submmited,
         [un]finished     only those whose jobs have [not] finished,
         [un]successful   only those whose jobs finished [un]succesfully,
         FROM-TO          deletes the range [FROM,TO].

  -s, --submit            Submit the jobs from templates
    ARG:  <all|[un]submitted|[un]finished|[un]successful|FROM-TO>
         all              all the templates are submitted/resubmitted,
         [un]submitted    only those which were [not] submmited,
         [un]finished     only those whose jobs have [not] finished,
         [un]successful   only those whose jobs finished [un]succesfully,
         FROM-TO          submit the range [FROM,TO].

  -p, --purge             Purge the existing jobs from templates
    ARG:  <all|[un]finished|[un]successful|FROM-TO>
         all              all the jobs are purged (waiting if necessary),
         [un]finished     only those which have [not] finished,
         [un]successful   only those [un]succesful,
         FROM-TO          purge the range [FROM,TO].

  -k, --kill              Kill the existing jobs from templates
    ARG:  <all|[un]finished|[un]successful|FROM-TO>
         all              all the jobs are killed (potentially slow),
         [un]finished     only those which have [not] finished,
         [un]successful   only those [un]succesful,
         FROM-TO          kill the range [FROM,TO],
    OPTION: --signal SIG
         SIG              kill with signal passed to gwkill.

  -i, --info              Information about status of the submitted jobs,
    ARG:  <history|now|evolution>
         history          full historic information about each job,
         now              last status update for each job,
         evolution        timely snapshots of job statuses,

 OPTIONS:
      --debug             Debugging information
      --config key=value  Assign a key=value pair for configuration settings.

 HELP and ABOUT:
  -v, --version           Version number of the programme.
  -l, --license           Credits and license.
  -h, --help              Print this help.

Each line in the argument file correspond, in order, to an argument to be passed to the worker binary. The syntax for each line is the following:

• LOOPTYPE=LIST VALUE=an_explicit_value [VALUE=another_explicit_value] [FUNCTION=perl_function]
• LOOPTYPE=[EXP]RANGE START=initial_value END=final_value [POINTS=number_of_points|STEP=increment] [SKIP=exclusion_value] [FUNCTION=perl_function]

Fields in brackets are optional.

• LOOPTYPE describes the structure, possible assigments are RANGE, EXPRANGE or LIST
• INITIAL_VALUE and FINAL_VALUE might be characters, integers or reals.
• POINTS is a positive integer > 1, default if absent is 2.
• STEP is an alternatively to POINTS, with it you can provide a fixed increment
• SKIP is an element you do not wish to appear
• FUNCTION is a perl function or operation. It preceedes the argument of the RANGE for sampling

1. Hello World! (1 string argument passed to the binary, 1 job template). The parameter file is

   LOOPTYPE=LIST, VALUE="Hello World!"

   Then with this command

   gw_job_template_manager --worker /bin/echo --create parameter.in
   

   you create a file '0_worker_Hello_World!.jt' with the following content

   NAME = 0_echo
   EXECUTABLE = echo
   ARGUMENTS = "Hello World!"
   STDOUT_FILE = 0_worker_Hello_World!.out
   STDERR_FILE = 0_worker_Hello_World!.err

2. A sample of x={1,2,3,4,5} (1 argument passed to the worker, 5 job templates)

   LOOPTYPE=RANGE, START=1, END=5, POINTS=5

   Your file 3_worker_4.jt in this case look like:

   NAME = 3_worker
   EXECUTABLE = worker
   ARGUMENTS = 4
   STDOUT_FILE = 3_worker_4.out
   STDERR_FILE = 3_worker_4.err

3. A more complex mixed sample of algorithm names and exponential sampling with output file (3 arguments, 6 job templates). The argument.in file

   LOOPTYPE=LIST, VALUE=Taylorexpansion, VALUE=Exactanalytic
   LOOPTYPE=EXPRANGE, START=1, END=10000, STEP=2
   LOOPTYPE=${JT_ID}.output

   and the template.in file for appending lines to the job templates

   OUTPUT_FILES = ${JT_ID}.output

   Then you have the following job template files:

   $ ls -1 *.jt
   0_worker_Taylorexpansion_1_0.output.jt
   1_worker_Taylorexpansion_100_1.output.jt
   2_worker_Taylorexpansion_10000_2.output.jt
   3_worker_Exactanalytic_1_3.output.jt
   4_worker_Exactanalytic_100_4.output.jt
   5_worker_Exactanalytic_10000_5.output.jt
   

   and the content of 5_worker_Exactanalytic_10000_5.output.jt

   NAME = 5_worker
   EXECUTABLE = worker
   ARGUMENTS = Exactanalytic 10000 5.output
   STDOUT_FILE = 5_worker_Exactanalytic_10000_5.output.out
   STDERR_FILE = 5_worker_Exactanalytic_10000_5.output.err
   OUTPUT_FILES = 5.output

1. Contribution to EGEE'09 - Uniting our strengths to realise a sustainable European grid