====== Grid[Way] Job Template Manager ====== ===== About ===== The Grid[Way] Job Template Manager is an extension to the GridWay Metascheduler especially useful for parameter sweeping. It provides a lots of features in order to create, delete, submit, purge, kill and get information about a set of job templates with similar structure. It is possible to define two levels of features: * Job Template manager: **Create** and **Delete** job templates. These two subcommands are always available. Their mission is, respectivly, to generate and remove the files containing the job description. An argument file with the description of how to span the parameter space is compulsory for the creation. * GridWay Job control: **Submit**, **Purge**, **Kill**, and **Info** subcommands which interact with the GridWay instance in order to perform control and monitoring over the jobs related to the job templates. ===== Development ===== The project development is hosted as a subspace of The GridWay Metascheduler. There you may find an activity log as well as the official repository and downloads. * [[http://dev.gridway.org/projects/gwjobtemplatemanager]] ===== Cites ===== The tool is an effort of academic research. If you used it or obtained benefit from it, it would be very nice to hear from you and get recognition through the following cite: Alejandro Lorca, Eduardo Huedo, Ignacio M. Llorente The Grid[Way] Job Template Manager, a tool for parameter sweeping Computer Physics Communications, 182 (2011) pp. 1047-1060 ISSN 0010-4655, DOI: 10.1016/j.cpc.2010.12.041. ===== Installation ===== ==== Download ==== === Latest release: Version 1.0, (26 Feb 2010) === * [[http://dev.gridway.org/attachments/download/48/gw_job_template_manager| The perl script]]\\ * [[http://dev.gridway.org/attachments/download/47/gw_job_template_manager.tar.gz| The gzipped tar archive with README, LICENSE and examples]] ==== Older preview releases ==== * [[http://dev.gridway.org/attachments/download/34/gw_job_template_manager| 1.0 gamma]] (30 Oct 2009)\\ * [[http://dev.gridway.org/attachments/download/25/gw_job_template_manager|1.0 beta]] (19 Sep 2009) ==== Configuration ==== The GridWay Job Template Manager does not require explicit configuration. It should work out of the box. In order to modify some parameters, use the ''--config'' option. ===== Usage ===== Usage: gw_job_template_manager [OPTION] SUBCOMMAND ARG(S) SUBCOMMANDS: -c, --create Create templates ARG: PARAMETER_FILE File with parameter description, OPTION[S]: --worker WORKER_FILE --template TEMPLATE_FILE WORKER_FILE Executable file to be run remotely, TEMPLATE_FILE Append extra variables from template file. -d, --delete Delete templates ARG: all all the templates are deleted, [un]submitted only those which were [not] submmited, [un]finished only those whose jobs have [not] finished, [un]successful only those whose jobs finished [un]succesfully, FROM-TO deletes the range [FROM,TO]. -s, --submit Submit the jobs from templates ARG: all all the templates are submitted/resubmitted, [un]submitted only those which were [not] submmited, [un]finished only those whose jobs have [not] finished, [un]successful only those whose jobs finished [un]succesfully, FROM-TO submit the range [FROM,TO]. -p, --purge Purge the existing jobs from templates ARG: all all the jobs are purged (waiting if necessary), [un]finished only those which have [not] finished, [un]successful only those [un]succesful, FROM-TO purge the range [FROM,TO]. -k, --kill Kill the existing jobs from templates ARG: all all the jobs are killed (potentially slow), [un]finished only those which have [not] finished, [un]successful only those [un]succesful, FROM-TO kill the range [FROM,TO], OPTION: --signal SIG SIG kill with signal passed to gwkill. -i, --info Information about status of the submitted jobs, ARG: history full historic information about each job, now last status update for each job, evolution timely snapshots of job statuses, OPTIONS: --debug Debugging information --config key=value Assign a key=value pair for configuration settings. HELP and ABOUT: -v, --version Version number of the programme. -l, --license Credits and license. -h, --help Print this help. ==== Argument file syntax ==== Each line in the argument file correspond, in order, to an argument to be passed to the worker binary. The syntax for each line is the following: * LOOPTYPE=LIST VALUE=an_explicit_value [VALUE=another_explicit_value] [FUNCTION=perl_function] * LOOPTYPE=[EXP]RANGE START=initial_value END=final_value [POINTS=number_of_points|STEP=increment] [SKIP=exclusion_value] [FUNCTION=perl_function] Fields in brackets are optional. * LOOPTYPE describes the structure, possible assigments are RANGE, EXPRANGE or LIST * INITIAL_VALUE and FINAL_VALUE might be characters, integers or reals. * POINTS is a positive integer > 1, default if absent is 2. * STEP is an alternatively to POINTS, with it you can provide a fixed increment * SKIP is an element you do not wish to appear * FUNCTION is a perl function or operation. It preceedes the argument of the RANGE for sampling ==== Examples ==== - Hello World! (1 string argument passed to the binary, 1 job template). The parameter file is LOOPTYPE=LIST, VALUE="Hello World!" Then with this command gw_job_template_manager --worker /bin/echo --create parameter.in you create a file '0_worker_Hello_World!.jt' with the following content NAME = 0_echo EXECUTABLE = echo ARGUMENTS = "Hello World!" STDOUT_FILE = 0_worker_Hello_World!.out STDERR_FILE = 0_worker_Hello_World!.err - A sample of x={1,2,3,4,5} (1 argument passed to the worker, 5 job templates) LOOPTYPE=RANGE, START=1, END=5, POINTS=5 Your file ''3_worker_4.jt'' in this case look like: NAME = 3_worker EXECUTABLE = worker ARGUMENTS = 4 STDOUT_FILE = 3_worker_4.out STDERR_FILE = 3_worker_4.err - A more complex mixed sample of algorithm names and exponential sampling with output file (3 arguments, 6 job templates). The ''argument.in'' file LOOPTYPE=LIST, VALUE=Taylorexpansion, VALUE=Exactanalytic LOOPTYPE=EXPRANGE, START=1, END=10000, STEP=2 LOOPTYPE=${JT_ID}.output and the ''template.in'' file for appending lines to the job templates OUTPUT_FILES = ${JT_ID}.output Then you have the following job template files: $ ls -1 *.jt 0_worker_Taylorexpansion_1_0.output.jt 1_worker_Taylorexpansion_100_1.output.jt 2_worker_Taylorexpansion_10000_2.output.jt 3_worker_Exactanalytic_1_3.output.jt 4_worker_Exactanalytic_100_4.output.jt 5_worker_Exactanalytic_10000_5.output.jt and the content of ''5_worker_Exactanalytic_10000_5.output.jt'' NAME = 5_worker EXECUTABLE = worker ARGUMENTS = Exactanalytic 10000 5.output STDOUT_FILE = 5_worker_Exactanalytic_10000_5.output.out STDERR_FILE = 5_worker_Exactanalytic_10000_5.output.err OUTPUT_FILES = 5.output ===== Training Material ===== ==== Presentations ==== - [[http://indico.cern.ch/contributionDisplay.py?contribId=353&sessionId=27&confId=55893|Contribution to EGEE'09 - Uniting our strengths to realise a sustainable European grid]]