Configuration Guide 5.4

GridWay requires that the environment variables GLOBUS_LOCATION and GW_LOCATION are set. These are set to the base of your Globus installation and GridWay installation. In GT 4.2, GridWay is installed in the same place as Globus, so you can set both of these environment variables to the same location.

Login as root account and follow the next steps:

1. All of the GridWay users must be members of the same UNIX group, “<gwgroup>”. We recommend creating a new group (called “gwusers”, for example), and make sure that all GridWay user accounts are members of this new group.
2. The GridWay administrator account, “<adminuser>”, can be an existing administrative login or a new login. We recommend using the Globus account for the GridWay administration user. This account will own all of the files in the GridWay installation plus all of the daemons in the GridWay execution and it can be used to configure GridWay once it is installed. Primary group of “<adminuser>” should be “<gwgroup>”.DO NOT use root account for the GridWay administrator account.
3. The “sudoers” file of the sudo command should include the following:

   ...
   # User alias specification
   ...
   Runas_Alias     GW_USERS = %<gwgroup>
   ...
   # GridWay entries
   globus ALL=(GW_USERS)     NOPASSWD: /home/gwadmin/gw/bin/gw_em_mad_prews *
   globus ALL=(GW_USERS)     NOPASSWD: /home/gwadmin/gw/bin/gw_em_mad_ws *
   globus ALL=(GW_USERS)     NOPASSWD: /home/gwadmin/gw/bin/gw_tm_mad_ftp *

Usually sudo clears all environment variables for security reasons. However MADs need the GW_LOCATION and GLOBUS_LOCATION variables to be set. To preserve those variables in the MAD environment, add the following line to your “sudoers” file:

Defaults>GW_USERS env_keep="GW_LOCATION GLOBUS_LOCATION"

The following line shouldn't be in the sudoers file, otherwise gridway could not use sudo as it will ask for a tty:

Defaults    requiretty

Please refer to the sudo manual page for more information.To test the sudo command configuration try to execute a MAD as a user in the “<gwgroup>” group, for example:

$ sudo -u <gw_user> /home/gwadmin/gw/bin/gw_em_mad_prews

The configuration files for GridWay are read from the following locations:

• “$GW_LOCATION/etc/gridway/gwd.conf”: Configuration options for the GridWay daemon (GWD).
• “$GW_LOCATION/etc/gridway/sched.conf”: Configuration options for the GridWay built-in scheduling policies (see Section “Scheduler Configuration” for more information).
• “$GW_LOCATION/etc/gridway/job_template.default”: Default values for job's templates (i.e. job definition files).
• “$GW_LOCATION/etc/gridway/gwrc”: Default environment variables for MADs.

Options are defined one per line, with the following format:

<option> = [value]

If the value is missing the option will fall back to its default. Blank lines and any character after a '#' are ignored. Note: Job template options can use job or host variables to define their value, these variables are substituted at run time with their corresponding values (see GridWay 5.4: User's Guide).

The GridWay daemon (GWD) configuration options are defined in “$GW_LOCATION/etc/gridway/gwd.conf”. The table below summarizes the configuration file options, their description and default values. Note that blank lines and any character after a '#' are ignored.

Table 2. GWD Configuration File Options.

Here is an example of a GWD configuration file:

     #--------------------------------
     # Example: GWD Configuration File
     #--------------------------------
     GWD_PORT              = 6725
     MAX_NUMBER_OF_CLIENTS = 20
     NUMBER_OF_ARRAYS = 20
     NUMBER_OF_JOBS   = 200
     NUMBER_OF_HOSTS  = 100
     NUMBER_OF_USERS  = 30
     JOBS_PER_SCHED = 10
     JOBS_PER_HOST  = 10
     JOBS_PER_USER  = 30
     SCHEDULING_INTERVAL = 30
     DISCOVERY_INTERVAL  = 300
     MONITORING_INTERVAL = 120
     POLL_INTERVAL       = 60
     IM_MAD = mds4:gw_im_mad_mds4:-s hydrus.dacya.ucm.es:gridftp:ws
     TM_MAD = gridftp:gw_tm_mad_ftp:
     EM_MAD = ws:gw_em_mad_ws::rsl2
     DM_SCHED = flood:gw_flood_scheduler:-h 10 -u 30 -c 5

Default values for every job template option can be set in “$GW_LOCATION/etc/gridway/job_template.default”. You can use this file to set the value of advanced job configuration options and use them for all your jobs. Note that the values set in a job template file override those defined in “job_template.default”. See GridWay 5.4: User's Guide for a detailed description of each job option.

GridWay reporting and accounting facilities provide information about overall performance and help troubleshoot configuration problems. GWD generates the following logs under the “$GW_LOCATION/var” directory:

• “$GW_LOCATION/var/gwd.log”: System level log. You can find log information of the activity of the middleware access drivers; and a coarse-grain log information about jobs.
• “$GW_LOCATION/var/sched.log”: Scheduler log. You can find log information to fit the scheduler policies to your organization needs.
• “$GW_LOCATION/var/$JOB_ID/job.log”: Detailed log information for each job, it includes details of job resource usage and performance.
• “$GW_LOCATION/var/acct”: Accounting information. Use the gwacct command to access the data bases. Note that you need Berkeley DB library (version 4.4.20).
• “$GW_LOCATION/var/.lock”: Used to prevent from running more than one instance of the daemon.
• “$GW_LOCATION/var/gwd.port”: TCP/IP port GWD is listening for client connection requests.
• “$GW_LOCATION/var/globus-gw.log”: Used to encapsulate GridWay in a GRAM service (please refer to the Grid4Utility project web page for more information). This log file follows the globus fork job starter's format (based on SEG, Scheduler Event Generator messages):

  001;TIMESTAMP;JOBID;STATE;EXIT_CODE

Since GridWay 4.9, when you start the daemon, gwd tries to recover its previous state. This is, any submitted job is stored in a persistent pool, and in case of a gwd (or client machine) crash these jobs are recovered. This includes, for jobs in wrapper state, contacting with the remote jobmanager.

Recovery actions are performed by default, if you do not want to recover the previous submitted jobs use the -c option.

For example, to start gwd in multi-user mode and clear its previous state, use:

$ gwd -c -m

Grid scheduling consists of finding a suitable (in terms of a given target) assignment between a computational workload (jobs) and computational resources. The scheduling problem has been thoroughly studied in the past and efficient algorithms have been devised for different computing platforms. Although some of the experience gained in scheduling can be applied to the Grid, it presents some characteristics that differ dramatically from classical computing platforms (i.e. clusters or MPPs), namely: different administration domains, limited control over resources, heterogeneity and dynamism.

Grid scheduling is an active research area. The Grid scheduling problem is better understood today and several heuristics, performance models and algorithms have been proposed and evaluated with the aid of simulation tools. However, current working Grid schedulers are only based on match-making, and barely consider multi-user environments.

In this section, we describe the state-of-the-art scheduling policies implemented in the GridWay system. The contents of this guide reflect the experience obtained since GridWay version 4, and a strong feedback from the GridWay user community.

The scheduler is responsible for assigning jobs to Grid resources; therefore, it decides when and where to run a job. These decisions are made periodically in an endless loop. The frequency of the scheduler interventions can be adjusted with the “SCHEDULER_INTERVAL” configuration parameter (see Section “GridWay Daemon (GWD) Configuration”).

In order to make job to resource assignments the scheduler receives information from the following sources (see Figure 5, “Job Scheduling in GridWay”:

1. List of jobs in the system, which includes pending jobs as well as running jobs (those in wrapper state). Those jobs that cannot be started are filtered out from the list, i.e., jobs with unmet dependencies, stopped or held.
2. Match-making results: The Information Manager drivers query the Grid information services to track the availability and status of Grid resources. The discovery and monitoring processes can both be configured as static or dynamic, see Section “Information Driver Configuration”. This information is used by the GridWay core to build a list of suitable resources for each job, i.e., resources meeting the job requirements, and to compute their rank.
3. Current resource behavior: The scheduler considers the way a resource is behaving when making its decisions. In particular, it evaluates the migration and failure rates and execution statistics (transfer, execution and queue wait times).
4. Past Grid Usage: The scheduler also considers the past history (behavior) of Grid resources to issue schedules. Note that database support needs to be compiled in GridWay for this feature.

The information gathered from the previous sources is combined with a given scheduling policy to prioritize jobs and resources. Then, the scheduler dispatches the highest priority job to the best resource for it. The process continues until all jobs are dispatched, and those that could not be assigned wait for the next scheduling interval.

Figure 5. Job Scheduling in GridWay

A scheduling policy is used to assign a dispatch priority to each job and a suitability priority to each resource. Therefore, a Grid scheduling policy comprises two components:

• Job prioritization policies. Pending jobs are prioritized according to four policies: fixed, share, deadline and waiting-time. The job policies are used to sort the jobs of the users of a given scheduling domain (GridWay instance). Note that these policies are only enforced in the scheduling domain and not for the whole Grid infrastructure as discussed above.
• Resource prioritization policies. A given job can be executed on those resources that match its requirements. The resource policies are used to sort the matching resource list of each job. The matching resources are prioritized according to four policies: fixed, usage, failure and rank. Note that these policies do not only depend on the Grid resource but also on the job owner, as each Grid user (or VO member) has its own access rights and usage history.

These two top-level policies can be combined to implement a wide range of scheduling schemes (see Figure 6, “Job and resource prioritization policies in GridWay.”. The above scheduling policies are described in the following sections.

Figure 6. Job and resource prioritization policies in GridWay.

The job prioritization policies allow Grid administrators to influence the dispatch order of the jobs, that is, to decide which job is sent to the Grid. Traditionally, DRMS implement different policies based on the owner of the job, the resources consumed by each user or the requirements of the job. Some of these scheduling strategies can be directly applied in a Grid, while others must be adapted because of their unique characteristics: dynamism, heterogeneity, high fault rate and site autonomy.

This policy assigns a fixed priority to each job. The fixed priority ranges from 00 (lowest priority) to 19 (highest priority), so jobs with a higher priority will be dispatched first. The default priority values are assigned, by the Grid administrator, using the following criteria:

• User. All jobs of a User are given a fixed priority.
• Group. All jobs of a user belonging to a given Group get a fixed priority.

The user priority prevails over the group one. Also there is a special user (“DEFAULT”) to define the default priority value when no criteria apply.

The users can set the priority of their own jobs (gwsubmit -p) but without exceeding their limit set by the administrator in the scheduler configuration file.

Here is an example configuration for the fixed priority (see also Section “Built-in Scheduler Configuration File”):

     # Weight for the Fixed priority policy
     FP_WEIGHT = 1

     # Fixed priority values for David's and Alice's jobs
     FP_USER[david]  = 2
     FP_USER[alice]  = 12

     # Fixed priority for every body in the staff group
     FP_GROUP[staff] = 5

     # Anyone else gets a default priority 3
     FP_USER[DEFAULT] = 3

The Grid administrator can also set the fixed priority of a job to 20. When a job gets a fixed priority of 20, it becomes an urgent job. Urgent jobs are dispatched as soon as possible, bypassing all the scheduling policies.

The fair-share policy allows you to establish a dispatching ratio among the users of a scheduling domain. For example, a fair-share policy could establish that jobs from David and Alice must be dispatched to the Grid in a 2:5 ratio. In this case, the scheduler tracks the jobs submitted to the Grid by these two users and dispatches the jobs so they target a 2:5 ratio of job submissions.

This policy resembles the well-known fair-share of traditional LRMS. However, note that what GridWay users share is the ability to submit a job to the Grid and not resource usage. Resource usage share cannot be imposed at a Grid level, as Grid resources are shared with other Grid users and with local users from the remote organization. In addition, the set of resources that can be potentially used by each user is not homogeneous, as each user may belong to a different VO.

GridWay tracks the jobs submitted to the Grid by the users over time. Grid administrators can specify a timeframe over which user submissions are evaluated. The amount of time considered by GridWay is defined by a number of time intervals (“SH_WINDOW_DEPTH”) and the duration of each one (“SH_WINDOW_SIZE”, in days). The effective number of submissions in a given window is exponentially damped, so present events become more relevant.

Here is an example configuration for the share policy (see also Section “Built-in Scheduler Configuration File”):

     # Weight for the Fair-share policy
     SH_WEIGHT = 1

     # Shared values for David's and Alice's submissions
     SH_USER[david]  = 2
     SH_USER[alice]  = 5

     # Anyone else gets a default share value of 1
     SH_USER[DEFAULT] = 1

     # Consider submissions in the last 5 days
     SH_WINDOW_SIZE = 1
     SH_WINDOW_DEPTH= 5

The goal of this policy is to prevent low-priority jobs from starving. So jobs in the pending state long enough will be eventually submitted to the Grid. This policy can be found in most of the DRMS today. In GridWay, the priority of a job is increased linearly with the waiting time.

Here is an example configuration for this policy:

      # Weight for the Waiting-time policy
      WT_WEIGHT = 1

GridWay includes support for specifying deadlines at job submission. The scheduler will increase the priority of a job as its deadline approaches.

Grid administrators should provide a way to qualify the remaining time to reach the job deadline by defining when a job should get half of the maximum priority assigned by this policy (“DL_HALF”, in days).

Here is an example configuration for the deadline policy (see also Section “Built-in Scheduler Configuration File”):

      # Weight of the Deadline Policy
      DL_WEIGHT = 1

      # Assign half of the priority two days before the deadline
      DL_HALF = 2

The list of all pending jobs is sorted by the dispatch priority, which is computed as a weighted sum of the contribution from the previous policies. In this way, the Grid administrator can implement different scheduling schemes by adjusting the policy weights.

The dispatch priority (P) of a job (j) is therefore computed as:

P_j = Σ_i w_i p_ij , where i={fixed, share, wait-time, deadline}

where w_i is the weight for each policy (integer value) and p_ij is the priority (normalized) contribution from each policy.

The resource prioritization policies allow Grid administrators to influence the usage of resources made by the users, that is, decide where to run a job. Usually, in classical DRMS, this resource usage is administered by means of the queue concept.

In GridWay, the scheduler builds a meta-queue (a queue consisting of the local queues of the Grid resources) for each job based on its requirements (e.g., operating system or architecture). Note that this meta-queue is not only built in terms of resource properties but is also based upon the owner of the job, (as each Grid user may belong to a different VO with its own access rights and usage privileges).

The meta-queue for a job consists of the queues of those resources that meet the job requirements specified in the job template and have at least one free slot. By default, this queue is sorted in a first-discovered first-used fashion. This order can be influenced by means of the subsequent resource prioritization policies.

Usually, GridWay is configured with several Information Managers (IM). Grid administrators can prioritize resources based upon the IM that discovered the resource. Grid administrators can also assign priorities to individual resources. For example, a fixed priority policy can specify that resources from the intranet (managed by an IM driver tagged intranet) should always be used before resources from other sites (managed by an IM driver tagged grid).

The priority of a resource ranges from 01 (lowest priority) to 99 (highest priority), so resources with a higher priority will be used first. Grid administrators can also prioritize individual resources based on business decisions.

When a resource gets the priority value 00, it becomes a banned resource, and will not be used for any job. So Grid administrators can virtually unplug resources from their scheduling domain.

Example configuration for the resource Fixed Priority Policy:

     # Weight for the Resource fixed priority policy
     RP_WEIGHT = 1

     # Fixed priority values for specific resources
     RP_HOST[my.cluster.com]    = 12
     RP_HOST[slow.machine.com]  = 02

     # Fixed priority for every resource in the intranet
     RP_IM[intranet] = 65

     # Fixed priority for every resource discovered by the grid IM
     RP_IM[grid] = 05

     # Anyone else gets a default priority 04 (i.e. other IM)
     RP_IM[DEFAULT] = 01

The goal of this policy is to prioritize those resources more suitable for the job, from its own point of view. For example, the rank policy for a job can state that resources with faster CPUs should be used first. This policy is configured through the “RANK” attribute in the job template, please refer to GridWay 5.4: User's Guide.

Example configuration for the Rank policy:

      # Weight of the Rank policy
      RA_WEIGHT = 1

This policy reflects the behavior of Grid resources based on job execution statistics. So, crucial performance variables, like the average queue wait time or network transfer times, are considered when scheduling a job. This policy is derived from the sum of two contributions: history and current.

• History contribution. Execution statistics on a given period of time (for example, average values in the last 3 days). This information is obtained from the accounting database, so GridWay must be compiled with the Berkeley DB libraries.
• Last job contribution. Execution statistics of the last job on that resource.

These values are used to compute an estimated execution time of a job on a given resource for a given user:

T = ( 1 - w )( T^h_exe + T^h_xfr + T^h_que) + w ( T^c_exe + T^c_xfr + T^c_que )

where T^c are the execution statistics of the last job (execution, transfer and queue wait-time), T^h are the execution statistics based on the history data; and w is the history ratio. Those resources with a lower estimated time are used first to execute a job.

The Usage policy can be configured with:

• “UG_HISTORY_WINDOW”. Number of days used to compute the execution statistics from the History contribution.

• “UG_HISTORY_RATIO”. The value of w, use 0 to use only data from the accounting database, and 1 to use only results from the last execution.

Example configuration for Usage policy:

     # Weight of the Usage policy
     UG_WEIGHT = 1

     # Number of days in the history window
     UG_HISTORY_WINDOW = 3

     # Accounting database to last execution ratio
     UG_HISTORY_RATIO = 0.25

When a resource fails, GridWay implements an exponential linear back-off strategy at resource level (and per each user); henceforth, resources with persistent failures are discarded (for a given user).

In particular, when a failure occurs a resource is banned for T seconds:

T=T_∞ ( 1 - e^-Δt/C )

where T_∞ is the maximum time that a resource can be banned, Δt is the time since last failure, and C is a constant that determines how fast the T_∞ limit is reached.

The failure rate policy can be configured with the following parameters:

• “FR_MAX_BANNED_TIME”. The value of T_∞, use 0 to disable this policy.
• “FR_BANNED_C”. The value of the C constant in the above equation.

Example configuration for the Failure Rate policy:

      # Maximum time that a resource will not be used, in seconds
      FR_MAX_BANNED_TIME = 3600
      # Exponential constant
      FR_BANNED_C        = 650

The list of all candidate resources is sorted by the suitability priority, which is computed as a weighted sum of the contribution from the previous policies. The suitability priority resource is therefore computed with:

P_h = Σ_i w_i p_ih , where i={fixed, usage, rank}

where w_i is the weight for each policy (integer value) and p_ih is the priority (normalized) contribution from each policy.

Also, the scheduler can migrate running jobs in the following situations:

• A better resource is discovered.
• A job has been waiting in the remote queue system more than a given threshold.
• The application changes its requirements.
• A performance degradation is detected.

See Section “GridWay Daemon (GWD) Configuration” and GridWay 5.4: User's Guide, for information on configuring these policies.

The built-in scheduler configuration options are defined in “$GW_LOCATION/etc/sched.conf”. The table below summarizes the configuration file options, their description and default values. Note that blank lines and any character after a '#' are ignored.

Table 3. Built-in Scheduler Configuration File Options.

GridWay uses an external and selectable scheduler module to schedule jobs. The following schedulers are distributed with GridWay:

• Built-in Scheduler (default), which implements the above policies.
• Round-robin/flood Scheduler. This is a simple scheduling algorithm. It maximizes the number of jobs submitted to the Grid. Available resources are flooded with user jobs in a round-robin fashion.

The schedulers are configured with the “DM_SCHED” option in the “gwd.conf” file, with the format:

DM_SCHED = <sched_name>:<path_to_sched>:[args]

where:

• sched_name: is a tag to further refer to this scheduler.
• path_to_sched: is the name of the Scheduler executable. Use an absolute path or include the Scheduler executable directory in the PATH variable (such directory is “$GW_LOCATION/bin” by default).
• arg: Additional arguments to be passed to the Scheduler executable.

By default, GridWay is configured to use the built-in policy engine described in the previous sections. If for any reason you need to recover this configuration, add the following line to “$GW_LOCATION/etc/gwd.conf”:

DM_SCHED  = builtin:gw_sched:

Do not forget to adjust the scheduler policies to your needs by editing the “$GW_LOCATION/etc/sched.conf” file.

To configure the round-robin/flood scheduler, first disable the built-in engine policy in the “$GW_LOCATION/etc/sched.conf” configuration file by adding the following line:

DISABLE = yes

Then add the following line to “$GW_LOCATION/etc/gwd.conf”:

DM_SCHED = flood:gw_flood_scheduler:-h 10 -u 30 -c 5 -s 15

where:

• -h: The max number of jobs that the scheduler submits to a given host. Default value is 10; use 0 to dispatch to each host as many jobs as possible.
• -u: The maximum number of simultaneous running jobs per user. Default value is 30; use 0 to dispatch as many jobs as possible.
• -c: Scheduling Chunk. Jobs of the same user are submitted in a round-robin fashion with the given chunk. Default value is 5.
• -s: Dispatch Chunk. The maximum number of jobs that will be dispatched each scheduling action. Default value is 15; use 0 to dispatch as many jobs as possible.

GridWay uses several Middleware Access Drivers (MAD) to interface with different Grid services. The following MADs are part of the GridWay distribution:

• Execution Managers to interface with both pre-WS GRAM and WS GRAM services.
• Information Managers to interface with both MDS2 (MDS and GLUE schemas) and MDS4 services.
• Transfer Managers to interface with GridFTP servers.

These drivers are configured and selected via the GWD configuration interface described in Section “GridWay Daemon (GWD) Configuration”. Additionally you may need to configure your environment (see Chapter 4, Testing) in order to successfully load the MADs into the GWD core. To do so, you can also use global and per user environment configuration files (“gwrc”).

There is one global config file and per user configuration files that can be used to set environment variables for MADs. These files are standard shell scripts that are sourced into the MAD environment before it is loaded. It can be used, for example, to set the variable “X509_USER_PROXY” so you can have it located elsewhere instead of the standard place (“/tmp/x509_u<uid>”). Other variables can be set and you can even source other shell scripts, for instance, you can prepare another globus environment for MADs for some users, like this:

X509_USER_PROXY=$HOME/.globus/proxy.pem

 
GLOBUS_LOCATION=/opt/globus-4.0
. $GLOBUS_LOCATION/etc/globus-user-env.sh

The file for global MAD environment configuration is “$GW_LOCATION/etc/gridway/gwrc” and the user specific one is “$HOME/.gwrc”.

You have to take into account a couple of things:

• The global environment file is loaded before the user one, so the variables set by the user file take precedence over the global ones.
• The files are sourced so you need to export the variables to make them visible in the environment of the called MAD. Right now there is a mechanism so variables set as “VARIABLENAME=VALUE” are automatically exported (without spaces preceding the variable name). If you are sourcing other files or you put variables inside an indented block (for example, in an if statement) you have to explicitly export them. For example:

  if [ -d /opt/globus-devel ]; then
   export GLOBUS_LOCATION=/opt/globus-devel
  fi

The Execution Driver interfaces with Grid Execution Services and is responsible for low-level job execution and management. The GridWay distribution includes the following Execution MADs:

• Pre-WS GRAM (Globus Toolkit 2.4 and above)
• WS GRAM (Globus Toolkit 4.0)

Note that the use of these MADs requires a valid proxy.

Execution MADs are configured with the “EM_MAD” option in the “$GW_LOCATION/etc/gwd.conf” file, with the following format:

EM_MAD = <mad_name>:<path_to_mad>:<args>:<rsl|rsl_nsh|rsl2>

where:

• mad_name: is a tag to further refer to this Execution Driver, and it is also useful for logging purposes.
• path_to_mad: is the name of the Execution Driver executable, which must be placed in the “$GW_LOCATION/bin” directory.
• args: Parameters passed to the mad when it is executed.
• rsl|rsl_nsh|rsl2: Selects the language that GWD will use to describe job requests. It can be rsl (intended to be used with pre-WS drivers), rsl_nsh (intended to be used with pre-WS drivers over resources with non-shared home directories, like in LCG) and rsl2 (intended to be used with WS drivers).

For example, the following line will configure GridWay to use the Execution Driver gw_em_mad_prews using RSL syntax with name prews:

EM_MAD = prews:gw_em_mad_prews::rsl

To use WS-GRAM services, you can include the following line in your “$GW_LOCATION/etc/gwd.conf” file:

EM_MAD = ws:gw_em_mad_ws::rsl2

Now it is possible to specify a different gatekeeper port than the standard one (8443) in the Web Service driver. The line to configure EM MADs in “gwd.conf” has changed so you can add parameters to it. The parameter to change the port is the “-p” followed by the port number. For example:

EM_MAD = osg_ws:gw_em_mad_ws:-p 9443:rsl2

This line tells the EM MAD to use port 9443 to connect to the GT4 Gatekeeper.

The File Transfer Driver interfaces with Grid Data Management Services and is responsible for file staging, remote working directory set-up and remote host clean up. The GridWay distribution includes:

• GridFTP server (version 1.1.2 and above)
• Dummy Transfer driver (to be used with clusters without shared home)

The use of this driver requires a valid Proxy.

File Transfer Managers are configured with the TM_MAD option in the “gwd.conf” file, with the format:

TM_MAD = <mad_name>:<path_to_mad>:[arg]

where:

• mad_name: is a tag to further refer to this File Transfer Driver, and it is also useful for logging purposes.
• path_to_mad: is the name of the File Transfer Driver executable, which must be placed in the “$GW_LOCATION/bin” directory.
• arg: Additional arguments to be passed to the File Transfer executable.

To configure the Transfer Driver, add a line to “$GW_LOATION/etc/gwd.conf”, with the following format:

TM_MAD = <mad_name>:<path_to_mad>:[arguments]>

The GridFTP driver does not require any command line arguments. So to configure the driver, add the following line to “$GW_LOCATION/etc/gwd.conf”:

TM_MAD = gridftp:gw_tm_mad_ftp:

The name of the driver will be later used to specify the transfer mechanisms with Grid resource.

The Dummy driver should be used with those resources (clusters) which do not have a shared home. In this case, transfer and execution are performed as follows:

• The Dummy Transfer MAD performs data movements from the cluster worker node and the client using a reverse server model.
• The rsl_nsh RSL generation function is used, which transfers the wrapper along with its stdout and stderr streams.
• The wrapper executing in the worker node automatically transfers job.env and input/output files from the client.

The following servers can be configured to access files on the client machine:

• GASS Server, started for each user.
• GridFTP, specified by its URL running on the GridWay server.

The Dummy driver behavior is specified with the following command line arguments:

• -u <URL>:URL of the GridFTP server.
• -g: Use a user GASS server to transfer files.

Sample configuration to use a GridFTP server:

TM_MAD = dummy:gw_tm_mad_dummy:-u gsiftp\://hostname

Sample configuration to use GASS servers:

TM_MAD = dummy:gw_tm_mad_dummy:-g

The Information Driver interfaces with Grid Monitoring Services and is responsible for host discovery and monitoring. The following Information Drivers are included in GridWay:

• Static host information data
• MDS2 with MDS schema (Globus Toolkit 2.4)
• MDS2 with GLUE schema (Globus Toolkit 2.4 and LCG middleware)
• MDS4 (Globus Toolkit 4.0)

To configure an Information Driver, add a line to “$GW_LOATION/etc/gwd.conf”, with the following format:

IM_MAD = <mad_name>:<path_to_mad>:[args]:<tm_mad_name>:<em_mad_name>

where:

• mad_name: is a tag to further refer to this Information Driver.
• path_to_mad: is the name of the Information Driver executable. Use an absolute path or include the Information Driver directory in the “PATH” variable (such directory is “$GW_LOCATION/bin” by default).
• arg: Additional arguments to be passed to the Information Driver executable.
• tm_mad_name: File Transfer Driver to be used with the hosts managed by this Information Driver.
• em_mad_name: Execution Driver to be used with the hosts managed by this Information Driver.

For example, to configure GWD to access a MDS4 hierarchical information service:

IM_MAD = mds4:gw_im_mad_mds4:-s hydrus.dacya.ucm.es:gridftp:ws

All the Information Drivers provided with GridWay use a common interface to configure their operation mode. The arguments used by the Information Drivers are:

• -s <server>: The information server in a hierarchical configuration, i.e. MDS2 GIIS or MDS4 root IndexService.
• -l <host list>: A host list file to be used by GridWay, only relevant for static discovery and monitoring. See the Information Driver operation mode below (Relative path to $GW_LOCATION).
• -b <base>: The Virtual Organization name in the DN of the LDIF entries, i.e. the Mds-Vo-name attribute, only relevant for MDS2.
• -f <filter>: Additional requirements to be imposed on all the hosts managed by this Information Driver, in LDIF format.

These options allow you to configure your Information Drivers in the three operation modes, described below.

In this mode, hosts are statically discovered by reading a host list file (note: each time it is read). Also the attributes of each host are read from files. Hint: Use this mode for testing purposes and not in a production environment. To configure a Information Driver in SS mode use the host list option, for example:

IM_MAD = static:gw_im_mad_static:-l examples/im/host.static:gridftp:ws

The host list file contains one host per line, with format:

FQDN    attribute_file

where:

• FQDN: is the Full Qualified Domain Name of the host.
• attribute_file: is the name of the file with the static attributes of this host. Relative to the “GW_LOCATION” directory.

For example (you can find this file, “host.list”, in “$GW_LOCATION/examples/im/”)

     hydrus.dacya.ucm.es examples/im/hydrus.attr
     draco.dacya.ucm.es examples/im/draco.attr

The “attribute_file” includes a single line with the host information and other lines with the information of each queue (one line per queue). Use the examples below as templates for your hosts.

Example of attribute file for a PBS cluster (you can find this file in “$GW_LOCATION/examples/im/”):

    HOSTNAME="hydrus.dacya.ucm.es" ARCH="i686" OS_NAME="Linux" OS_VERSION="2.6.4"
    CPU_MODEL="Intel(R) Pentium(R) 4 CPU 2" CPU_MHZ=2539 CPU_FREE=098 CPU_SMP=1
    NODECOUNT=4 SIZE_MEM_MB=503 FREE_MEM_MB=188 SIZE_DISK_MB=55570
    FREE_DISK_MB=39193 FORK_NAME="jobmanager-fork" LRMS_NAME="jobmanager-pbs"
    LRMS_TYPE="pbs" QUEUE_NAME[0]="q4small" QUEUE_NODECOUNT[0]=1
    QUEUE_FREENODECOUNT[0]=4 QUEUE_MAXTIME[0]=0 QUEUE_MAXCPUTIME[0]=20
    QUEUE_MAXCOUNT[0]=4 QUEUE_MAXRUNNINGJOBS[0]=0 QUEUE_MAXJOBSINQUEUE[0]=1
    QUEUE_STATUS[0]="enabled" QUEUE_DISPATCHTYPE[0]="batch"
    QUEUE_NAME[1]="q4medium" QUEUE_NODECOUNT[1]=4 QUEUE_FREENODECOUNT[1]=4
    QUEUE_MAXTIME[1]=0 QUEUE_MAXCPUTIME[1]=120 QUEUE_MAXCOUNT[1]=4
    QUEUE_MAXRUNNINGJOBS[1]=0 QUEUE_MAXJOBSINQUEUE[1]=1
    QUEUE_STATUS[1]="enabled" QUEUE_DISPATCHTYPE[1]="batch"

Example of attribute file for a Fork Desktop (you can find this file in “$GW_LOCATION/examples/im/”):

    HOSTNAME="draco.dacya.ucm.es" ARCH="i686" OS_NAME="Linux" OS_VERSION="2.6-xen"
    CPU_MODEL="Intel(R) Pentium(R) 4 CPU 3" CPU_MHZ=3201 CPU_FREE=185 CPU_SMP=2
    NODECOUNT=2 SIZE_MEM_MB=431 FREE_MEM_MB=180 SIZE_DISK_MB=74312
    FREE_DISK_MB=40461 FORK_NAME="jobmanager-fork" LRMS_NAME="jobmanager-fork"
    LRMS_TYPE="fork" QUEUE_NAME[0]="default" QUEUE_NODECOUNT[0]=1
    QUEUE_FREENODECOUNT[0]=1 QUEUE_MAXTIME[0]=0 QUEUE_MAXCPUTIME[0]=0
    QUEUE_MAXCOUNT[0]=0 QUEUE_MAXRUNNINGJOBS[0]=0 QUEUE_MAXJOBSINQUEUE[0]=0
    QUEUE_STATUS[0]="0" QUEUE_DISPATCHTYPE[0]="Immediate"

To use the WS version of these files just change jobmanager-fork with Fork and jobmanager-pbs with PBS.

Hosts are discovered by reading a host list file. However, the information of each host is gathered by querying its information service (GRIS in MDS2 or the DefaultIndexService in MDS4). Hint: Use this mode if the resources in your Grid does not vary too much, i.e. resource are not added or removed very often. To configure an Information Driver in SD mode, use the host list option, for example:

IM_MAD = glue:gw_im_mad_mds2_glue:-l examples/im/host.list:gridftp:prews

In this case the host list file contains one host per line, with the format:

      FQDN
      ...
      FQDN

where:

• FQDN: is the Full Qualified Domain Name of the host.

For example (you can find this file in “$GW_LOCATION/examples/im/”)

      hydrus.dacya.ucm.es
      ursa.dacya.ucm.es
      draco.dacya.ucm.es

In this mode, hosts are discovered and monitored by directly accessing the Grid Information Service. Hint: Use this mode if the resources in your Grid does vary too much, i.e. resource are added or removed very often. To configure a Information Driver in SD mode, use the server option, for example:

IM_MAD = mds4:gw_im_mad_mds4:-s hydrus.dacya.ucm.es:gridftp:ws

If you are using an MDS2 information service you may need to specify the Virtual Organization name in the DN of the LDIF entries (Mds-vo-name) with the base option described above.

There is a way to specify a different machine to be used as gsiftp endpoint than the one that has the gatekeeper installed. This is useful when the CE machine does not have gsiftp server configured but there is another machine that works as a Storage Element. Right now, this information could be set statically but the rest of the information can be updated dynamically. To use this feature you have to create a file for each host you want to configure with extra information and another file with pairs of host and file name (as described above for the SS mode). The filename can be a full path or a relative path to “GW_LOCATION”. Then in the IM MAD you must specify the list file with “-l”, like this (in “gwd.conf”):

IM_MAD = mds4:gw_im_mad_mds4:-l etc/gridway/host.list:gridftp:ws

The file list should look like this:

     wsgram-host1.domain.com etc/gridway/wsgram-host1.attr
     wsgram-host2.domain.com etc/gridway/wsgram-host2.attr

And the attributes file for each node should look like this:

SE_HOSTNAME="gridftp-host1.domain.com"

GridWay architecture flexibility allows it to interoperate with grids based on different middleware stacks. The following documents states how to configure GridWay for the following infrastructures:

• Integration of GridWay within the EGEE infrastructure[HTML][PDF]
• Integration of GridWay within the OSG infrastructure[HTML][PDF]
• Integration of GridWay within the Teragrid infrastructure[HTML][PDF]
• Integration of GridWay within the Nordugrid infrastructure[HTML][PDF]
• Creating MADs: SSH Example[HTML][PDF]

In order to test the GridWay installation, login as your user account, in the single-mode installation, or as the “<gwadmin>” account, in the multiple-user installation, and follow the steps listed below:

1. Set up the environment variables “GW_LOCATION” and “PATH”:

   $ export GW_LOCATION=<path_to_GridWay_installation>
   $ export PATH=$PATH:$GW_LOCATION/bin

   or

   $ setenv GW_LOCATION <path_to_GridWay_installation>
   $ setenv PATH $PATH:$GW_LOCATION/bin

   depending on the shell you are using.
2. Generate a valid proxy

   $ grid-proxy-init
   Your identity: /O=Grid/OU=GRIDWAY/CN=GRIDWAY User
   Enter GRID pass phrase for this identity:
   Creating proxy ................................. Done
   Your proxy is valid until: Mon Oct 29 03:29:17 2005

3. Show the GridWay license:

   $ gwd -v
   Copyright 2002-2008 GridWay Team, Distributed Systems Architecture
   Group, Universidad Complutense de Madrid
   
   GridWay 5.4.0 is distributed and licensed for use under the terms of the
   Apache License, Version 2.0 (www.apache.org/licenses/LICENSE-2.0).

4. Start the GridWay daemon (GWD) (in multiple-mode add the -m option):

   $ gwd

5. Check the connection to GWD:

   $ gwps
   USER         JID DM   EM   START    END      EXEC    XFER    EXIT NAME            HOST
   $ gwhost
   HID PRIO  OS              ARCH   MHZ %CPU  MEM(F/T)     DISK(F/T)     N(U/F/T) LRMS                 HOSTNAME

6. Stop GWD:

   $ pkill gwd

To perform more sophisticated tests, check the GridWay 5.4: User's Guide. If you experience problems, check Troubleshooting.

GridWay is shipped with a test suite, available in the test directory. The test suite exercises different parts of GridWay, and can be used to track functionality bugs. However you need a working GridWay installation and testbed to execute the suite. Usage information is available with “gwtest -h”. Tests can be performed individually (using the test id) or all together automatically.

Table 4. GridWay tests description.

GridWay also ships with a DRMAA test suite, conceived to test the DRMAA Java implementations. Download and untar the following tarball, then follow the instructions found in the README file.

Access authorization to the GridWay server is done based on the Unix identity of the user (accessing GridWay directly or through a Web Services GRAM, as in GridGateWay). Hence, security in GridWay has the same implications as the Unix accounts of their users.

Also, GridWay uses proxy certificates to use Globus services, so the security implications of managing certificates also must be taken into account

• Lock file exists

GridWay finishes with the following message when you try to start it:

Error! Lock file <path_to_GridWay>/var/.lock exists.

Be sure that no other GWD is running, then remove the lock file and try again.

• Error in MAD initialization

GridWay finishes with the following message, when you try to start it:

Error in Execution MAD prews initialization, exiting. Check path,
you have a valid proxy...

Check that you have generated a valid proxy (for example with the grid-proxy-info command). Also, check that the directory “$GW_LOCATION/bin” is in your path, and the executable name of all the MADs is defined in “gwd.conf”.

• Error contacting GWD

Client commands, like gwps, finish with the message:

connect(): Connection refused
Could not connect to gwd

Be sure that GWD is running (ex. pgrep -l gwd). If it is running, check that you can connect to GWD (ex. telnet `cat $GW_LOCATION/var/gwd.port`)