WASD Web Services - Scripting

1 - Introduction

1.1 - Scripting Accounts 1.2 - Scripting Processes 1.2.1 - Process Management 1.2.2 - Detached Process Scripting 1.2.2.1 - Persona Scripting 1.2.2.2 - Restricting Persona Scripting 1.2.2.3 - Process Priorities 1.2.3 - Subprocess Scripting 1.2.4 - Script Process Default 1.2.5 - Script Process Parse Type 1.2.6 - Script Process Run-Down 1.2.7 - Client Recalcitrance 1.3 - Script Proctor 1.4 - Caching Script Output 1.5 - Enabling A Script 1.6 - Script Mapping 1.7 - Script Run-Time 1.8 - Unix Syntax 1.9 - Scripting Logicals 1.10 - Scripting Scratch Space 1.11 - DCL Processing of Requests 1.12 - Scripting Function Library 1.13 - Script-Requested, Server-Generated Error Responses
next previous contents full-page

This document is not a general tutorial on authoring scripts, CGI or any other. A large number of references in the popular computing press covers all aspects of this technology, usually quite comprehensively. The information here is about the specifics of scripting in the WASD environment, which is generally very much like any other implementation, VMS or otherwise (although there are always annoying idiosyncracies, see 1.12 - Scripting Function Library for a partial solution to smoothing out some of these wrinkles for VMS environments).

Scripts are mechanisms for creating simple Web applications or Web services, sending data to (and often receiving data from) a client, extending the capabilities of the basic HTTPd. Scripts execute in processes and accounts separate from the actual HTTP server but under its control and interacting with it.

By default WASD manages a script's process environment in an independent detached process created by the HTTP server or as a network process created using DECnet. By configuration the server will use subprocesses in place of detached.

WASD scripting can deployed in a number of environments. Other chapters cover the specifics of these. Don't become bewildered or be put off by all these apparent options, they are basically variations on a CGI theme.

2 - CGI
3 - CGIplus
4 - Run-Time Environments
5 - WebSocket
6 - CGI Callouts
7 - ISAPI
8 - DECnet & OSU
9 - Other Environments
10 - Request Redaction
11 - Raw TCP/IP Socket

1.1 - Scripting Accounts

It is strongly recommended to execute scripts in an account distinct from that executing the server. This minimises the risk of both unintentional and malicious interference with server operation through either Inter-Process Communication (IPC) or scripts manipulating files used by the server.

The default WASD installation creates two such accounts, with distinct UICs, usernames and default directory space. The UICs and home areas can be specified differently to the displayed defaults. Nothing should be assumed or read into the scripting account username - it's just a username.

Default Accounts
UsernameUICDefaultDescription
HTTP$SERVER[077,001]WASD_ROOT:[HTTP$SERVER]Server Account
HTTP$NOBODY[076,001]WASD_ROOT:[HTTP$NOBODY]Scripting Account

During startup the server checks for the existence of the default scripting account and automatically configures itself to use this for scripting. If it is not present it falls-back to using the server account. Other account names can be used if the startup procedures are modified accordingly. The default scripting username may be overridden using the /SCRIPT=AS=<username> qualifier (see WASD Web Services - Install and Config ). The default scripting account cannot be a member of the SYSTEM group and cannot have any privilege other than NETMBX and TMPMBX (Privileged User Scripting describes how to configure to allow this).

Scripting under a separate account is not available with subprocess scripting and is distinct from PERSONA scripting (even though it uses the same mechanism, see below).

1.2 - Scripting Processes

Process creation under the VMS operating system is notoriously slow and expensive. This is an inescapable overhead when scripting via child processes. An obvious strategy is to avoid, at least as much as possible, the creation of these processes. The only way to do this is to share processes between multiple scripts/requests, addressing the attendant complications of isolating potential interactions between requests. These could occur through changes made by any script to the process' enviroment. For VMS this involves symbol and logical name creation, and files opened at the DCL level. In reality few scripts need to make logical name changes and symbols are easily removed between uses. DCL-opened files are a little more problematic, but again, in reality most scripts doing file manipulation will be images.

A reasonable assumption is that for almost all environments scripts can quite safely share processes with great benefit to response latency and system impact (see "WASD VMS Web Services - Features and Facilities"; 11 - Server Performance WASD Web Services - Features and Facilities ) for a table with some comparative performances). If the local environment requires absolute script isolation for some reason then this process-persistance may easily be disabled with a consequent trade-off on performance.

The term zombie is used to describe processes when persisting between uses (the reason should be obvious, they are neither "alive" (processing a request) nor are they "dead" (deleted :^) Zombie processes have a finite time to exist (non-life-time?) before they are automatically run-down (see below). This keeps process clutter on the system to a minimum.

1.2.1 - Process Management

Scripting processes are created on-demand, within configuration limits and timeout periods. There are no arbitrary limits, only system resource limits, on the number of scripting processes. WASD_CONFIG_GLOBAL directives control the configuration limits of these (see "WASD VMS Web Services - Install and Config"; 8 - Global Configuration WASD Web Services - Install and Config ).

[DclHardLimit]  50
[DclSoftLimit]  40
[DclZombieLifeTime]  00:10:00
[DclCgiPlusLifeTime]  00:30:00
Other configuration directives are discussed later in this chapter.

Hard Limit

Scripting processes of all kinds (CGI, CGIplus and RTE) are created on-demand up until [DclHardLimit] is reached. If all scripting processes are busy with requests at that limit then the server provides a 503 (too busy) response.

Soft Limit

If there are more than [DclSoftLimit] scripting processes then the least-recently-used of any idle processes (those not currently processing a request) are proactively run-down until the soft-limit is reached. This provides head-room for the immediate creation of additional scripting processes for new requests that cannot be satisfied from currently instantiated processes. Soft-limit should of course be configured less than hard-limit (and if not WASD makes it that way).

Lifetimes

Idle scripting processes (those not having been given a request to process) are proactively run-down (see 1.2.6 - Script Process Run-Down) after configured periods.

[DclZombieLifeTime] specifies the period in minutes a CGI scripting process can remain idle.

[DclCgiPlusLifeTime] specifies the period a CGIplus script (inside a CGIplus scripting process) or a RTE process can remain idle.

If requests being serviced by scripts drop to zero for a period (governed by the above lifetimes) then eventually all scripting processes should be run-down leaving only the server process.

1.2.2 - Detached Process Scripting

The default is for WASD to execute scripts in detached processes created completely independently of the server process itself. This offers a significant number of advantages over spawned subprocesses

without too many disadvantages

Creation of a detached process is slightly more expensive in terms of system resources and initial invocation response latency (particularly if extensive login procedures are required), but this quickly becomes negligable as most script processes are used multiple times for successive scripts and/or requests.

Detached Process Management

With detached processes the server must explicitly ensure that each scripting process is removed from the system during server shutdown (with subprocesses the VMS executive provides this automatically). This is performed by the server exit handler. With VMS it is possible to bypass the exit handler (using a $DELPRC or the equivalent $STOP/ID= for instance), making it possible for "orphaned" scripting processes to remain - and potentially accumulate on the system!

To address this possibility the server scans the system for candidate processes during each startup. These are identified by a terminal mailbox (SYS$COMMAND device), and then further that the mailbox has an ACL with two entries; the first identifying itself as a WASD HTTPd mailbox and the second allowing access to the account the script is being executed under. Such a device ACL looks like the following example.

Device MBA335:, device type local memory mailbox, is online, record-oriented
  device, shareable, mailbox device.

  Error count                    0    Operations completed                  0
  Owner process                 ""    Owner UIC             [WEB,HTTP$NOBODY]
  Owner process ID        00000000    Dev Prot              S:RWPL,O:RWPL,G,W
  Reference count                1    Default buffer size                2048
  Device access control list:
    (IDENTIFIER=WASD_HTTPD_80,ACCESS=NONE)
    (IDENTIFIER=[WEB,HTTP$NOBODY],ACCESS=READ+WRITE+PHYSICAL+LOGICAL)

This rights identifier is generated from the server process name and is therefore system-unique (so multiple autonomous servers will not accidentally cleanup the script processes of others), and is created during server startup if it does not already exist. For example, if the process name was "HTTPd:80" (the default for a standard service) the rights identifier name would be "WASD_HTTPD_80" (as shown in the example above).

SYLOGIN and LOGIN Procedures

Detached scripting processes are created through the full "LOGINOUT" life-cycle and execute all system and account LOGIN procedures. Although immune to the effects of most actions within these procedures, and absorbing any output generated during this phase of the process life-cycle, some consideration should be given to minimising the LOGIN procedure paths. This can noticably reduce initial script latency on less powerful platforms.

The usual recommendations for non-interactive LOGIN procedures apply for script environments as well. Avoid interactive-only commands and reduce unnecessary interactive process environment setup. This is usually accomplished though code structures such as the following

$ IF F$MODE() .EQS. "INTERACTIVE"
$ THEN
     ...
$ ENDIF

$ IF F$MODE() .NES. "INTERACTIVE" THEN EXIT

WASD scripting processes can be specifically detected using DCL tests similar to the following. This checks the mode, that standard output is a mailbox, and the process name. These are fairly reliable (but not absolutely infallible) indicators.

$ IF F$MODE() .NES. "INTERACTIVE" .AND. -
     F$GETDVI("SYS$OUTPUT","MBX") .AND. -
     F$EXTRACT(0,4,F$PROCESS()) .EQS. "WASD" .AND. -
     F$EXTRACT(5,1,F$PROCESS()) .EQS. ":" .AND. -
     F$ELEMENT(1,"-",F$PROCESS()) .NES. "-"
$ THEN
$!   WASD scripting process!
     ...
$ ENDIF

1.2.2.1 - Persona Scripting

There are advantages in running a script under a non-server account. The most obvious of these is the security isolation it offers with respect to the rest of the Web and server environment. It also means that the server account does not need to be resourced especially for any particularly demanding application.

Enabling Persona Scripting

The $PERSONA functionality must be explicitly enabled at server startup using the /PERSONA qualifier ( "Technical Overview, Server Account and Environment"). The ability for the server to be able to execute scripts under any user account is a very powerful (and potentially dangerous) capability, and so is designed that the site administrator must explicitly and deliberately enable the functionality. Configuration files need to be rigorously protected against unauthorized modification.

A specific script or directory of scripts can be designated for execution under a specified account using the WASD_CONFIG_MAP configuration file set script=as= mapping rule. The following example illustrates the essentials.

# one script to be executed under the account
SET  /cgi-bin/a_big_script*  script=as=BIG_ACCOUNT
# all scripts in this area to be executed under this account
SET  /database-bin/*  script=as=DBACCNT

Access to package scripting directories (e.g. WASD_ROOT:[CGI-BIN]) is controlled by ACLs and possession of the rights identifier WASD_HTTP_NOBODY. If a non-server account requires access to these areas it too will need to be granted this identifier.

User Account Scripting

In some situations it may be desirable to allow the average Web user to experiment with or implement scripts. If the set script=as= mapping rule specifies a tilde character then for a user request the mapped SYSUAF username is substituted. Note that this requires the script to be colocated with the user account Web location and that the script is run under that account.

The following example shows the essentials of setting up a user environment where access to a subdirectory in the user's home directory, [.WWW] with script's located in a subdirectory of that, [.WWW.CGI-BIN].

SET   /~*/www/cgi-bin/*  script=AS=~
UXEC  /~*/cgi-bin/*  /*/www/cgi-bin/*
USER  /~*/*  /*/www/*
REDIRECT  /~*  /~*/
PASS  /~*/*  /dka0/users/*/*
To enable user CGIplus scripting include something like
UXEC+  /~*/cgiplus-bin/*  /*/www/cgi-bin/*

Where the site administrator has less than full control of the scripting environment it may be prudent to put some constraints on the quantity of resource that potentially can be consumed by non-core or errant scripting. The following WASD_CONFIG_MAP rule allows the "maximum" CPU time consumed by a single script to be constrained.

SET   /cgi-bin/cgi_process  script=CPU=00:00:05

Note that this is on a per-script basis, contrasted to the sort of limit a CPULM-type constraint would place on a scripting process.

The following WASD_CONFIG_GLOBAL rule specifies at which priority the scripting process executes. This can be used to provide the server and its infrastructure an advantage over user scripts.

[DclDetachProcessPriority]  1,2
See 1.2.2.3 - Process Priorities for further detail.

Authenticated User Scripting

If the set script=as= mapping rule specifies a dollar then a request that has been SYSUAF authenticated has the SYSUAF username substituted. Note that the script itself can be located anywhere provided the user account has read and/or execute access to the area and file.

SET   /cgi-bin/cgi_process  script=AS=$

If the script has not been subject to SYSUAF authorization then this causes the script activation to fail. To allow authenticated requests to be executed under the corresponding VMS account and non-authenticated requests to script as the usual server/scripting account use the following variant.

SET   /cgi-bin/cgi_process  script=AS=$?

If the server startup included /PERSONA=AUTHORIZED then only requests that have been subject to HTTP authorization and authentication are allowed to script under non-server accounts.

Privileged User Scripting

By default a privileged account cannot be used for scripting. This is done to reduce the chance of unintended capabilities when executing scripts. With additional configuration it is possible to use such accounts. Great care should be exercised when undertaking this.

To allow the server to activate a script using a privileged account the keyword /PERSONA=RELAXED must be used with the persona startup qualifier. If the keywords /PERSONA=RELAXED=AUTHORIZED are used then privileged accounts are allowed for scripting but only if the request has been subject to HTTP authorization and authentication.

1.2.2.2 - Restricting Persona Scripting

By default, activating the /PERSONA server startup qualifier allows all the modes described above to be deployed using appropriate mapping rules. Of course there may be circumstances where such broad capabilities are inappropriate or otherwise undesirable. It is possible to control which user accounts are able to be used in this fashion with a rights identifier. Only those accounts granted the identifier can have scripts activated under them. This means all accounts ... including the server account!

Recommendation

The simplest solution might appear to be to just grant all required accounts the WASD_HTTP_NOBODY identifier described above. While this is certainly possible it does provide read access to all parts of the server package this identifier controls, and write access to the WASD_ROOT:[SCRATCH] default file scratch space (1.10 - Scripting Scratch Space). If scripting outside of the site administrator's control is being deployed it may be better to create a separate identifier as just described.


This is enabled by specifying the name of a rights identifier as a parameter to the /PERSONA qualifier. This may be any identifier but the one shown in the following example is probably as good as any.

$ HTTPD /PERSONA=WASD_SCRIPTING

This identifier could be created using the following commands

$ SET DEFAULT SYS$SYSTEM
$ MCR AUTHORIZE
UAF> ADD /IDENTIFIER WASD_SCRIPTING
and granted to accounts using
UAF> GRANT /IDENTIFIER WASD_SCRIPTING HTTP$NOBODY

Meaningful combinations of startup parameters are possible:

/PERSONA=(RELAXED)
/PERSONA=(RELAXED=AUTHORIZED)
/PERSONA=(AUTHORIZED,RELAXED)
/PERSONA=(ident-name,RELAXED)
/PERSONA=(ident-name,AUTHORIZED,RELAXED)
/PERSONA=(ident-name,RELAXED=AUTHORIZED)

1.2.2.3 - Process Priorities

When detached processes are created they can be assigned differing priorities depending on the origin and purpose. The objective is to give the server process a slight advantage when competing with scripts for system resources. This allows the server to respond to new requests more quickly (reducing latency) even if a script may then take some time to complete the request.

The allocation of base process priorities is determined from the WASD_CONFIG_GLOBAL [DclDetachProcessPriority] configuration directive, which takes one or two (comma-separated) integers that determine how many priorities lower than the server scripting processes are created. The first integer determines server processes. A second, if supplied, determines user scripts. User scripts may never be a higher priority that server scripts. The following provides example directives.

[DclDetachProcessPriority]  1
[DclDetachProcessPriority]  0,1
[DclDetachProcessPriority]  1,2

Scripts executed under the server account, or those created using a mapped username (i.e. "script=as=username"), have a process priority set by the first/only integer.

Scripts activated from user mappings (i.e. "script=as=~" or "script=as=$") have a process priority set by any second integer, or fall back to the priority of the first/only integer.

1.2.3 - Subprocess Scripting

The WASD_CONFIG_GLOBAL directive [DclDetachProcess] can be used to disable the default detached process scripting.

[DclDetachProcess]  disabled
Note that other server configuration such as /PERSONA and/or /SCRIPT=AS= overrides this directive and so must also be disabled before subprocess scripting can be used.
Subprocess Scripting is Not Recommended

This section is included mainly for historical reference. There are so many advantages to detached process scripting and so many considerations with subprocess scripting that detached scripting is basically a "no-brainer" for production environments.


When the subprocess is spawned by the server none of the parent's environment is propagated. Hence the subprocess has no symbols, logical names, etc., created by the site's SYLOGIN.COM, the server account's LOGIN.COM, etc. This is done quite deliberately to provide a pristine and standard default environment for the script's execution. For this reason all scripts must provide all of their required environment to operate. In particular, if a verb is made available via a SY/LOGIN.COM this will not be available to the script. Verbs available via the DCLTABLES.EXE or DCL$PATH of course will be available.

There are two basic methods for supplying a script with a required environment.

With persistent subprocess scripting the pooled-resource BYTLM can become a particular issue. After the first subprocess-based script is executed the WATCH report provides some information on the BYTLM required to support both the desired number of incoming network connections and script subprocess IPC mailboxes. When using these numbers to resource the BYTLM quota of the server account keep in mind that as well as server-subprocess IPC consumption of BYTLM there may be additional requirements whatever processing is performed by the script.

For a standard configuration 15,000 bytes should be allowed for each possible script subprocess, 1,000 bytes for each potential client network connection, an additional 20,000 bytes overhead, plus any additional requirements for script processing, etc. Hence for a maximum of 30 scripts and 100 network clients, a BYTLM of approximately 260,000 minimum should be allowed.

When scripts are executed within unprivileged subprocesses created by the HTTP server, the processes are owned by the HTTP server account (HTTP$SERVER). Script actions could potentially affect server behaviour. For example it is possible for subprocesses to create or modify logical name values in the JOB table (e.g. change the value of LNM$FILE_DEV altering the logical search path). Obviously these types of actions are undesirable. In addition scripts can access any WORLD-readable and modify any WORLD-writable resource in the system/cluster, opening a window for information leakage or mischievous/malicious actions (some might argue that anyone with important WORLD-accessible resources on their system deserves all that happens to them - but we know they're out there :^) Script authors should be aware of any potential side-effects of their scripts and Web administrators vigilant against possible malicious behaviours of scripts they do not author.

1.2.4 - Script Process Default

For standard CGI and CGIplus script the script process' default device and directory is established using a SET DEFAULT command immediately before activating the script. This default is derived from the script file specification.

An alternative default location may be specified using the mapping rule shown in the following example.

set /cgi-bin/this-script* script=default=WEB:[THIS-SCRIPT]

The default may be specified in VMS or Unix file system syntax as appropriate. If in Unix syntax (beginning with a forward-slash) no SET DEFAULT is performed using DCL. The script itself must access this value using the SCRIPT_DEFAULT CGI variable and perform a chdir().

1.2.5 - Script Process Parse Type

On platforms where the Extended File Specification (EFS) is supported a SET PROCESS /PARSE=EXTENDED or SET PROCESS /PARSE=TRADITIONAL is executed by the scripting process before script activation depending on whether the script path is located on an ODS-2 or ODS-5 volume.

1.2.6 - Script Process Run-Down

The server can stop a script process at any point, although this is generally done at a time and in such a way as to eliminate any disruption to request processing. Reasons for the server running-down a script process.

In running down a script process the server must both update its own internal data structures as well as manage the run-down of the script process environment and script process itself. These are the steps.

  1. Exit handling.
  2. Input and output to all of the process' streams is cancelled. For scripts that may still be still processing this can result in I/O stream errors. The server waits for all queued I/O to disappear.
  3. If the script process has not already deleted itself the server issues a $DELPRC against it.
  4. The server receives the process termination AST and this completes the process run-down sequence.

1.2.7 - Client Recalcitrance

If a client disconnects from a running script (by hitting the browser Stop button, or selecting another hyperlink) the loss of network connectivity is detected by the server at the next output write.

Generally it is necessary for there to be some mechanism for a client to stop long-running (and presumably resource consuming) scripts. Network disconnection is the only viable one. Experience would indicate however that most scripts are short running and most disconnections are due to clients changing their minds about waiting for a page to build or having seen the page superstructure moving on to something else.

With these considerations in mind there is significiant benefit in not running-down a script immediately the client disconnection is detected. A short wait will result in most scripts completing their output elegantly (the script itself unaware the output is not being transmitted on to the client), and in the case of persistent scripts remaining available for the next request, or for standard CGI the process remaining for use in the next CGI script.

The period allowing the script to complete its processing may be set using the WASD_CONFIG_GLOBAL configuration directive [DclBitBucketTimeout]. It should be set to say fifteen seconds, or whatever is appropriate to the local site.

[DclBitBucketTimeout]  00:00:15

NB. "Bit-bucket" is a common term for the place discarded data is stored. :^)

1.3 - Script Proctor

Script proctoring proactively creates and maintains the specified minimum number of scripting processes, configured persistent scripts, and scripting environments (RTEs). It is primarily intended for those environments that have significant startup latency but can also be used to maintain idle scripting processes ready for immediate use.

The script proctor initially instantiates configured items during server startup and before enabling request acceptance and processing.

Then during subsequent request processing, at each scripting process run-down it scans current DCL task list counting the number of instances of each configured item. The proctor facility can differentiate between idle and active instances of the script/RTE and will optionally maintain a specified number of idle processes in addition to any currently active. If fewer than the configured requirement(s) one or more new processes are instantiated.

It is possible (and probably likely) that a proctored script specification will at some stage fail to activate the script (activation specification error, script unavailable for some reason, etc.) which would lead to a runaway series of attempts to proctor with each process exit. To help avoid this situation proctored processes that exit before successfully completing initial startup are quickly suppressed from further proctoring action. This suppression then more slowly times out, again allowing proctoring for that item.

Proctored scripts and RTEs contain nothing of the usual request-related environment. No CGI variables to speak of, no service, no request method, nothing! This means that rules used for proctor activations must be outside all virtual service conditionals (i.e. outside of any specific [[service:port]] in the rules, can be inside [[*:*]]) and anything else that may be dependent on a request characteristic.

The easiest way for a script to detect if its been proctored into existence is to look for the absence of this or these. No REQUEST_METHOD is a fair indicator as it should exist with all "real" requests. Of course a proctored script is really just there to instantiate itself, not to do anything directly productive, and so a script/RTE can just recognise this and conclude with perhaps a 204 HTTP status (no content) and then remain quiescent (awaiting its first actual request). Any and all output from a proctored script goes to the bit-bucket.

Once proctored into existance the script process is then subject to the normal scripting process management and (for example) if idle for a period exceeding a lifetime value will be procactively removed. Of course, during that process rundown the proctor facility will effectively replace it with a new instance, maintaining the overall requirement.

The Server Admin, DCL Report includes a Proctor List with the currently configured proctor items and associated statistics.

Proctored script activation can be WATCHed just like any other script activation using the [x]CGI and [x]DCL items. To explicitly trigger such an event merely $STOP/ID=pid a proctored scripting process.

Proctor Configuration

Proctor global configuration is introduced with the WASD_CONFIG_GLOBAL [DclScriptProctor] item with each following line representing one script/RTE to be proctored. Each line contains three mandatory and one optional, space-separated components.

integer[+integer] identification activation notepad
which are, in order
  1. the minimum integer number of instances of the item
    plus an optional minimum integer number of idle instances
  2. an identification string used to match already running instances of the item
  3. the activation path that can be used to activate the item
  4. an optional string that is passed to the mapping notepad facility

The zombie form is

integer * [activation]
where the specified number of idle processes is maintained.

The minimum plus any idle requirement cannot exceed the [DclSoftLimit] configuration value (in order to minimise potential process thrashing).

The proctor facility works by matching the identification string to the script paths as present in the DCL task list (and as presented in the Server Admin, DCL Report). So it needs to contain something unique to that script or environment and often contains a wildcard specification.

The activation path used to activate the script/RTE is the same as if it was activated via a scripting request.

For an RTE the activation script specification does not actually need to exist. It must contain a minimum path to activate the desired environment but the script itself is not checked to exist by WASD and does not need to exist. If it does then it should do whatever is required to instantiate its required environment and return a 204 (no content) response. If it does not exist then the RTE itself should detect it's a proctored activation and return a default 204 response itself, instantiating only the RTE environment.

Remember

Rules for mapping proctored scripts and RTEs must be outside of any request-dependent conditionals including specific virtual services.


Proctored scripts can be detected during mapping using

if (request-method:)
or
if (!request-method:%)
(i.e. no request method) and specific data passed using the optional notepad string (see "WASD VMS Web Services - Install and Config"; 7.3.1 - Notepad: Keyword WASD Web Services - Install and Config ) and then conditionally processed using something like
if (notepad:blah)  

Specific information can also be passed to the proctored script during mapping using such conditional processing in concert with the SET

script=param=name=value
mapping rule. This appears as a [WWW_]NAME CGI variable containing the value specified. Proctored scripts could then act according to any such data.

The combination of these allows some control of proctored scripting.

A proctor item with a minimum (and optionally idle) value of zero can be specified as a place-marker; the facility ignores zero valued items.

Proctor Example

This example illustrates a number of non-trivial proctoring scenarios. Only configuration items directly involved in the proctoring are shown; others would be involved in the general web-server infrastructure.

# WASD_CONFIG_GLOBAL
[DclScriptProctor]
2 /cgiplus-bin/mgd* /cgiplus-bin/mgd proctor=daniel
2 /apps/script /apps/script anyoldname=dothis
3+1 (*pyrte*)* /py-bin/proctor.py
2 *
3 * proctor=daniel

The [DclScriptProctor] contains five items. The first two specify that two scripts each be maintained, the third specifies four, the final two maintain zombie processes. The mapping rules (below) contain a conditional detecting the absence of a REQUEST_METHOD and processing the proctored scripts inside that decision structure. Proctor-specific mapping rules tend to be used only to supplement otherwise fundamental (but in this case proctored) scripting.

# WASD_CONFIG_MAP
if (request-method:)
   if (notepad:proctor=daniel) set * script=as=daniel
   if (notepad:anyoldname=dothis) set * script=param=DOTHIS=one
   # not a real script of course!
   script proctor=daniel proctor=daniel script=as=daniel
endif
Each of the five items explained in order:
  1. Matches the CGIplus script "/cgiplus-bin/mgd" and the trailing wildcard any supplementary path provided to that script. The activation path is a straight-forward scripting path. An optional notepad datum is passed to the mapping facility. In the mapping rules the notepad datum supplied is detected ("if (notepad:proctor=daniel)") and the username under which the script is to be executed specified. A minimum of two instances of this script are maintained.
  2. Matches the script "/apps/script" without trailing wildcard (as it is - a hypothetical - never used with a supplementary path). The activation path is again the straight-forward scripting path. An optional notepad datum is also passed from the proctor configuration to mapping which specifically detects it and set as CGI variable name and value that can subsequently be detected and acted upon by the proctored script. A minimum of two instances of this script are maintained.
  3. Maintains a scripting environment (RTE) and is therefore a little less straight-forward. The intention is to maintain a minimum number of Python RTEs (a rather expensive-to-instantiate interpreter). The matching string is more focused on the underlying RTE. The RTE is not obvious in the activation path (as all RTE mapping is transparent to the script path). The RTE is the environment of interest though and so is the matching string of interest; "(*pyrte*)*", where the parentheses indicate an underlying RTE, the wildcards delimit the RTE name of interest, and the trailing wildcard matches any current script that the RTE may be executing. The Server Admin menu, DCL Report can be used to gain insight into any script or scripting environment strings to be matched. In this third case there is no supplementary mapping required. A minimum of three instances of this RTE are maintained at least one of which must be idle or an additional instance will be created.
  4. Maintains two idle scripting processes (under the default scripting account) available for scripting use without the latency of process creation.
  5. Maintains three idle scripting processes with an associated mapping rule to activate them using the specified username. The activation string is arbitrary, should be unique, and is the "path" when being mapped. A notepad string can be specified.

1.4 - Caching Script Output

The WASD cache was originally provided to reduce file-system access (a somewhat expensive activity under VMS). With the expansion in the use of dynamically generated page content (e.g. PHP, Perl, Python) there is an obvious need to reduce the system impact of some of these activities. While many such responses have content specific to the individual request a large number are also generated as general site pages, perhaps with simple time or date components, or other periodic information. Non-file caching is intended for this type of dynamic content.

Revalidation of non-file content is difficult to implement for a number of reasons, both by the server and by the scripts, and so is not provided. Instead the cache entry is flushed on expiry of the [CacheValidateSeconds], or as otherwise specified by path mapping, and the request is serviced by the content source (script, PHP, Perl, etc.) with the generated response being freshly cached. Browser requests specifying no-caching are honoured (within server configuration parameters) and will flush the entry, resulting in the content being reloaded.

Controlling Script Caching

Determining which script content is to be cached and which not, and how long before flushing, is done using mapping rules (described in detail in the "Technical Overview"). The source of script cache content is specified using one or a combination of the following SET rules against general or specific paths in WASD_CONFIG_MAP. All mapping rules (script and non-script) are described here to put the script oriented ones into context. Those specific to script output caching are noted.

cache=[no]cgi from Common Gateway Interface (CGI) responses (for script output)
cache=[no]file from the file system (default and pre-8.4 cache behaviour)
cache=[no]net caches the full data stream irrespective of the source
cache=[no]nph full stream from Non-Parse Header (NPH) response (for script output)
cache=[no]query cache requests with query strings (use with care)
cache=[no]script both CGI and NPH responses (for script output)
cache=[no]ssi from Server-Side Includes (SSI) documents

A good understanding of site requirements and dynamic content sources, along with considerable care in specifying cache path SETings, is required to cache dynamic content effectively. It is especially important to get the content revalidation period appropriate to the content of the pages. This is specified using the following path SETings.

cache=expires=0 cancels any expiry
cache=expires=DAY expires when the day changes
cache=expires=HOUR when the hour changes
cache=expires=MINUTE when the minute changes
cache=expires=<hh:mm:ss> expires after the specified period in the cache

Examples

To cache the content of PHP-generated home pages that contain a time-of-day clock, resolving down to the minute, would require a mapping rule similar to the following.

set /**/index.php cache=cgi cache=expires=minute

To prevent requests from flushing a particular scripts output (say the main page of a site) using no-cache fields until the server determines that it needs reloading use the cache guard period.

set /index.py cache=script cache=expires=hour cache=guard=01:00:00

1.5 - Enabling A Script

By default the server accesses scripts using the search list logical name CGI-BIN, although this can be significantly changed using mapping rules. CGI-BIN is defined to first search WASD_ROOT:[CGI-BIN] and then WASD_ROOT:[AXP-BIN], WASD_ROOT:[IA64-BIN], or WASD_ROOT:[VAX-BIN] depending on the platform. [CGI-BIN] is intended for architecture-neutral script files (.CLASS., COM, .PL, .PY, etc.) and the architecture specific directories for executables (.EXE, .DLL, etc.)

These directories are delivered empty and it is up to the site to populate them with the desired scripts. A script is made available by copying its file(s) into the appropriate directory. By default ACLs will be propagated to allow access by the default scripting account. Scripts can be made unavailable by deleting them from these directories.

NOTE

It is good security practice to deploy only those scripts a site is actually using. This minimises vulnerability by simply reducing the number of possibly problematic scripts. A periodic audit of script directories is a good policy.

WASD script executables are built into the WASD_ROOT:[AXP], WASD_ROOT:[IA64] or WASD_ROOT:[VAX] directories depending on the architecture. Other script files, such as DCL procedures, Perl examples, Java class examples, etc. are located in other directories in the WASD_ROOT:[SRC] tree. The procedure WASD_ROOT:[INSTALL]SCRIPTS.COM assists in the installation or deinstallation of groups of WASD scripts.

1.6 - Script Mapping

Scripts are enabled using the exec/uxec or script rules in the mapping file (also see "Technical Overview, Mapping Rules"). The script portion of the result must be a URL equivalent of the physical VMS procedure or executable specification.

All files in a directory may be mapped as scripts using the exec rule. For instance, in the WASD_CONFIG_MAP configuration file can be found a rule

exec /cgi-bin/* /cgi-bin/*
which results in request paths beginning "/cgi-bin/" having the following path component mapped as a script. Hence a path "/cgi-bin/cgi_symbols.com" will result in the server attempting to execute a file named CGI-BIN:[000000]CGI_SYMBOLS.COM.

Multiple such paths may be designated as executable, with their contents expected to be scripts, either directly executable by VMS (e.g. .EXEs and .COMs) or processable by a designated interpreter, etc., (e.g. .PLs, .CLASSes) (1.7 - Script Run-Time).

In addition individual files may be specified as scripts. This is done using the script rule. In the following example the request path "/help" activates the "Conan The Librarian" script.

script /help* /cgi-bin/conan*

Of course, multiple such rules may be used to map such abbreviated or self-explanatory script paths to the actual script providing the application.

Mapping Local or Third-Party Scripts

It is not necessary to move/copy scripts into the server directory structure to make them accessible. In fact there are probably good reasons for not doing so! For instance, it keeps a package together so that at the next upgrade there is no possibility of the "server-instance" of that application being overlooked.

To make scripts provided by third party packages available for server activation three requirements must be met.

  1. The server account (HTTP$SERVER by default) must have read and execute access to the directory containing the scripts. Script files are searched for by the server before activation is attempted. This can be enabled using the SECHAN utility (see "Technical Overview").
    $ SECHAN /ASIF=CGI-BIN device:[directory]script-directory.DIR
    
  2. The scripting account (HTTP$NOBODY by default) must have read and execute access to any and all images and other resources required to use the application. There may be some consideration of file protections required when multiple accessors need to be accomodated (e.g. scripting and application accounts) so a specific solution may be required. If only the scripting account requires read access then the SECHAN utility could again be used to provide that to the directory (or directories) and contained files.
    $ SECHAN /ASIF=CGI-BIN device:[000000]directory.DIR
    $ SECHAN /ASIF=CGI-BIN device:[directory]*.*
    
  3. Mapping rules must exist to make the script and any required resources accessible.

Most packages having such an interface for Web server access would provide details on mapping into the package directory. For illustration the following mapping rules provide access to a package's scripts (assuming it provides more than one) and also into a documentation area.

The hypothetical "Application X" directory locations are

APPLICATIONX_ROOT:[DOC]
APPLICATIONX_ROOT:[CGI-BIN]

The required mapping rules would be

pass /applicationX/* /applicationX_root/docs/*
exec /appX-bin/* /applicationX_root/cgi-bin/*

Access to X's scripts would be using a path such as

http://the.host.name/appx-bin/main_script?plus=some&query=string
NOTE

When allowing the server and scripting account access into parts of the file system outside of the WASD package it is recommended to control the environment very carefully. Third-party scripting areas in particular should be modelled on those present in the package itself. The SECHAN utility described in the "Technical Overview" may be of some assistance with this.

"Wrapping" Local or Third-Party Scripts

Sometimes it may be necessary to provide a particular non-WASD, local, or third-party script with a particular environment in which to execute. This can be provided by wrapping the script executable or interpreted script in a DCL procedure (of course, if the local or third-party script is already activated by a DCL procedure, then that may need to be directly modified). Simply create a DCL procedure, in the same directory as the script executable, containing the required environmental commands.

For example, the following DCL procedure defines a scratch directory and provides the location of the configuration file. It is assumed the script executable is APPLICATIONX_ROOT:[CGI-BIN]APPX.EXE and the script wrapper APPLICATIONX_ROOT:[CGI-BIN]APPX.COM.

$! wrapper for APPX CGI executable
$ SET DEFAULT APPLICATIONX_ROOT:[000000]
$ DEFINE /USER SYS$SCRATCH APPLICATIONX_ROOT:[SCRATCH]
$ APPX == "$APPLICATIONX_ROOT:[CGI-BIN]APPX"
$ APPX /CONFIG=APPLICATIONX_ROOT:[CONFIG]APPX.CONF

1.7 - Script Run-Time

A script is merely an executed or interpreted file. Although by default VMS executables and DCL procedures can be used as scripts, other environments may also be configured. For example, scripts written for the Perl language may be transparently given to the Perl interpreter in a script process. This type of script activation is based on a unique file type (extension following the file name), for the Perl example this is most commonly ".PL", or sometimes ".CGI". Both of these may be configured to automatically invoke the site's Perl interpreter, or any other for that matter.

This configuration is performed using the WASD_CONFIG_GLOBAL [DclScriptRunTime] directive, where a file type is associated with a run-time interpreter. This parameter takes two components, the file extension and the run-time verb. The verb may be specified as a simple, globally-accessible verb (e.g. one embedded in the CLI tables), or in the format to construct a foreign-verb, providing reasonable versatility. Run-time parameters may also be appended to the verb if desired. The server ensures the verb is foreign-assigned if necessary, then used on a command line with the script file name as the final parameter to it.

The following is an example showing a Perl interpreter being specified. The first line assumes the "Perl" verb is globally accessible on the system (e.g. perhaps provided by the DCL$PATH logical) while the second (for the sake of illustration) shows the same Perl interpreter being configured for a different file type using the foreign verb syntax.

[DclScriptRunTime]
.PL PERL
.CGI $PERL_EXE:PERL

A file contain a Perl script then may be activated merely by specifying a path such as the following

/cgi-bin/example.pl

To add any required parameters just append them to the verb specified.

[DclScriptRunTime]
.XYZ XYZ_INTERPRETER -vms -verbose -etc
.XYZ $XYZ_EXE:XYZ_INTERPRETER /vms /verbose /etc

If a more complex run-time interpreter is required it may be necessary to wrap the script's execution in a DCL procedure.

Script File Extensions

The WASD server does not require a file type (extension) to be explicitly provided when activating a script. This can help hide the implementation detail of any script. If the script path does not contain a file type the server searches the script location for a file with one of the known file types, first ".COM" for a DCL procedure, then ".EXE" for an executable, then any file types specified using script run-time configuration directive, in the order specified.

For instance, the script activated in the Perl example above could have been specified as below and (provided there was no "EXAMPLE.COM" or "EXAMPLE.EXE" in the search) the same script would have been executed.

/cgi-bin/example

1.8 - Unix Syntax

CGI environment variables SCRIPT_FILENAME and PATH_TRANSLATED can be provided to any script (CGI, CGIplus, RTE) in Unix file-system syntax should that script require or prefer it using this format.

The path mapping rule "SET script=syntax=unix" changes the default syntax from VMS to Unix file-system. For example; by default using the URL

/cgi-bin/cgi_symbols/wasd_root/src/
would provide the request CGI data
WWW_PATH_INFO == "/wasd_root/src/"
WWW_PATH_TRANSLATED == "WASD_ROOT:[SRC]"
WWW_REQUEST_URI == "/cgi-bin/cgi_symbols/wasd_root/src/"
WWW_SCRIPT_FILENAME == "CGI-BIN:[000000]CGI_SYMBOLS.COM"
WWW_SCRIPT_NAME == "/cgi-bin/cgi_symbols"

If the script path had been specifically mapped using

set /cgi-bin/cgi_symbols* script=syntax=unix
the same CGI data would be provided as
WWW_PATH_INFO == "/wasd_root/src/"
WWW_PATH_TRANSLATED == "/wasd_root/SRC/"
WWW_REQUEST_URI == "/cgi-bin/cgi_symbols/wasd_root/src/"
WWW_SCRIPT_FILENAME == "/CGI-BIN/000000/CGI_SYMBOLS.COM"
WWW_SCRIPT_NAME == "/cgi-bin/cgi_symbols"

Note that the CGI or CGIplus script file is still activated using VMS file-system syntax, it is just the CGI representation that is changed. This can be particularly useful for environments ported from Unix expecting to manipulate paths using Unix syntax. This would most commonly occur with RTE engines such as PHP, Perl, etc.

1.9 - Scripting Logicals

Two logicals provide some control of and input to the DCL process scripting environment (which includes standard CGI, CGIplus and ISAPI, DECnet-based CGI, but excludes DECnet-based OSU).

Note that most WASD scripts also contain logical names that can be set for debugging purposes. These are generally in the format script_name$DBUG and if exist activate debugging statements throughout the script.

1.10 - Scripting Scratch Space

Scripts often require temporary file space during execution. Of course this can be located anywhere the scripting account (most often HTTP$SERVER) has appropriate access. The WASD package does provide a default area for such purposes with permissions set during startup to allow the server account full access. The default area is located in

WASD_ROOT:[SCRATCH]
as is accessed by the server and scripts using the logical name
WASD_SCRATCH:

The server provides for the routine clean-up of old files in WASD_SCRATCH: left behind by aborted or misbehaving scripts (although as a matter of design all scripts should attempt to clean up after themselves). The WASD_CONFIG_GLOBAL directives

[DclCleanupScratchMinutesMax]
[DclCleanupScratchMinutesOld]
control how frequently the clean-up scan occurs, and how old files need to be before being deleted. Whenever script processes are active the scratch area is scanned at the maximum period specified, or whenever the last script process is purged from the system by the server.

Of course there is always the potential for interaction between scripts using a common area for such purposes. At the most elemetary, care must be taken to ensure unique file name are generated. At worst there is the potential for malicious interaction and information leakage. Use such common areas with discretion.

NOTE

Beware of shared scratch areas. They rely on cooperation between scripts for minimising potential interactions. They can also be a source of unintended or malicious information leakage.

Unique File Names - DCL

The "UNIQUE_ID" CGI variable provides a unique 19 character alpha-numeric string (UNIQUE_ID Note) suitable for many uses including the type extension of temporary files. The following DCL illustrates the essentials of generating a script-unqiue file name. For mutliple file names add further text to the type, as shown below.

$ SCRATCH_DIR = "WASD_SCRATCH:"
$ PROC_NAME = F$PARSE(F$ENVIRONMENT("PROCEDURE"),,,"NAME")
$ INFILE_NAME = SCRATCH_DIR + PROC_NAME + "." + WWW_UNIQUE_ID + "_IN"
$ OUTFILE_NAME = SCRATCH_DIR + PROC_NAME + "." + WWW_UNIQUE_ID + "_OUT"

Unique File Names - C Language

A similar approach can be used for script coded using the C language, with the useful capacity to mark the file for delete-on-close (of course this is only really useful if it is, say, only to be written, rewound and then re-read without closing first - but I'm sure you get the idea).

#define WASD_SCRATCH "WASD_SCRATCH:"
#define SCRIPT_NAME "EXAMPLE"

char  *unqiueId;
char  tmpFileName [256];
FILE  *tmpFile;

if ((uniqueId = getenv("WWW_UNIQUE_ID")) == NULL)
{
   printf ("Error: WWW_UNIQUE_ID absent!\n");
   exit (1);
}
sprintf (tmpFileName, WASD_SCRATCH SCRIPT_NAME ".%s", uniqueId);

if ((tmpFile = fopen (tmpFileName, "w+", "fop=dlt")) == NULL)
   exit (vaxc$errno); 

1.11 - DCL Processing of Requests

DCL is the native scripting environment for VMS and provides a rich set of constructs and capabilities for ad hoc and low usage scripting, and as a glue when several processing steps need to be undertaken for a particular script. In common with many interpreted environments care must be taken with effective exception handling and data validation. To assist with the processing of request content and response generation from within DCL procedures the CGIUTL utility is available in

WASD_ROOT:[SRC.MISC]

Functionality includes

Most usefully it can read the request body, decoding form-URL-encoded contents into DCL symbols and/or a scratch file, allowing a DCL procedure to easily and effectively process this form of request.

NOTE

Never substitute the contents of CGI variables directly into the code stream using interpreters that will allows this (e.g. DCL, Perl). You run a very real risk of having unintended content maliciously change the intended function of the code. For example, never use comma substitution of a CGI variable at the DCL command line as in
$ COPY 'WWW_FORM_SRC' 'WWW_FORM_DST'
Always pre-process the content of the variable first, ensuring there has been nothing inserted that could subvert the intended purpose. The CGIUTL assists complying with this rule by providing an explicit, non-DCL substitution character for use on the command-line (see source code descriptive prologue).

1.12 - Scripting Function Library

A source code collection of C language functions useful for processing the more vexing aspects of CGI and general script programming is available in CGILIB. This and an example implementation is available in

WASD_ROOT:[SRC.MISC]

Functionality includes

The WASD scripts use this library extensively and may serve as example applications.

1.13 - Script-Requested, Server-Generated Error Responses

Of course a script can generate any output it requires including non-success (non-200) pages (e.g. 400, 401, 302, etc.) For error pages a certain consistency results from making these substantially the same layout and content as those generated by the server itself. To this end, script response header output can contain one or more of several extension fields to indicate to the server that instead of sending the script response to the client it should internally generate an error response using the script-supplied information. These fields are listed in Script-Control: section of 2.2.1 - CGI Compliant Output and are available in any scripting environment.

If a "Script-Control: X-error-text="text of error message"" field occurs in the script response header the server stops processing further output and generates an error message. Other error fields can be used to provide additional or message-modifying information. A significant example is the "Script-Control: X-error-vms-status=integer" field which supplies a VMS status value for a more detailed, status-related error message explanation.

Essentially the script just generates a standard CGI "Status: nnn" response and includes at least the "X-error-text=" field before the header-terminating empty record (blank line). Some variations are shown in the following DCL examples.

$! vanilla error message
$ say = "write sys$output"
$ say "Status: 400"
$ say "Script-Control: X-error-text=""Confusing URL components!"""
$ say ""

$! VMS status error message 
$ say = "write sys$output"
$! "status: 000" allows the server to select the HTTP status code
$ say "Status: 000"
$ say "Script-Control: X-error-text=""/a/file/name.txt"""
$ say "Script-Control: X-error-vms-status=%X00000910"
$ say "Script-Control: X-error-vms-text=""A:[FILE]NAME.TXT"""
$ say ""

$! add META source module name and line generating message
$ say = "write sys$output"
$ say "Status: 500"
$ say "Script-Control: X-error-text=""Don't know what to do now..."""
$ say "Script-Control: X-error-module=EXAMPLE; X-error-line=999"
$ say ""

Interestingly, because CGI environments should ignore response fields unknown to them, for scripts deployed across multiple server platforms it should be possible to have these WASD-specific elements in every header for WASD uses followed by other explicitly error page content for use in those other environments.

$! WASD error content, plus other platform content
$ say = "write sys$output"
$ say "Status: 404"
$ say "Script-Control: X-error-text=""Requested object not found."""
$ say "Content-Type: text/html"
$ say ""
$ say "<B>ERROR 404:</B>&nbsp; Requested object not found."

An example implemented using DCL is available

WASD_ROOT:[SRC.OTHER]REQUEST_ERROR_MSG.COM

and if currently enabled for scripting

/cgi-bin/request_error_msg


next previous contents full-page