## proc getdata

Version 2.32
Scripts

Manual page for proc_getdata(PL)

proc getdata gets data for plotting, and must be invoked before plotting any data.`` Data may be read from files, from commands, or specified directly, and various input data formats are supported.`` The data which are read become the current data set that subsequent plotting procs will access, and replace any data that were read in (or generated) previously.``

proc getdata will set the variable NRECORDS to the number of data rows gotten.`` The variable NFIELDS will be set to the number of fields per data row.``

## Attributes

Default field delimitation method is `spacequoted`; `delim` must be set for other types such as tab-delimited or comma-quote delimited.``

## Specifying a data source

One (and only one) of the following data sources must be specified:

file filename

  Shell-expandable name of a file containing plotting data.`` This name will be used along with cat(1) in a shell command, thus exported shell variables and metacharacters may be part of the name.`` A dash (-) may be used if data is to be read from the standard input, (or the `standardinput` attribute may be used).``

  Security concern: user-supplied values, such as CGI user variables, should not be used to build `filename`.

  On win32 platforms, or when operating in direct CGI mode (2.31-08+) `filename` automatically maps to `pathname` .``

  Example: `filename: myfile.dat`

pathname filename

  Name of a file containing plotting data.`` The file will be opened directly.``

  Security concern: user-supplied values, such as CGI user variables, should not be used to build the `pathname`, unless proper measures are taken to detect and remove the `../` construct (used as a hack to see higher levels of the file system).``

command shell command line

  An external shell command that will produce plot data on its standard output.``

  Security concern: user-supplied values, such as CGI user variables, should not be used to build the shell command.`` If this must be done, use `#shell / #endshell` along with `data:`, which provides some protection against shell metacharacter hacks.``

  Example: `command: cat mydat | uniq -c`

commandmr multi-row shell command

  Same as `command` but shell command can occupy multiple rows, and is is terminated by a blank line. Command can be up to ~1000 characters long.``

  Security concern: user-supplied values, such as CGI user variables, should not be used to build the shell command.`` If this must be done, use #shell / #endshell along with `data:`, which provides some protection against shell metacharacter hacks.``

  You can overlay flow-of-control directives as in this example:

         commandmr:  cat foo.dat |
                     process_step1 |
             #if @sortmode = 1
                     sort +2 -3
             #else
                     cat -
             #endif

standardinput `yes` | `no`

  If `yes`, data stream is read from the ploticus standard input.``

data multiline-text

  Literal specification of plotting data. Terminates at first blank (empty) line. Data may include a field name header.`` Example:

      data:   "Case 1"   0   4   4.1
              "Case 2"   1   5   4.4
              "Case 3"   2   2   4.0
              "Case 4"   3   9   4.8

  ``

sql sql command

  Submit the given sql command and capture the results as tabular data, using sql result column names as the field names.`` This is currently available only for builds related to the QUISP/SHSQL package.``

sqlmr multi-row sql command

  Same as `sql` but SQL command can occupy multiple rows, and is is terminated by a blank line.`` Command can be up to ~1000 characters long.`` You can overlay flow-of-control directives as in this example:

         sqlmr: select id, lastname, firstname, balance
                from accounts
           #if @mode = 1
                where acct_type = g
           #endif

#intrailer

  Indicates that a `data` attribute will be given in a #proc trailer, at the end of the script file.`` See EXAMPLES, below.``

## Characteristics of the data stream

delim `spacequoted` | `whitespace` | `comma` | `tab`

  The type of delimiting method to be used when parsing the data. `spacequoted` is the default (`space` is equivalent to `spacequoted`). See dataformat for details.``
  Example: `delim: comma`

nfields n

  If specified, this sets the expected number of fields per record.`` If a data row has more than the expected number of fields, extra fields are silently ignored. If a data row has less than the expected number of fields, blank fields are silently added.`` This is applied after any `filter` processing.`` If not specified, the first non-comment non-header row encountered will set the expected number of fields.``

commentchar string

  A character or group of characters that is used to signify a comment in the data file.`` Commented lines will be skipped. Default is `//`.`` Example: `commentchar: #`

## Setting field names

fieldnameheader `yes` | `no`

  Allows field names to be embedded in the input data.`` If `yes`, the first non-comment line in the data is expected to contain field names. This line is not considered part of the data.`` The field name header should be delimited like the rest of the data.`` Field names may not contain embedded white space, commas, or quote characters but (2.30+) there is a way to encode spaces and commas... see proc settings encodenames.``

fieldnames namelist

  Specify field names for input data fields.`` namelist is a list delimited by spaces and/or commas.`` Names may include any alphanumeric characters with a maximum length of 48, and are case-insensitive.`` Field names may not contain embedded spaces or commas, but (2.30+) you can encode them... see proc settings encodenames.`` Note: if you are using a `filter` (see below) you may want to use `pf_fieldnames` (see below) to name the result fields.``
  Example: `fieldnames: date group n`

fieldnamerows multiline text

  Same as `fieldnames` (see above), except that field names are given one per line.`` Must be terminated by a blank line.`` Example:

      fieldnamerows:
        id
        type
        age
        sex

pf_fieldnames namelist

  Assign new field names to `filter` result.`` See `filter` attribute described below.`` Useful when the filter result has a different logical record format than the input.`` See also `fieldnames` above.``
  Example: `pf_fieldnames: date z sum1 sum2`

## Development & debugging

showresults `yes` | `no`

  If `yes`, the results, after any selecting and/or filtering, are written to the diagnostic file, which may be useful in debugging, etc.``

## Selecting and manipulating input rows

select select expression

  This allows certain data records to be selected for inclusion based upon a selection expression.`` Incoming data fields are referenced by number, using a double at-sign (`@@`) prefix. Hint: use the `showresults` attribute when debugging.``
  Example: `select: @@3 = g`
  This would select all data records having 3rd field equal to `g`.``

rotate `yes` | `no`

  Allows data to be given all in one row, even when plotting proc expects one record per instance (which most do).`` Only applicable data set has one row.`` To rotate more than one row, use proc processdata with `action: rotate`.``

filter multiline text

  An embedded script which allows flexible processing to be applied to incoming data records one at a time.`` Typical uses are for concatenating or splitting fields, doing on-the-fly date conversions, or generating derived fields such as the sum of several fields or the difference between two fields.``
  The embedded script will be applied once to every incoming data record.`` The script should produce some "output"; generally the last statement is a ##print. The output must use the same delimitation method as the input.`` The output may have a different logical record format than the input.`` If you are using field names, the `pf_fieldnames` attribute (see above) may be used to name the filter result fields when result record format differs from that of the input.``
  The script uses the same syntax as the greater ploticus script, except that:

  - directives must begin with two pound signs (`##`) instead of one
  - local variables begin with two at signs (`@@`) instead of one
  - fields on the incoming data record are accessed like this: `@@1` for the first field, `@@2` for the second, etc. If you are using field names, these may be used as well, eg: `@@score`.``
  - the only directives that may be used are ##set, ##if, ##elseif, ##else, ##print, ##call, and ##exit

  Other things worth noting:

  - the filter script is terminated using a blank line.``
  - use the `showresults` attribute when debugging.``
  - if `filter` is used along with select, the `select` is applied first.
  - `filter` cannot be used on data specified using the `data` attribute.``
  - `filter` can only access fields from one data record at a time.``
  - variables from the ploticus script may be referenced within the `filter` script (use one at-sign `@`).`` Evaluation occurs before the filter script executes.``
  - ploticus script #if/#else statements (single pound sign) may be used to selectively execute portions of the filter script.`` Interpretation occurs before the filter script executes.``
  - Example: This filters out data records having field 2 or field 3 equal to M. It then calulates the difference in days between two dates and puts this difference in the variable DIFF.`` Finally it "prints" incoming field 1 along with DIFF.`` Thus the result of this #proc getdata will have be data records having two fields.``

         filter:
              ##if @@2 = M || @@3 = M
                ##exit
              ##endif
              ##set DIFF = $daysdiff(@@3,@@2)
              ##print @@1 @@DIFF

    ``
  - There are several more `filter` examples in the FAQ

## More examples

Data specification may be located at the end of the script file by using #intrailer and #proc trailer. This may be useful in "getting the data out of the way", or with automated building of script files.`` Here is how this is done:

     #proc getdata
     #intrailer

    other #procs, etc.

     #proc trailer
     Data:	0.3 0.5 2.3
    	3.5 9.4 1.4
    	..etc..
    end of file

## Variables that are set by proc getdata

NRECORDS = the number of data rows gotten

NFIELDS = the number of fields per record

## Hints

It is possible to set working variables from within the data file.``

During debugging, set `showresults: yes` in order to see the data after it is read and parsed.`` Especially useful when working with `filter`.``

In dynamic content environments it's good to gracefully handle the situation of an empty data file or command that produced no output. Example:

       #proc getdata
          ...

       #proc endproc
       #if @NRECORDS = 0
         #proc annotate
         location: 3 3
         text: No data found.

         #exit
       #endif

  data display engine
  Copyright Steve Grubb

Markup created by unroff 1.0, `` `` August 23, 2005.