## proc processdata

Version 2.32
Scripts

Manual page for proc_processdata(PL)

proc processdata performs various types of manipulations on the current data set.`` Here are some of the things this proc can do:

  - accumulate
  - break processing
  - count/summate on instances
  - joins
  - row numbering
  - compute percents
  - reverse row order
  - rotation of row/column matrix
  - find mean, sd, min, max
  - compute totals

The `action` attribute controls the type of processing to be done.`` The result is usually a new data set (but sometimes the result is simply the setting of some variables, as with `action: totals`).``

When a new data set is created, it will be kept in memory along with the original data by default.`` To discard the new data set and return to the original one, use proc usedata.`` For large data sets you can cause the result to overwrite the original data set in memory by setting (`stack: no`).`` In either case, after proc processdata is finished with an operation that creates a new data set the result will automatically become the "current data set".``

You can write the results to a file (`outfile`) for processing by other programs, or to read back in and process further using proc getdata `filter`.``

For additional processing capabilities see also proc getdata `filter`.``

## Example

A Gallery example where this is used is stock where the available data is in reverse chronological order.`` Since lineplot must work from left to right, proc processdata is used to reverse the record order.``

## Variables that are set by processdata

NRECORDS = Number of rows in the data result.``

NFIELDS = Number of fields per row in the data result.``

TOTALS = If totals, percents, or accumulation are being done, this variable will be set to hold the field total(s). If more than one field is being operated on, this will be a comma-delimited list of totals; individual totals may be accessed in your script using something like the following, which would access the first total in the list: `#set T = $nmember(1,@TOTALS)`

BREAKFIELD1 .. n = Current contents of break fields with `action: breaks`

`action: stats` sets a number of other variables, described in the `stats` section below.``

## The action attribute

The `action` attribute controls the type of operation that will be done.`` Often the `fields` attribute is used to indicate which field(s) are involved.

  action: accumulate Rewrite field as a cumulative series (accumulation).`` The field(s) to operate on must be given in the `fields` attribute.`` For example, the data set on the left would be transformed to the one on the right (`fields: 2`):

      	A21 3			A21 3
      	A22 5		-->	A22 8
      	A23 2			A23 10
      	A24 1			A24 11

  action: breaks Perform break processing (the act of passing through an ordered data set and taking some action when a key field or fields change).`` Break processing is significantly more efficient than scanning a data set multiple times with a `select` statement.`` The data set must be sorted such that key fields are grouped.`` The key field(s) must be specified using the `fields` attribute.`` The BREAKFIELD1 .. n variable(s) will be set to the current contents of the break field(s). Your script can detect when the entire data set has been processed by checking the NRECORDS variable (equal to 0), or the BREAKFIELD1 variable ($strlen of 0).`` For more discussion of break processing see below.``
  action: breakreset Reset the "current row" to the beginning of the data set, for the occasional time when more than one pass through a data set will be done using `action: breaks` .`` This is automatically done if a new data set is read by proc getdata.``
  action: count Collapse data by counting the number of instances of a key field.`` This action can also summate some other field based on a key field. Input data must be sorted (or at least grouped) on the key field.`` Resulting data set will always have two fields.`` One or two `fields` must be specified using the `fields` attribute. If one field is specified, the result fields will be 1) key field, 2) count.`` For example (`fields: 1`):

      	062698	 		062698 2
      	062698 		  -->	062898 1
      	062898 			070198 3
      	070198 			070498 1
      	070198
      	070198
      	070498

  If two `fields` are specified, the result fields will be 1) key field, 2) sum of the numeric contents found in the second specified field.`` For example (`fields: 1 2`):

      	062698 4		062698 10
      	062698 6	  -->	062898 3
      	062898 3		070198 9
      	070198 2		070498 2
      	070198 4
      	070198 3
      	070498 2

  See also `action: segment` and `action: segmentb` described below, which are similar.`` A gallery example that uses `action: count` is hitcount
  action: join Perform a relational join operation. (2.30+) All records come from the current data set; left side records are selected using the `leftselect` attribute and right side records are selected using `rightselect`.`` The records are joined on one or more fields specified in the `fields` attribute.`` The data must be ordered on these fields (integer fields should be ordered numerically; alphanumeric fields should be ordered alphabetically).`` One (and only one) left side record and one (and only one) right side record must exist in order to be joined (but see leftjoin and rightjoin below).`` For example, suppose we want a scatterplot where the X component and Y component are represented in different sets of records within the data set:

      	001 X 4.3		001 X 4.3 001 Y 5.2
      	001 Y 5.2	-->	002 X 3.2 002 Y 2.9
      	002 X 3.2
      	002 Y 2.9

  In order to do a scatterplot of X vs. Y we need to do a join so that all data for case 001 are on the same row, and all data for case 002 are on the same row. We can do this using:

      	#proc processdata
      	showresults: yes
      	action: join
      	fields: 1
      	leftselect: @2 = X
      	rightselect: @2 = Y

  Also available are action: leftjoin and action: rightjoin. With a left join, if a left side record exists with no right side companion, the right side is filled in with a missing data code (`=` by default; can be specified in `missingdatacode` attribute). A right join is the same thing but in reverse.``
  action: numberrows Prepend a row number field to each row. For example:

      	A AA AAA 0		1 A AA AAA 0
      	B BB BBB 0     --->  	2 B BB BBB 0
      	C CC CCC 1		3 C CC CCC 1
      	D DD DDD 1		4 D DD DDD 1
      	E EE EEE 0		5 E EE EEE 0
      	F FF FFF 1		6 F FF FFF 1

  action: percent Rewrite one or more fields as percentages of its field (column) total.`` The field(s) to operate on must be given in the `fields` attribute.`` For example (`fields: 1`):

      	8		40
      	4	   -->	20
      	3		15
      	5		25

  action: reverse The last record becomes the first one; the record order is reversed. For example (`fields: 2`):

      	AXB 34		DIF 14
      	BYA 22	   -->	CES 52
      	CES 52		BYA 22
      	DIF 14		AXB 34

  action: rotate First row becomes 1st field, 2nd row becomes 2nd field, and so on. This may be useful in that most of the plotting procs work from data fields, but sometimes data is given (or is more intuitive) in rows.`` For example:

      	A 2 4 6 8 10	-->	A B
      	B 3 6 9 12 15		2 3
      				4 6
      				6 9
      				8 12
      				10 15

  action: segment Similar to `action: count`, except that instead of a count result, a range is produced, useful for plotting bar segments, etc.`` Resulting data set will always have three fields: 1) key field, 2) start position, 3) end position.``

  If one `field` is specified, the start and end position will be record numbers of the first and last records in the range (where first record is 1). For example (`fields: 1`):

      	062698			062698 1 3
      	062698          --->	070198 4 6
      	062698			070498 7 7
      	070198
      	070198
      	070198
      	070498

  If two `fields` are specified, the start and end position will be the contents of the second specified field, for the first and last records in the range. For example (`fields: 1 2`):

      	062698 A		062698 A C
      	062698 B         --->	070198 D F
      	062698 C		070498 G G
      	070198 D
      	070198 E
      	070198 F
      	070498 G

  action: segmentb Same as `action: segment`, except that groups will butt up against one another.`` For example (`fields: 1 2`):

      	062698 A		062698 A D
      	062698 B         --->	070198 D G
      	062698 C		070498 G G
      	070198 D
      	070198 E
      	070198 F
      	070498 G

  action: stats Find the mean, SD, min, max, etc. for data field(s), and put mean, SD, min, max (etc) into ploticus variables. No new data set is created. Versions 2.30+.`` The data field(s) to operate on should specified in the `fields` attribute. Sets the following ploticus variables:
  MEAN = mean of all numeric values found
  SD = standard deviation
  N = number of numeric values found
  MIN = the lowest numeric value found
  MIN_ID = contents of identifier/abscissa field in data row where minima was found
  MAX = the highest numeric value found
  MAX_ID = contents of identifier/abscissa field in data row where maxima was found
  NMISSING = number of non-numeric observations
  TOTAL = sum of all numeric values in the field

  (The following is a change in functionality starting in 2.31:) This action can operate on one data field or several. The data field(s) to operate on should specified in the `fields` attribute. If more than one data field is specified, the data values from all the fields will be examined as a group and the result will still be a single mean, SD, etc. `tagfield` may be specified (see below) to indicate an identifier or abscissa field which will be used to identify min and max cases (MIN_ID and MAX_ID) and may be useful in annotating the min and/or max with a later invocation of #proc annotate.``

  action: total Compute field total(s) only and place total(s) into the variable TOTALS (see above). No new data set is created.`` Field(s) to be totalled are specified in the `fields` attribute.``
  The decimal format of the total(s) is controlled by the `resultformat` attribute. If total(s) are to be written in presentable notation (a spacer for thousands, etc.) the `resultformat` attribute may be preceded by a `n`, e.g. `n%7.0f`.``

## Other attributes

fields dfield list

  The field(s) to be operated on. Required for any `action` that involves data fields.``
  Example: `fields: 2 5 6 7`

keepfields dfield list

  If specified, only the listed fields in the original data set will be kept. The others will be rejected. `action` may be anything.``
  Example: `keepfields: 4 5 6`

rejectfields dfield list

  If specified, the listed fields in the original data set will not be kept. The others will be. `action` may be anything.``
  Example: `rejectfields: 1 2 3 4`

tagfield dfield

  Used with `action: stats` to specify an identifier or abscissa data field, which will be used to identify min and max cases.``

fieldnames namelist

  If specified, result data fields will be assigned the given names.`` These names can later be used in any plotting proc to identify data fields. namelist is a space- or comma- delimited list of names. Names may include any alphanumeric characters with a maximum length of 38, and are case-insensitive.``
  Note that if field names are specified in proc getdata and then proc processdata is used to alter the order of fields or delete fields, then this `fieldnames` attribute must be used in order to redefine the field names properly.``
  Example: `fieldnames: date group n`

resultformat printf-spec

  Controls the decimal format of rewritten percents, accumulations, totals.`` Default is `%g`.``
  Example: `resultformat: %f`
  Example for `action: totals` (see above): `resultformat: n%g`

keepall `yes` | `no`

  If `yes`, original fields are preserved when doing accumulate, percent, or total.`` Thus the result will have more fields than the original data set. Default is `no`.``

select selection expression

  Used when `action` is `select` in order to specify the selection condition. Beginning in version 2.20 `select` can be used along with any `action`.``
  Example: `select: @@4 = A`

stack `yes` | `no`

  The default is `yes` which causes the result data set to be "stacked", and the original data set is preserved in memory.`` If `no`, the result data set replaces the original data set in memory (this might be desired for larger data sets).``

showresults `yes | no`

  If `yes` the data are shown after processing, as a diagnostic aid.``

outfile filename

  If specified, results are written to filename, in tab-delimited format, for processing by other programs, or to read back in and process further using proc getdata `filter`.`` filename can be a pathname or `stdout` or `stderr`.`` Use of this attribute implies `stack: no`

leftselect select expression
rightselect select expression

  Used with `action: join` (and leftjoin, rightjoin).`` Specifies selection conditions for left side of join and right side of join.``

missingdatacode string

  Used with `action: leftjoin` (and rightjoin) to specify a missing data code to use for filling in missing fields.``

## More on break processing

`action: breaks` performs break processing. In data processing terminology, "break processing" is the act of passing through a sorted data set and doing something special when a change is encountered in key field(s).`` For example, if we were processing a list of charges ordered by paying budget number, we could use a break processing strategy to pause and generate a statement for one budget number, when we reached the point in the data set where the budget numbers changed. Then we would continue on.`` `action: breaks` allows a similar thing to be done with plotting.`` Break processing is significantly more efficient than scanning the entire data set multiple times with a `select statement`, especially with larger data sets.``

The data set must be sorted such that key fields are grouped.`` The key field(s) must be specified in the `fields` attribute.`` proc processdata is generally called within a #loop. When proc processdata finishes, the current data set will be the block of data from the previous break to the current break. proc usedata must be used at the bottom of the loop, to set the current data set back to the larger data set that we are passing thru.`` Subsequent invocations of proc processdata then continue from the most recent break location.``

Your script can access the current contents of the break field(s) via the BREAKFIELD1 .. n variable(s).`` Your script can detect when the entire data set has been processed by checking the NRECORDS variable (equal to 0), or the BREAKFIELD1 variable ($strlen of 0).`` The following is an example:

     	#loop
     	   #proc processdata
     	     action: breaks
     	     fields: 1 2 3
     	   #proc endproc

     	   #if @NRECORDS = 0
     	     #break
     	   #endif

     	   #proc page
     	   title: Account @BREAKFIELD1

     	   #proc bars
     	   ...

              #proc usedata
                original: yes

     	 #endloop

Limits: up to 5 break fields may be used.`` Comparisons for equality are limited to the first 50 characters.``
data display engine
Copyright Steve Grubb

Markup created by unroff 1.0, `` `` August 23, 2005.