The STDIZE Procedure

Computing Quantiles

PROC STDIZE offers two methods for computing quantiles: the one-pass approach and the order-statistics approach (like that used in the UNIVARIATE procedure).

The one-pass approach used in PROC STDIZE modifies the normal upper P squared algorithm for histograms proposed by Jain and Chlamtac (1985). The primary difference comes from the movement of markers. The one-pass method allows a marker to move to the right (or left) by more than one position (to the largest possible integer) as long as it does not result in two markers being in the same position. The modification is necessary in order to incorporate the FREQ variable.

You might obtain inaccurate results if you use the one-pass approach to estimate quantiles beyond the quartiles (that is, when you estimate quantiles < P25 or quantiles > P75). A large sample size (10,000 or more) is often required if the tail quantiles (quantiles less-than-or-equal-to P10 or quantiles greater-than-or-equal-to P90) are requested. Note that, for variables with highly skewed or heavy-tailed distributions, tail quantile estimates might be inaccurate.

The order-statistics approach for estimating quantiles is faster than the one-pass method but requires that the entire data set be stored in memory. The accuracy in estimating the quantiles is comparable for both methods when the requested percentiles are between the lower and upper quartiles. The default is PCTLMTD=ORD_STAT if enough memory is available; otherwise, PCTLMTD=ONEPASS.

Computational Methods for the PCTLDEF= Option

You can specify one of five methods for computing quantile statistics when you use the order-statistics approach (PCTLMTD=ORD_STAT); otherwise, the PCTLDEF=5 method is used when you use the one-pass approach (PCTLMTD=ONEPASS).

Percentile Definitions

Let n be the number of nonmissing values for a variable, and let x 1 comma x 2 comma ellipsis comma x Subscript n Baseline represent the ordered values of the variable. For the tth percentile, let p equals t slash 100. In the following definitions numbered 1, 2, 3, and 5, let

n p equals j plus g

where j is the integer part and g is the fractional part of np. For definition 4, let

left-parenthesis n plus 1 right-parenthesis p equals j plus g

Given the preceding definitions, the tth percentile, y, is defined as follows:

PCTLDEF=1

weighted average at x Subscript n p

y equals left-parenthesis 1 minus g right-parenthesis x Subscript j Baseline plus g x Subscript j plus 1

where x 0 is taken to be x 1

PCTLDEF=2

observation numbered closest to np

y equals x Subscript i

where i is the integer part of n p plus 1 slash 2 if g not-equals 1 slash 2. If g equals 1 slash 2, then y equals x Subscript j if j is even, or y equals x Subscript j plus 1 if j is odd

PCTLDEF=3

empirical distribution function

y equals x Subscript j Baseline normal i normal f g equals 0
y equals x Subscript j plus 1 Baseline normal i normal f g greater-than 0
PCTLDEF=4

weighted average aimed at x Subscript p left-parenthesis n plus 1 right-parenthesis

y equals left-parenthesis 1 minus g right-parenthesis x Subscript j Baseline plus g x Subscript j plus 1

where x Subscript n plus 1 is taken to be x Subscript n

PCTLDEF=5

empirical distribution function with averaging

y equals left-parenthesis x Subscript j Baseline plus x Subscript j plus 1 Baseline right-parenthesis slash 2 normal i normal f g equals 0
y equals x Subscript j plus 1 Baseline normal i normal f g greater-than 0

Weighted Percentiles

When you specify a WEIGHT statement, or specify the NOTRUNCATE option in a FREQ statement, the percentiles are computed differently. The 100pth weighted percentile y is computed from the empirical distribution function with averaging

y equals StartLayout Enlarged left-brace 1st Row 1st Column one-half left-parenthesis x Subscript i Baseline plus x Subscript i plus 1 Baseline right-parenthesis 2nd Column normal i normal f sigma-summation Underscript j equals 1 Overscript i Endscripts w Subscript j Baseline equals p upper W 2nd Row 1st Column x Subscript i plus 1 Baseline 2nd Column normal i normal f sigma-summation Underscript j equals 1 Overscript i Endscripts w Subscript j Baseline less-than p upper W less-than sigma-summation Underscript j equals 1 Overscript i plus 1 Endscripts w Subscript j EndLayout

where w Subscript i is the weight associated with x Subscript i, and where upper W equals sigma-summation Underscript i equals 1 Overscript n Endscripts w Subscript i is the sum of the weights.

For PCTLMTD= ORD_STAT, the PCTLDEF= option is not applicable when a WEIGHT statement is used, or when a NOTRUNCATE option is specified in a FREQ statement. However, in this case, if all the weights are identical, the weighted percentiles are the same as the percentiles that would be computed without a WEIGHT statement and with PCTLDEF=5.

For PCTLMTD= ONEPASS, the quantile computation currently does not use any weights.

Last updated: December 09, 2022