The MULTTEST Procedure

requests adjusted p-values by using the Hochberg and Benjamini (1990) adaptive step-up Bonferroni method. See the section Adaptive Adjustments for more details.

requests adjusted p-values by using the Hochberg and Benjamini (1990) adaptive step-down Bonferroni method. See the section Adaptive Adjustments for more details.

requests adjusted p-values by using the Benjamini and Hochberg (2000) adaptive linear step-up method (AFDR). The UNRESTRICT option estimates the AFDR as defined in Benjamini and Hochberg (2000), which allows the adjustment to reduce the raw p-value. By default, the AFDR is constrained to be greater than or equal to the raw p-value. See the section Adaptive False Discovery Rate for more details.

specifies that the Bonferroni adjustments (number of tests p-value) be computed for each test. These adjustments can be extremely conservative and should be viewed with caution. When exact tests are specified via the PERMUTATION= option in the TEST statement, the actual permutation distributions are used, resulting in a much less conservative version of this procedure (Westfall and Wolfinger 1997). See the section Bonferroni for more details.

specifies that the p-values be adjusted by using the bootstrap method to resample vectors (Westfall and Young 1993). Resampling is performed with replacement and independently within levels of the STRATA variable. Continuous variables are mean-centered by default prior to resampling; specify the NOCENTER option to change this. See the section Bootstrap for more details. The BOOTSTRAP option is not allowed with the Peto test.

If the PERMUTATION= suboption is used with the CA test in the TEST statement, the exact permutation distribution is recomputed for each bootstrap sample. Caution: This can be very time-consuming. It is preferable to use permutation resampling when permutation base tests are used.

requests that continuous variables be mean-centered prior to resampling. The default action is to mean-center for bootstrap resampling and not to mean-center for permutation resampling.

names the input SAS data set to be used by PROC MULTTEST. The default is to use the most recently created data set. The DATA= and INPVALUES= options cannot both be specified.

requests adjusted p-values by using the method of Benjamini and Yekutieli (2001). See the section Dependent False Discovery Rate for more details.

specifies the amount by which two p-values must differ to be declared unequal. The value number must be between 0 and 1; the default value is 1000 times the machine epsilon, which is approximately 1E–12. See Westfall and Young (1993, pp. 165–166) for more information.

requests adjusted p-values by using the linear step-up method of Benjamini and Hochberg (1995). These p-values do not control the familywise error rate, but they do control the false discovery rate in some cases. See the section False Discovery Rate Controlling Adjustments for more details.

A bootstrap-resampling false discovery rate controlling method due to Yekutieli and Benjamini (1999). This method uses the same resampling algorithm as the BOOTSTRAP option. Every resample is saved in order to compute a quantile of the resampled p-values; therefore, this method can use a lot of memory. The parameter designates that a 100 quantile is used in the computations for determining the adjustments; by default, . See the section False Discovery Rate Resampling Adjustments for details.

A permutation-resampling false discovery rate controlling method due to Yekutieli and Benjamini (1999). This method uses the same resampling algorithm as the PERMUTATION option. Every resample is saved in order to compute a quantile of the resampled p-values; therefore, this method can use a lot of memory. The parameter designates that a 100 quantile is used in the computations for determining the adjustments; by default, . See the section False Discovery Rate Resampling Adjustments for details.

requests adjusted p-values by using Fisher’s combination method. See the section Fisher Combination for more details.

requests adjusted p-values by using the step-up Bonferroni method due to Hochberg (1988). See the section Hochberg for more details.

requests adjusted p-values by using the method of Hommel (1988). See the section Hommel for more details.

is an alias for the STEPBON adjustment.

names an input SAS data set that includes a variable containing raw p-values. The MULTTEST procedure adjusts the collection of raw p-values for multiplicity. Resampling-based adjustments are not permitted with this type of data input. The CLASS, CONTRAST, FREQ, STRATA, and TEST statements are ignored when an INPVALUES= data set is specified. The INPVALUES= and DATA= options cannot both be specified. The pvalue-name enables you to specify the name of the p-value column from your data set. By default, pvalue-name=’raw_p’. The INPVALUES= data set can contain variables in addition to the raw p-values variable; see Example 86.5 for an example.

is an alias for the STOUFFER adjustment.

requests that continuous variables not be mean-centered prior to resampling. The default action is to mean-center for bootstrap resampling and not to mean-center for permutation resampling.

suppresses the normal display of results. Note that this option temporarily disables the Output Delivery System (ODS); see Chapter 23, Using the Output Delivery System, for more information.

suppresses the display of the "p-Values" table of raw and adjusted p-values. This option is most useful when you are adjusting many tests and need to create only an OUT= data set or display graphics.

suppresses display of the "Discrete Variable Tabulations" and "Continuous Variable Tabulations" tables.

suppresses display of tables having zero occurrences for all CLASS levels.

specifies the number of resamples for use with the resampling methods. The value number must be a positive integer; by default, 20,000 resamples are used. Large values of number (20,000 or more) are usually recommended for accuracy, but long execution times can result, particularly with large data sets.

Controls the method used to estimate the number of true NULL hypotheses () for the adaptive methods. This option is ignored unless one of the adaptive methods is specified. By default, PROC MULTTEST uses the DECREASESLOPE method for the ADAPTIVEHOLM and ADAPTIVEHOCHBERG adjustments, and the LOWESTSLOPE method for ADAPTIVEFDR adjustment. For the PFDR adjustment, the SPLINE method is attempted first. If the estimate is nonpositive or if the slope of the spline at the last is greater than 0.1 times the range of the fitted spline values, then the BOOTSTRAP method is used.

You can specify a positive integer as the value, or you can specify one of the keywords in the following list. Alternatively, you can specify the proportion of true NULL hypotheses by using the PTRUENULL= option. Suppose you have m tests with ordered p-values , and define .

In all cases is constrained to lie between 0 and m; if the computed , then the adaptive adjustments do not produce results. If you specify , then it is reduced to m. Values of are displayed in the "Estimated Number of True Null Hypotheses" table.

specifies the sort order for the levels of the classification variables (which are specified in the CLASS statement). This option applies to the levels for all classification variables, except when you use the (default) ORDER=FORMATTED option with numeric classification variables that have no explicit format. In that case, the levels of such variables are ordered by their internal value.

The ORDER= option can take the following values:

By default, ORDER=FORMATTED. For ORDER=FORMATTED and ORDER=INTERNAL, the sort order is machine-dependent. For more information about sort order, see the chapter on the SORT procedure in the Base SAS Procedures Guide and the discussion of BY-group processing in the "Grouping Data" section of SAS Programmers Guide: Essentials.

names the output SAS data set containing variable names, contrast names, intermediate calculations, and all associated p-values. See OUT= Data Set for more information.

names the output SAS data set containing entire permutation distributions (upper-tail probabilities) for all tests when the PERMUTATION= option is specified. See OUTPERM= Data Set for more information. Caution: This data set can be very large.

names the output SAS data set containing information from the resampled data sets when resampling is performed. See OUTSAMP= Data Set for more information. Caution: This data set can be very large.

is an alias for the INPVALUES= option.

computes adjusted p-values in identical fashion as the BOOTSTRAP option, with the exception that PROC MULTTEST resamples without replacement rather than with replacement. Resampling is performed independently within levels of the STRATA variable. Continuous variables are not mean-centered prior to resampling; specify the CENTER to change this. See the section Bootstrap for more details. The PERMUTATION option is not allowed with the Peto test.

computes the "q-values" of Storey (2002) and Storey, Taylor, and Siegmund (2004). PROC MULTTEST treats these "q-values" as adjusted p-values. The computations depend on selecting a parameter and an estimation method for the false discovery rate; see the section Positive False Discovery Rate for computational details. The available options for choosing the method are as follows:

The available options for controlling the search are the bootstrap-options, the spline-options, and the following options:

controls the plots produced through ODS Graphics. If you specify only one plot-request, you can omit the parentheses. For example, the following statements are valid specifications of the PLOTS= option:

plots = all
plots = (rawprob adjusted)
plots(sigonly) = (rawprob adjusted(unpack))

ODS Graphics must be enabled before plots can be requested. For example:

ods graphics on;
proc multtest plots=adjusted inpvalues=a pfdr;
run;
ods graphics off;

For more information about enabling and disabling ODS Graphics, see the section Enabling and Disabling ODS Graphics in Chapter 24, Statistical Graphics Using ODS.

By default, no graphs are created; you must specify the PLOTS= option to make graphs. You need at least two tests to produce a graph. If you are not using an INPVALUES= data set, then each test is given a name constructed as "variable-name contrast-label". If you specify a MEAN test in the TEST statement, the t-test names are prefixed with "Mean:". See Example 86.6 for examples of the ODS graphical displays.

The following global-plot-options are available:

The following plot-requests are available:

is alias for the NTRUENULL= option, except that you can specify the proportion of true null hypotheses as a value between 0 and 1, instead of specifying the number of true null hypotheses. The available keywords are also the NTRUENULL= options.

uses an old random number generating algorithm. By default, the random number generator is the Mersenne Twister, which has better performance when bootstrapping.

specifies the initial seed for the random number generator used for resampling. The value for number must be an integer. If you do not specify a seed, or if you specify a value less than or equal to zero, then PROC MULTTEST uses the time of day from the computer’s clock to generate an initial seed. For more details about seed values, see SAS Programmers Guide: Essentials.

computes the Šidák adjustment for each test. These adjustments take the form

where p is the raw p-value and m is the number of tests. These are slightly less conservative than the Bonferroni adjustments, but they still should be viewed with caution. When exact tests are specified via the PERMUTATION= option in the TEST statement, the actual permutation distributions are used, resulting in a much less conservative version of this procedure (Westfall and Wolfinger 1997). See the section Šidák for more details.

requests adjusted p-values by using the step-down Bonferroni method of Holm (1979). See the section Step-Down Methods for more details.

requests that adjusted p-values be computed by using bootstrap resampling as described under the BOOTSTRAP option, but in step-down fashion. See the section Step-Down Methods for more details.

requests that adjusted p-values be computed by using permutation resampling as described under the PERMUTATION option, but in step-down fashion. See the section Step-Down Methods for more details.

requests adjusted p-values by using the Šidák method as described in the SIDAK option, but in step-down fashion. See the section Step-Down Methods for more details.

requests adjusted p-values by using the Stouffer-Liptak combination method. See the section Stouffer-Liptak Combination for more details.