The PSMATCH Procedure

Sensitivity Analysis

Propensity score analysis assumes that all the confounders (variables that affect both the outcome and the treatment assignment) have been measured. If some confounders are unobserved, individuals that have the same observed covariates might not have the same probability of being assigned to the treated group. The assumption of no unmeasured confounders cannot be verified, so you should analyze the sensitivity of inferences to departures from this assumption. Sensitivity analysis considers how strong the unobserved covariates would have to be in order to negate the conclusion of the study (assuming that the initial analysis found a significant effect of the treatment).

Liu, Kuramoto, and Stuart (2013) describe seven commonly used techniques for sensitivity analysis. Based on the study objectives, these methods are classified into two groups. One group finds the tipping point that negates the statistical significance of the outcome-treatment association (Liu, Kuramoto, and Stuart 2013). The other group (not discussed here) derives the point estimate of the true outcome-treatment association with a 95% confidence interval (Liu, Kuramoto, and Stuart 2013).

Rosenbaum (2010, p. 77) provides a sensitivity analysis based on the odds ratio,

StartFraction pi Subscript k Baseline slash left-parenthesis 1 minus pi Subscript k Baseline right-parenthesis Over pi Subscript l Baseline slash left-parenthesis 1 minus pi Subscript l Baseline right-parenthesis EndFraction

where pi Subscript k and pi Subscript l are the probabilities that the kth and lth individuals are assigned to the treated group, given that they have the same observed covariates, x Subscript k Baseline equals x Subscript l.

For all kth and lth individuals with x Subscript k Baseline equals x Subscript l, assume that the odds ratio is bounded by

StartFraction 1 Over normal upper Gamma EndFraction less-than-or-equal-to StartFraction pi Subscript k Baseline slash left-parenthesis 1 minus pi Subscript k Baseline right-parenthesis Over pi Subscript l Baseline slash left-parenthesis 1 minus pi Subscript l Baseline right-parenthesis EndFraction less-than-or-equal-to normal upper Gamma

where normal upper Gamma greater-than-or-equal-to 1.

The parameter normal upper Gamma measures the degree of hidden bias from unobserved confounders. For example, with normal upper Gamma equals 2,

StartFraction pi Subscript k Baseline slash left-parenthesis 1 minus pi Subscript k Baseline right-parenthesis Over pi Subscript l Baseline slash left-parenthesis 1 minus pi Subscript l Baseline right-parenthesis EndFraction equals 2

which indicates that even though they have the same values of the observed covariates, the kth individual is twice as likely as the lth individual to be in the treated group because of hidden bias.

Propensity score analysis assumes that if the kth and lth individuals have the same observed covariates, then pi Subscript k Baseline equals pi Subscript l and normal upper Gamma equals 1. When an outcome analysis leads to a significant result, Rosenbaum’s sensitivity analysis finds a tipping point, normal upper Gamma equals gamma, that negates the conclusion of the study. A large value of gamma is evidence that only a large departure from random treatment assignment can negate the conclusion of the study. If normal upper Gamma equals gamma is plausible, the study conclusion is not robust to hidden bias from an unobserved confounder.

For the case of one-to-one matched observations, Rosenbaum (2010, pp. 78–84) provides a sensitivity analysis that is based on paired observations. The following subsection describes this approach.

Sensitivity Analysis on Matched Observations

In a set of one-to-one matched observations, if individuals k and l are in the same matched set, then the probability that individual k is in the treated group and individual l is in the control group is

StartFraction pi Subscript k Baseline Over pi Subscript k Baseline plus pi Subscript l Baseline EndFraction

and the following equation can be used for sensitivity analysis:

StartFraction 1 Over 1 plus normal upper Gamma EndFraction less-than-or-equal-to StartFraction pi Subscript k Baseline Over pi Subscript k Baseline plus pi Subscript l Baseline EndFraction less-than-or-equal-to StartFraction normal upper Gamma Over 1 plus normal upper Gamma EndFraction

If normal upper Gamma equals 1, then pi Subscript k Baseline equals pi Subscript l.

For example, let y Subscript j t and y Subscript j c be the responses for the treated and control units in the jth matched set. The response is the improvement after treatment, and a positive value indicates a beneficial effect. Let

d Subscript j Baseline equals y Subscript j t Baseline minus y Subscript j c

be the difference in responses between the treated and control units.

Suppose that a signed rank test is used in the outcome analysis. The signed rank statistic is

upper S equals sigma-summation Underscript j colon d Subscript j Baseline greater-than 0 Endscripts d Subscript j Superscript plus

where d Subscript j Superscript plus is the rank of StartAbsoluteValue d Subscript j Baseline EndAbsoluteValue.

Assume that normal upper Gamma=1. Then under the hypothesis of no treatment effect, S has mean

mu 0 equals StartFraction n Subscript t Baseline left-parenthesis n Subscript t Baseline plus 1 right-parenthesis Over 4 EndFraction

where n Subscript t is the number of matched sets and the variance V (assuming that all d Subscript j is distinct) is

upper V 0 equals StartFraction n Subscript t Baseline left-parenthesis n Subscript t Baseline plus 1 right-parenthesis left-parenthesis 2 n Subscript t Baseline plus 1 right-parenthesis Over 24 EndFraction

When n Subscript t Baseline greater-than 20, the significance of

StartFraction upper S minus mu 0 Over StartRoot upper V 0 EndRoot EndFraction

can be computed from the Student’s t distribution with n Subscript t Baseline minus 1 degrees of freedom.

For normal upper Gamma equals gamma, S has mean

mu 1 equals StartFraction gamma Over 1 plus gamma EndFraction StartFraction n Subscript t Baseline left-parenthesis n Subscript t Baseline plus 1 right-parenthesis Over 2 EndFraction equals StartFraction 2 gamma Over 1 plus gamma EndFraction mu 0

and variance

upper V 1 equals StartFraction gamma Over left-parenthesis 1 plus gamma right-parenthesis squared EndFraction StartFraction n Subscript t Baseline left-parenthesis n Subscript t Baseline plus 1 right-parenthesis left-parenthesis 2 n Subscript t Baseline plus 1 right-parenthesis Over 6 EndFraction equals StartFraction 4 gamma Over left-parenthesis 1 plus gamma right-parenthesis squared EndFraction upper V 0

If a signed rank test shows a significantly better benefit in the treated group with normal upper Gamma equals 1, the sensitivity analysis searches for a tipping point that negates the study conclusion. A study conclusion is robust to hidden bias from the unobserved confounder if an extreme value of normal upper Gamma is needed to alter the study conclusion.

Example 101.9 illustrates a sensitivity analysis on a set of one-to-one matched observations.

Last updated: December 09, 2022