The SURVEYFREQ Procedure

Kappa Coefficients

Simple Kappa Coefficient

The KAPPA option (or the AGREE option) provides an estimate of the simple kappa coefficient, its standard error, and the confidence limits. This statistic is available for replication variance estimation methods (which you can request by specifying VARMETHOD=BOOTSTRAP, VARMETHOD=BRR, or VARMETHOD=JACKKNIFE or by including a REPWEIGHTS statement).

The simple kappa coefficient (Cohen 1960) is a measure of interrater agreement, where the row and column variables of the two-way table are viewed as two independent ratings. When there is perfect agreement between the two ratings, the kappa coefficient is +1. When the observed agreement exceeds chance agreement, the value of kappa is positive, and its magnitude reflects the strength of agreement. The minimum value of kappa is between –1 and 0, depending on the marginal proportions. For more information, see Fleiss, Levin, and Paik (2003).

PROC SURVEYFREQ computes the simple kappa coefficient as

ModifyingAbove kappa With caret equals left-parenthesis upper P Subscript o Baseline minus upper P Subscript e Baseline right-parenthesis slash left-parenthesis 1 minus upper P Subscript e Baseline right-parenthesis

where

upper P Subscript o Baseline equals sigma-summation Underscript i Endscripts ModifyingAbove upper P With caret Subscript i i
upper P Subscript e Baseline equals sigma-summation Underscript i Endscripts ModifyingAbove upper P With caret Subscript i dot Baseline ModifyingAbove upper P With caret Subscript dot i

where ModifyingAbove upper P With caret Subscript i i is the estimate of the proportion in table cell (i, i), ModifyingAbove upper P With caret Subscript i dot is the estimate of the proportion in row i, and ModifyingAbove upper P With caret Subscript dot i is the estimate of the proportion in column i. For information about how PROC SURVEYFREQ computes the proportion estimates, see the section Proportions.

PROC SURVEYFREQ estimates the variance of the simple kappa coefficient as described in the section Replication Variance Estimation. PROC SURVEYFREQ computes confidence limits for the simple kappa coefficient as

ModifyingAbove kappa With caret plus-or-minus left-parenthesis t Subscript d f comma alpha slash 2 Baseline times normal upper S normal t normal d normal upper E normal r normal r left-parenthesis ModifyingAbove kappa With caret right-parenthesis right-parenthesis

where normal upper S normal t normal d normal upper E normal r normal r left-parenthesis ModifyingAbove kappa With caret right-parenthesis is the standard error of the kappa coefficient and t Subscript d f comma alpha slash 2 is the 100 left-parenthesis 1 minus alpha slash 2 right-parenthesisth percentile of the t distribution with df degrees of freedom. (For more information, see the section Degrees of Freedom.) The value of the confidence coefficient alpha is determined by the ALPHA= option; by default, ALPHA=0.05, which produces 95% confidence limits.

Kappa Details

The kappa component upper P Subscript o measures the observed agreement, and the component upper P Subscript e measures the chance-expected agreement. These values are displayed in the "Kappa Details" table when you specify the AGREE(DETAILS) or KAPPA(DETAILS) option. The "Kappa Details" table also displays the maximum possible value of the simple kappa coefficient given the marginal proportions of the two-way table. The maximum kappa is computed as

max left-parenthesis kappa right-parenthesis equals left-parenthesis max left-parenthesis upper P Subscript o Baseline right-parenthesis minus upper P Subscript e Baseline right-parenthesis slash left-parenthesis 1 minus upper P Subscript e Baseline right-parenthesis

where

max left-parenthesis upper P Subscript o Baseline right-parenthesis equals left-parenthesis sigma-summation Underscript i Endscripts min left-parenthesis ModifyingAbove upper N Subscript i dot Baseline With caret comma ModifyingAbove upper N Subscript dot i Baseline With caret right-parenthesis slash ModifyingAbove upper N With caret

The "Kappa Details" table displays the upper B Subscript n measure (Bangdiwala 1988; Bangdiwala et al. 2008), which is computed as

upper B Subscript n Baseline equals left-parenthesis sigma-summation Underscript i Endscripts ModifyingAbove upper N With caret Subscript i i Superscript 2 Baseline right-parenthesis slash left-parenthesis sigma-summation Underscript i Endscripts sigma-summation Underscript j Endscripts ModifyingAbove upper N With caret Subscript i dot Baseline ModifyingAbove upper N With caret Subscript dot i Baseline right-parenthesis

When the two-way table is 2 times 2, the "Kappa Details" table displays the prevalence index and the bias index. The prevalence index is the absolute difference between the agreement proportions, StartAbsoluteValue ModifyingAbove upper P With caret Subscript 11 Baseline minus ModifyingAbove upper P With caret Subscript 22 Baseline EndAbsoluteValue. The bias index is the absolute difference between the disagreement proportions, StartAbsoluteValue ModifyingAbove upper P With caret Subscript 12 Baseline minus ModifyingAbove upper P With caret Subscript 21 Baseline EndAbsoluteValue. For more information, see Sim and Wright (2005) and Byrt, Bishop, and Carlin (1993).

Weighted Kappa Coefficient

The WTKAPPA option (or the AGREE option) provides an estimate of the weighted kappa coefficient, its standard error, and the confidence limits. This statistic is available for replication variance estimation methods.

The weighted kappa coefficient is a generalization of the simple kappa coefficient that uses agreement weights to quantify the relative difference between categories (levels). By default, PROC SURVEYFREQ uses Cicchetti-Allison agreement weights to compute the weighted kappa coefficient; if you specify the WTKAPPA(WT=FC) option, the procedure uses Fleiss-Cohen agreement weights. For information about how the agreement weights are computed, see the section Kappa Agreement Weights. For more information, see Fleiss, Cohen, and Everitt (1969) and Fleiss, Levin, and Paik (2003).

For 2 times 2 tables, the weighted kappa coefficient equals the simple kappa coefficient; PROC SURVEYFREQ displays the weighted kappa coefficient only for tables larger than 2 times 2.

PROC SURVEYFREQ computes the weighted kappa coefficient as

ModifyingAbove kappa With caret Subscript w Baseline equals left-parenthesis upper P Subscript o left-parenthesis w right-parenthesis Baseline minus upper P Subscript e left-parenthesis w right-parenthesis Baseline right-parenthesis slash left-parenthesis 1 minus upper P Subscript e left-parenthesis w right-parenthesis Baseline right-parenthesis

where

upper P Subscript o left-parenthesis w right-parenthesis Baseline equals sigma-summation Underscript i Endscripts sigma-summation Underscript j Endscripts w Subscript i j Baseline ModifyingAbove upper P With caret Subscript i j
upper P Subscript e left-parenthesis w right-parenthesis Baseline equals sigma-summation Underscript i Endscripts sigma-summation Underscript j Endscripts w Subscript i j Baseline ModifyingAbove upper P With caret Subscript i dot Baseline ModifyingAbove upper P With caret Subscript dot j

where w Subscript i j is the agreement weight for table cell (i, j), ModifyingAbove upper P With caret Subscript i j is the estimate of the proportion in table cell (i, j), ModifyingAbove upper P With caret Subscript i dot is the estimate of the proportion in row i, and ModifyingAbove upper P With caret Subscript dot i is the estimate of the proportion in column i. For information about how PROC SURVEYFREQ computes the proportion estimates, see the section Proportions.

The weighted kappa component upper P Subscript o left-parenthesis w right-parenthesis measures the observed agreement, and the component upper P Subscript e left-parenthesis w right-parenthesis measures the chance-expected agreement. These values are displayed in the "Weighted Kappa Details" table when you specify the AGREE(DETAILS) or WTKAPPA(DETAILS) option.

PROC SURVEYFREQ estimates the variance of the simple kappa coefficient as described in the section Replication Variance Estimation. PROC SURVEYFREQ computes confidence limits for the weighted kappa coefficient as

ModifyingAbove kappa With caret Subscript w Baseline plus-or-minus left-parenthesis t Subscript d f comma alpha slash 2 Baseline times normal upper S normal t normal d normal upper E normal r normal r left-parenthesis ModifyingAbove kappa With caret Subscript w Baseline right-parenthesis right-parenthesis

where normal upper S normal t normal d normal upper E normal r normal r left-parenthesis ModifyingAbove kappa With caret Subscript w Baseline right-parenthesis is the standard error of the weighted kappa coefficient and t Subscript d f comma alpha slash 2 is the 100 left-parenthesis 1 minus alpha slash 2 right-parenthesisth percentile of the t distribution with df degrees of freedom. (For more information, see the section Degrees of Freedom.) The value of the confidence coefficient alpha is determined by the ALPHA= option; by default, ALPHA=0.05, which produces 95% confidence limits.

Kappa Agreement Weights

PROC SURVEYFREQ computes the weighted kappa coefficient by using the Cicchetti-Allison form (by default) or the Fleiss-Cohen form of agreement weights. These weights are based on the scores of the column variable in the two-way table request. If the column variable is numeric, the column scores are the numeric values of the column levels. If the column variable is a character variable, the column scores are the column numbers, where the columns are numbered in the order in which they appear in the crosstabulation table.

PROC SURVEYFREQ computes Cicchetti-Allison agreement weights as

w Subscript i j Baseline equals 1 minus left-parenthesis StartAbsoluteValue upper C Subscript i Baseline minus upper C Subscript j Baseline EndAbsoluteValue slash left-parenthesis upper C Subscript c Baseline minus upper C 1 right-parenthesis right-parenthesis

where upper C Subscript i is the score for column i and c is the number of columns (categories). For more information, see Cicchetti and Allison (1971).

PROC SURVEYFREQ computes Fleiss-Cohen agreement weights as

w Subscript i j Baseline equals 1 minus left-parenthesis left-parenthesis upper C Subscript i Baseline minus upper C Subscript j Baseline right-parenthesis slash left-parenthesis upper C Subscript c Baseline minus upper C 1 right-parenthesis right-parenthesis squared

For more information, see Fleiss and Cohen (1973).

The agreement weights w Subscript i j are constructed so that w Subscript i i Baseline equals 1 for all i, and w Subscript i j Baseline equals w Subscript j i. For i not-equals j, the agreement weights must be nonnegative and less than 1, which is always true for character variables (where the scores are the column numbers). For numeric variables, you should assign numeric variable levels (scores) so that all agreement weights are nonnegative and less than 1.

You can assign numeric values to the variable levels in a way that reflects their degree of similarity. For example, suppose the column variable is numeric and has four levels, which you order according to similarity. If you assign the values 0, 2, 4, and 10 to the column variable levels, the Cicchetti-Allison agreement weights take the following values: w 12 = 0.8, w 13 = 0.6, w 14 = 0.0, w 23 = 0.8, w 24 = 0.2, and w 34 = 0.4. For this example, the Fleiss-Cohen agreement weights are as follows: w 12 = 0.96, w 13 = 0.84, w 14 = 0.00, w 23 = 0.96, w 24 = 0.36, and w 34 = 0.64.

To display the kappa agreement weights, you can specify the WTKAPPA(PRINTKWTS) option.

AC1 Agreement Coefficient

When you specify the AGREE(AC1) option, PROC SURVEYFREQ provides Gwet’s first-order agreement coefficient, AC1 (Gwet 2008), its standard error, and the confidence limits. This option is available for replication variance estimation methods.

The AC1 agreement coefficient is computed as

ModifyingAbove gamma With caret equals left-parenthesis upper P Subscript o Baseline minus upper P Subscript e left-parenthesis gamma right-parenthesis Baseline right-parenthesis slash left-parenthesis 1 minus upper P Subscript e left-parenthesis gamma right-parenthesis Baseline right-parenthesis

where upper P Subscript o Baseline equals sigma-summation Underscript i Endscripts ModifyingAbove upper P With caret Subscript i i, upper P Subscript e Baseline equals sigma-summation Underscript i Endscripts e Subscript i Baseline left-parenthesis 1 minus e Subscript i Baseline right-parenthesis slash left-parenthesis upper R minus 1 right-parenthesis, and e Subscript i Baseline equals left-parenthesis ModifyingAbove upper P With caret Subscript i dot Baseline plus ModifyingAbove upper P With caret Subscript dot i Baseline right-parenthesis slash 2. The component upper P Subscript o is the proportion of observed agreement, and the component upper P Subscript e left-parenthesis gamma right-parenthesis represents the proportion of chance-expected agreement. For more information, see Xie (2013) and Blood and Spratt (2007).

PROC SURVEYFREQ estimates the variance of the AC1 coefficient as described in the section Replication Variance Estimation and computes the confidence limits as

ModifyingAbove gamma With caret plus-or-minus left-parenthesis t Subscript d f comma alpha slash 2 Baseline times normal upper S normal t normal d normal upper E normal r normal r left-parenthesis ModifyingAbove gamma With caret right-parenthesis right-parenthesis

where normal upper S normal t normal d normal upper E normal r normal r left-parenthesis ModifyingAbove gamma With caret right-parenthesis is the standard error of the AC1 coefficient and t Subscript d f comma alpha slash 2 is the 100 left-parenthesis 1 minus alpha slash 2 right-parenthesisth percentile of the t distribution with df degrees of freedom. (For more information, see the section Degrees of Freedom.) The value of the confidence coefficient alpha is determined by the ALPHA= option; by default, ALPHA=0.05, which produces 95% confidence limits.

Prevalence-Adjusted Bias-Adjusted Kappa Coefficient

When you specify the AGREE(PABAK) option, PROC SURVEYFREQ provides the prevalence-adjusted bias-adjusted kappa coefficient (PABAK) (Byrt, Bishop, and Carlin 1993). This statistic is available for replication variance estimation methods.

This coefficient is computed as

ModifyingAbove kappa With caret Subscript a Baseline equals left-parenthesis upper P Subscript o Baseline minus 1 slash upper R right-parenthesis slash left-parenthesis 1 minus 1 slash upper R right-parenthesis

where upper P Subscript o Baseline equals sigma-summation Underscript i Endscripts ModifyingAbove upper P With caret Subscript i i and R is the dimension of the square, two-way table. The component upper P Subscript o is the proportion of observed agreement, and the component 1 slash upper R represents the chance-expected agreement. When the table is 2 times 2, ModifyingAbove kappa With caret Subscript a Baseline equals 2 upper P Subscript o Baseline minus 1. For more information, see Sim and Wright (2005), Xie (2013), and Holley and Guilford (1964).

PROC SURVEYFREQ estimates the variance of PABAK as described in the section Replication Variance Estimation and computes the confidence limits as

ModifyingAbove kappa With caret Subscript a Baseline plus-or-minus left-parenthesis t Subscript d f comma alpha slash 2 Baseline times normal upper S normal t normal d normal upper E normal r normal r left-parenthesis ModifyingAbove kappa With caret Subscript a Baseline right-parenthesis right-parenthesis

where normal upper S normal t normal d normal upper E normal r normal r left-parenthesis ModifyingAbove kappa With caret Subscript a Baseline right-parenthesis is the standard error of PABAK coefficient and t Subscript d f comma alpha slash 2 is the 100 left-parenthesis 1 minus alpha slash 2 right-parenthesisth percentile of the t distribution with df degrees of freedom. (For more information, see the section Degrees of Freedom.) The value of the confidence coefficient alpha is determined by the ALPHA= option; by default, ALPHA=0.05, which produces 95% confidence limits.

Last updated: December 09, 2022