Coding and Intercoder Reliability
Outline
Data Coding Basics
Example Codebook (White collar lawyer project)
What’s intercoder reliability
Why does it matter?
Example: binary var coding results of two coders
Use SPSS to calculate Cohen’s Kappa
Acceptance standard: Neuendorf (2002)
How to improve Intercoder reliability (Lombard et. Al. 2002)
Reference

Mathematical measures that are commonly reported on intercoder reliability
Popping (1988) identified 39 different “agreement indices” for coding nominal categories.
Commonly used ones:
Percent agreement: PA0=totalAs/n
Scott’s pi (p): p=(PA0-PAe)/(1-PAe) [when PAe=Sigma(pi_squared)]
Cohen’s kappa (k): k=(PA0-PAe)/(1-PAe) [when PAe=(1/n_squared)*Sigma(pi_squared)]
Krippendorff’s alpha (a): (Krippendorff’s Alpha 3.12a software)

There is no consensus on a single, “best” one.
Percent agreement is widely used, but is misleading. Tends to over estimate reliability.
Cohen’s Kappa is being criticized but still the most frequently used.
Hand calculations:
http://astro.temple.edu/~lombard/reliability