Cohen’s Kappa Calculator (2×2 Agreement Table)
Use this tool to measure how strongly two raters agree after correcting for chance agreement.
What is Cohen’s kappa?
Cohen’s kappa (κ) is a statistic used to evaluate agreement between two raters who classify items into categories. Unlike raw accuracy or simple percent agreement, kappa adjusts for the agreement that could happen by chance. This makes it a better choice when you want a more honest measure of inter-rater reliability.
Why percent agreement alone can mislead
Imagine two reviewers both label most items as “negative.” Their percent agreement may look high, even if they rarely agree on the harder “positive” cases. Kappa corrects this by accounting for expected random agreement based on each rater’s overall labeling tendencies.
How this calculator works
This page uses the standard 2×2 confusion-style table:
- a: both raters chose positive/yes
- b: rater A chose positive, rater B chose negative
- c: rater A chose negative, rater B chose positive
- d: both raters chose negative/no
From these values, the calculator computes:
- Observed agreement (Po) = (a + d) / N
- Expected agreement (Pe) from row and column marginals
- Kappa = (Po - Pe) / (1 - Pe)
Interpreting kappa values
A common interpretation scale (Landis & Koch) is:
- < 0.00: Poor agreement
- 0.00–0.20: Slight agreement
- 0.21–0.40: Fair agreement
- 0.41–0.60: Moderate agreement
- 0.61–0.80: Substantial agreement
- 0.81–1.00: Almost perfect agreement
These are guidelines, not strict rules. Context matters: clinical diagnosis, hiring decisions, legal coding, and machine-learning annotation projects may each require different reliability thresholds.
When to use a kappa calculator
Great use cases
- Two clinicians independently rating symptoms
- Two researchers coding qualitative text into binary labels
- Two reviewers classifying image findings as present/absent
- Comparing agreement between human labels and model labels
Important limitations
- Kappa can be sensitive to class imbalance (very rare outcomes).
- It assumes independent raters and fixed categories.
- For more than two categories or ordered categories, weighted kappa may be better.
- For multiple raters, consider Fleiss’ kappa or Krippendorff’s alpha.
Quick tips for better reliability studies
- Define labeling rules clearly before scoring.
- Train raters with examples and edge cases.
- Pilot-test agreement, then refine instructions.
- Report both percent agreement and kappa for transparency.
If you need stronger reliability, improve your codebook and rater training first. Statistical correction helps interpretation, but it cannot fix unclear definitions.