Cohen’s Kappa Calculator
Use this tool to calculate inter-rater agreement between two raters on a binary classification task. Enter counts in the 2×2 table below.
| Rater B | ||
|---|---|---|
| Rater A | Positive | Negative |
| Positive | ||
| Negative | ||
Formula: κ = (Po - Pe) / (1 - Pe)
What is Cohen’s Kappa?
Cohen’s Kappa (κ) is a reliability statistic that measures how much two raters agree, while correcting for agreement that could happen by chance. If you are labeling survey responses, clinical diagnoses, sentiment tags, or any yes/no outcome, kappa gives a more honest score than simple percent agreement alone.
Why not just use accuracy or agreement rate?
Raw agreement can look high even when raters are mostly guessing the same dominant category. Cohen’s Kappa adjusts for this by estimating expected chance agreement. This makes it especially useful when class distributions are imbalanced.
How to use this kappa cohen calculator
- Step 1: Fill in the 2×2 table with counts from both raters.
- Step 2: Click Calculate Kappa.
- Step 3: Review observed agreement (Po), expected agreement (Pe), and final κ value.
- Step 4: Read the interpretation band (slight, fair, moderate, substantial, etc.).
Interpreting kappa values
A common interpretation guide (Landis & Koch) is:
- < 0.00: Poor agreement
- 0.00 – 0.20: Slight agreement
- 0.21 – 0.40: Fair agreement
- 0.41 – 0.60: Moderate agreement
- 0.61 – 0.80: Substantial agreement
- 0.81 – 1.00: Almost perfect agreement
These thresholds are guidelines, not rigid rules. In medical or legal contexts, a kappa considered acceptable in one domain may be too low in another.
Worked example
Suppose two reviewers classify 100 documents as relevant or not relevant. If they agree on 85 of 100 cases, raw agreement is 85%. But if expected agreement by chance is 49%, then:
κ = (0.85 - 0.49) / (1 - 0.49) = 0.7059
That result indicates substantial agreement.
Best practices for inter-rater reliability
- Train raters on a shared codebook before production labeling.
- Pilot on a small sample and revise unclear instructions.
- Track disagreement patterns, not just overall kappa.
- Recalibrate regularly during long annotation projects.
- Use weighted kappa for ordered categories (e.g., severity scales).
Limitations to keep in mind
Cohen’s Kappa is sensitive to prevalence (class imbalance) and rater bias (different base rates used by each rater). A low kappa does not always mean poor raters; sometimes it reflects difficult data or skewed category frequencies. For multi-class data with more than two raters, consider related measures such as Fleiss’ Kappa or Krippendorff’s alpha.
Conclusion
This calculator gives you a fast, practical way to evaluate inter-rater consistency. Use kappa alongside confusion matrices and error analysis for a complete view of annotation quality. High-quality labels lead to better decisions, stronger models, and more trustworthy research outcomes.