Cohen’s Kappa (κ) Calculator
Use this tool to estimate inter-rater agreement for two raters and two categories (for example: Yes/No, Positive/Negative).
Formula: κ = (Po - Pe) / (1 - Pe) where Po is observed agreement and Pe is expected agreement by chance.
What is the kappa index?
The kappa index (usually written as Cohen’s kappa, κ) measures how much two raters agree beyond chance. If two reviewers classify the same set of items, they may agree simply by luck. Kappa adjusts for that and gives a more realistic estimate of reliability.
In practical terms, kappa helps you answer this question: “Are these raters truly consistent, or just coincidentally matching?”
Why not just use percent agreement?
Percent agreement is useful, but incomplete. Imagine two clinicians diagnosing a rare disease: both may say “No disease” most of the time and appear to agree at a high rate, even if they disagree on the critical positive cases. Kappa corrects for this by subtracting expected chance agreement.
- Percent agreement can be inflated when one category is very common.
- Kappa is stricter and often more trustworthy for quality and research reporting.
How to use this calculator
Step 1: Build your 2×2 table
Enter counts for each rating combination:
- a: both raters say Yes
- b: A says Yes, B says No
- c: A says No, B says Yes
- d: both raters say No
Step 2: Click “Calculate Kappa”
The calculator returns:
- Total sample size
- Observed agreement (Po)
- Expected agreement (Pe)
- Kappa score (κ)
- Plain-language interpretation
Interpreting kappa values
A common rule-of-thumb interpretation (Landis & Koch) is:
- < 0.00 = Poor agreement
- 0.00 to 0.20 = Slight agreement
- 0.21 to 0.40 = Fair agreement
- 0.41 to 0.60 = Moderate agreement
- 0.61 to 0.80 = Substantial agreement
- 0.81 to 1.00 = Almost perfect agreement
Interpretation should always be paired with context (sample size, class balance, and consequences of disagreement).
Worked example
Suppose two reviewers independently rate 100 records:
- a = 45
- b = 5
- c = 10
- d = 40
Observed agreement is high (85%), but kappa is lower than 0.85 because some agreement was expected by chance. In this example, κ is 0.70, which indicates substantial agreement.
When kappa can be misleading
Kappa is powerful, but not perfect:
- Prevalence effect: highly imbalanced categories can reduce kappa even when agreement seems strong.
- Bias effect: if raters have systematically different tendencies (one says “Yes” much more often), kappa changes.
- Two-rater limit here: this calculator is for two raters and binary categories only.
If your study has multiple raters or multiple ordinal categories, consider Fleiss’ kappa or weighted kappa.
Best practices for inter-rater reliability studies
- Use clear rating definitions and decision rules.
- Train raters with pilot examples before formal coding.
- Report both percent agreement and kappa.
- Include the raw contingency table for transparency.
- Discuss disagreements and refine the coding manual.
Quick FAQ
Is this Cohen’s kappa or Fleiss’ kappa?
This page calculates Cohen’s kappa for two raters and two categories.
Can I use decimals?
Counts are usually whole numbers, but the calculator accepts non-negative numeric values.
What does a negative kappa mean?
Negative κ means agreement is worse than chance. It can happen with systematic disagreement or data entry issues.