Cohen's Kappa Calculator
Use this free online tool to measure inter-rater reliability between two raters classifying the same items into categories. Enter your contingency table counts, then click calculate.
What is Cohen's kappa?
Cohen's kappa (κ) is a statistical measure of agreement between two raters who each classify the same set of items into mutually exclusive categories. Unlike simple percent agreement, kappa adjusts for the agreement that could occur just by chance.
In practice, this makes kappa a better reliability metric for many coding, annotation, diagnostic, and screening tasks where raters may agree often simply because one category is very common.
How the kappa formula works
The formula is:
κ = (Po - Pe) / (1 - Pe)
- Po = observed agreement (the diagonal total divided by all ratings)
- Pe = expected agreement by chance (computed from row and column marginals)
If raters agree no better than chance, kappa is around 0. If agreement is perfect, kappa is 1. Negative values indicate agreement worse than chance.
How to use this online calculator
- Set the number of categories.
- (Optional) Enter category names like Positive, Neutral, Negative.
- Click Generate Matrix.
- Fill in counts where rows represent Rater A and columns represent Rater B.
- Click Calculate Kappa to get κ, observed agreement, expected agreement, and interpretation.
Matrix input meaning
Each cell is the count of items given a pair of labels by the two raters. For example, the cell in row "Category 1" and column "Category 2" is how many items Rater A marked as Category 1 while Rater B marked as Category 2.
Interpreting kappa values
A commonly used interpretation scale is:
- < 0.00: Less than chance agreement
- 0.00 to 0.20: Slight agreement
- 0.21 to 0.40: Fair agreement
- 0.41 to 0.60: Moderate agreement
- 0.61 to 0.80: Substantial agreement
- 0.81 to 1.00: Almost perfect agreement
Interpretation should always be contextual. A kappa of 0.60 may be excellent in a difficult clinical coding task but weak in a simple binary quality-control process.
Cohen's kappa vs percent agreement
Percent agreement is easy to understand, but it can be misleading when categories are imbalanced. For example, if almost all items belong to one class, two raters can agree frequently even if their reliability is mediocre. Kappa corrects for this by subtracting chance agreement.
Common pitfalls when calculating kappa
- Using categories that are not mutually exclusive
- Including missing or unclear ratings as if they were valid categories without a plan
- Ignoring strong class imbalance
- Interpreting kappa without reporting sample size and contingency table details
When should you use weighted kappa instead?
Standard Cohen's kappa treats all disagreements equally. If your categories are ordered (for example: mild, moderate, severe), weighted kappa is usually better because it gives partial credit for near agreements and larger penalties for far disagreements.
Quick FAQ
Is this calculator for exactly two raters?
Yes. Cohen's kappa is defined for two raters. For more raters, use Fleiss' kappa or related methods.
Can I use decimal values in the matrix?
You can, but counts are typically integers because they represent number of items.
What if kappa is undefined?
If expected agreement is 1.0 (denominator becomes zero), kappa cannot be computed in the usual way. This often happens when there is no variation in ratings.