Why your QA scores vary by auditor (and how to fix it)
Calibration sessions ensure QA scores are consistent across auditors. Learn how to run objective meetings that eliminate bias and improve agent trust in your data.
QA calibration sessions are the only way to ensure that an agent’s performance score is determined by their behavior rather than the specific person grading the call. Without a formal calibration process, the data in your quality management system is often just a collection of individual opinions, leading to inconsistent feedback and a total loss of agent trust. Calibration is the mechanism that aligns every auditor to the same standard, turning subjective observations into objective operational data.
Key takeaways
- Calibration eliminates auditor bias: By having multiple graders review the same call, you identify who is a "hard grader" and who is too lenient.
- Focus on the outliers: Efficient sessions prioritize interactions where scores vary by more than 5% to 10% rather than reviewing agreed-upon points.
- Blind scoring is mandatory: To get honest results, auditors must score the interaction independently before seeing how their peers rated it.
- AI streamlines the selection: Modern tools can flag high-variance calls, ensuring human calibration time is spent on the most complex or ambiguous interactions.
Why does QA score variance happen?
QA score variance happens because human interpretation is naturally subjective. Even with a well-defined rubric, one auditor might view a "missed greeting" as a minor oversight, while another sees it as a critical failure of brand standards. This inconsistency creates "grade inflation" or "grade deflation," making it impossible to compare performance across different teams or sites.
According to the Gartner Customer Service & Support practice, which tracks the maturity of support technologies through its Hype Cycle, the shift toward domain-specific AI is helping to standardize baseline compliance. However, for nuanced human interactions—like empathy or de-escalation—human calibration remains the final word on what "good" looks like. If Auditor A gives an agent an 85% and Auditor B gives the same call a 95%, the problem isn't the agent; it is the lack of a shared definition of success.
The workflow for a successful calibration session
To run a session that actually changes how people score, you must move away from "group-think" meetings where everyone agrees with the loudest person in the room. The goal is to surface disagreement, not suppress it.
1. Select the right interactions
Don't just pick random calls. Choose interactions that fall into the "gray areas" of your scorecard. This might include calls where an agent followed the process but the customer remained unhappy, or cases involving new product launches where the script hasn't fully matured. Designing QA Scorecards That Actually Improve Agent Performance requires testing those scorecards against real-world ambiguity.
2. Implement blind scoring
Distribute the call or chat transcript to all participants (QA leads, supervisors, and sometimes high-performing agents) 24 hours before the meeting. Every participant must score the interaction in your QA platform—whether that is Salesforce Service Cloud, Zendesk, or a specialized tool—without seeing the others' scores. This prevents "anchoring," where people subconsciously align their scores with the first person who speaks.
3. Facilitate the discussion around the delta
In the meeting, the facilitator should immediately display the scores. Do not spend time on sections where everyone gave the same mark. Focus exclusively on the "delta"—the points of disagreement. If one person marked "Empathy" as a 'Yes' and another as a 'No,' ask them to cite the specific timestamp or phrase that justified their choice. This forces a return to the evidence rather than a reliance on "gut feeling."
How to use technology to identify calibration needs
Manual calibration is time-consuming, which is why many teams only do it once a month. However, waiting 30 days to fix a scoring discrepancy means hundreds of agents may have received incorrect feedback in the interim.
Modern operations use conversation intelligence to scale this process. For example, teams pair a CCaaS platform like Five9 or Genesys with a conversation-intelligence layer such as Hear.ai (powers our network). These systems analyze 100% of calls, allowing QA managers to see if certain auditors are consistently flagging things differently than the AI or their peers. If the automated system identifies a pattern of "soft skills" failures that a human auditor is passing, that specific auditor is a candidate for a targeted calibration session. This moves the department toward The QA Playbook: Moving from 2% sampling to total coverage by using humans to calibrate the "exceptions" while the system handles the baseline.
Setting a "Variance Threshold"
You will never achieve 100% agreement on every single call, and trying to do so is a waste of resources. Instead, define an acceptable variance threshold. Most high-performing contact centers aim for a 5% variance. If the group's scores for a call are all within 5 points of each other on a 100-point scale, the team is sufficiently aligned. If the variance is 10% or higher, it indicates that the scorecard question is either too vague or the auditors need more training on that specific behavior.
Research from Metrigy suggests that companies that tie AI-driven insights to human-led quality processes see better alignment between internal metrics and external customer satisfaction. Calibration is the bridge between those two worlds; it ensures that the "internal" score actually reflects the "external" reality of the customer experience.
The role of the calibration facilitator
The facilitator should not be the person who has the final say on the score. Their job is to remain neutral and ensure the discussion stays focused on the rubric. If the group cannot agree, the facilitator's role is to note the ambiguity and update the "QA Style Guide"—a living document that provides specific examples of what constitutes a 'pass' or 'fail' for every question on the scorecard.
FAQ
How often should we hold calibration sessions?
Weekly sessions are best for teams with new auditors or when launching new products. For established teams, bi-weekly sessions are usually sufficient to maintain alignment. The key is consistency; skipping sessions leads to "score drift" where auditors slowly revert to their own personal preferences.
Should agents attend calibration sessions?
Occasionally, yes. Bringing a rotating "Agent of the Month" into the session can demystify the QA process. It shows agents that scoring is a rigorous, thought-out process rather than an arbitrary judgment, which significantly improves buy-in for coaching sessions.
What if we can't agree on a score during the meeting?
If a group of experts cannot agree on a score, the fault lies with the scorecard, not the auditors. Use that disagreement as a signal to rewrite the question or the help text in the rubric to be more objective. If it's too subjective to be calibrated, it's likely too subjective to be measured fairly.
How many calls should be calibrated per session?
Aim for 2 to 3 calls in a 60-minute session. Any more than that, and the discussion becomes rushed. It is better to deeply align on two complex interactions than to skim through five simple ones.
Calibration is the backbone of a fair QA program. Without it, your data is noise. With it, you create a culture of accountability where agents know exactly what is expected of them and auditors have the confidence to provide feedback that sticks.
Explore our guide on designing effective QA scorecards to ensure your calibration sessions are built on a solid foundation.