Quality and analytics
Quality review on every conversation
How every conversation is scored against the standard your team writes, and how the ones that fall short reach a person for review.
On this page
Most quality teams check a handful of conversations per person each month and hope the sample is representative. Telonic scores every conversation against the quality standard your own team writes. Your quality lead sees how the whole month went, not a sample of it, and the conversations that fall short of your standard are routed to a person to review. Consistency stops being something you assume and becomes something you can show.
From a sample to every conversation
Sampling tells you how a few conversations went. Scoring every conversation tells you how your service went, and where it varies.
| Sampling | Every conversation scored | |
|---|---|---|
| What is reviewed | A small selection, chosen by a reviewer or at random | Every conversation the agent holds, on every channel |
| How problems are found | If one happens to fall in the sample | Each conversation below your threshold goes to a review queue |
| What the results show | An impression of quality | How many conversations met your standard, criterion by criterion, and which did not |
| Where reviewers spend their time | Listening to conversations that were fine | On the conversations that need a person's judgement |
How a conversation is scored
When a conversation ends, it is assessed against each criterion in your quality standard: whether the agent confirmed the customer's identity, answered from your approved sources, offered a person when it should, and so on. Each criterion is marked as met or not met, with the reason, and the results combine into one score using the weights your team set. The score, the result for each criterion and the reasons are stored with the conversation, so anyone reviewing it can see exactly why it scored as it did.
The standard is yours. Telonic helps you write it during implementation, starting from what a good conversation looks like in your industry, but the criteria, their weights and the threshold are decisions your team makes. See Writing a quality standard your team can check.
The technical detail
| Question | Answer |
|---|---|
| What is assessed? | The full conversation record: the transcript, the summary, the outcome, the actions taken and the decision log (what the agent looked up, and why it did what it did) |
| Where is scoring done? | In your deployment, in the region you chose, like all other processing. If you have chosen a language model outside your region, personal information is redacted before any text reaches it. See How data flows through a conversation |
| How is a change to the standard handled? | Each version of your standard is recorded, and every score records the version it was scored against, so scores from before and after a change are never mixed without you knowing |
| How long are scores kept? | With the conversation, for the retention period you set |
Scores against your threshold
Your team sets a threshold: the score below which a conversation needs a person to look at it. You can also mark individual criteria as critical, such as a required disclosure, so that a conversation missing one goes to review whatever its overall score.
Human calibration keeps the scores honest
Automated scoring earns its place when your quality team agrees with it. Before scores are used, your quality reviewers score a set of real conversations themselves, and we compare their results with the automated scores criterion by criterion. Where they differ, the criterion is usually worded too loosely, and we rewrite it with your team until the two agree.
Calibration continues after go-live. Your reviewers score a sample of conversations each month alongside the automated scores, and any drift is investigated and corrected. A change to the standard follows the same route as any other change: it is tested before it goes live. See How changes go live.
What happens to a conversation sent to review
Conversations below the threshold, or missing a critical criterion, appear in a review queue in the Quality view of the console. A reviewer opens the conversation and sees the transcript, the recording where the customer consented to one, the score for each criterion with its reason, and the decision log.
The reviewer records their own assessment and what should happen next. That might be a follow-up with the customer, a correction to one of your source documents, a change to a workflow, or a change to the standard itself. Every review is logged against the conversation. See The console and Audit trail and decision records.
A quality score measures a conversation against your standard. It is a signal for a person to act on, not a finding. Decisions about a customer, a complaint or a colleague stay with your team.
Conversations your team handles
Where conversations held by your own team are captured into the customer record, which is configured during implementation, they can be scored against the same standard. Your quality team then works from one standard across the agent and your people, and coaching can draw on what the best conversations have in common. See Conversations your team handles and Insights and trends.
In practice
An insurer runs the agent on its motor claims status line in the UAE.
- During implementation, its quality and compliance teams write a standard of eight criteria, including "Confirmed identity before sharing claim details" and "Gave the next step and when to expect it". Identity confirmation is marked as critical.
- Two quality reviewers score sixty conversations from testing. On one criterion, "Explained the excess correctly", they disagree with the automated result more often than on the others. The criterion is rewritten to name the policy schedule as the source, and the scores then agree.
- In the first full month, every conversation is scored. The review queue holds the conversations below the threshold of 70 and the few that missed a critical criterion.
- Reem, a quality lead, works through the queue. Several low scores share a cause: customers asking about courtesy cars, which the source documents do not cover. She asks for the courtesy car wording to be added.
- The next month, Reem compares the criterion results for courtesy car questions with the rest of the standard to check the new wording is working.
What your team controls
- The criteria in your standard, their weights, and which are critical.
- The threshold below which a conversation goes to review.
- Who reviews, and who can see scores, by role.
- When the standard changes, with each version recorded and tested before it goes live.