An AI agent as the fifth
member of the shift
My AI agent works on a round-the-clock line and handles about a third of the traffic. This is not a pilot on a carved-out sample and not a demo: it sits in the general queue and answers live customers. What matters here is not the technology but the management part: who owns its quality, which standard it is scored against, and what counts as a result.
What counts as a result
The first decision to make concerns the metric. "Percentage of conversations closed" looks convenient and means almost nothing: a conversation also closes when the customer leaves without an answer. I measure the share of contacts carried through to a resolution, meaning those where the customer's question was solved and they did not come back with the same one a day later.
On that metric the night line rose from 34% to 82%. The difference between the two ways of counting matters here: on "conversations closed" the figure would be higher and prettier, but you cannot make decisions with it.
Acceptance testing before going live
- A set of real scenarios assembled from contact history, not invented ones
- Twenty-nine conversations read in full, by hand, with no spot checking
- Acceptance criterion: not a single factual error about the product
- Boundary behaviour checked separately: what the bot must never do at all
The boundary of responsibility is defined in advance and strictly: refunds, disputes, claims and anything requiring a human decision go to an agent. The bot does not negotiate or persuade; it answers the question or hands it on.
The quality standard extends to the bot
This is the key decision, and it is not a technical one. The AI agent is covered by the same conversation scoring standard on sixteen parameters as people: the same full weekly sampling, the same scale, the same misses. The result is 97.7%.
There is deliberately no softer standard for the bot. As soon as automation is given its own, more lenient scale, the following happens: the share of contacts handed to the bot grows, while a comparable measurement of quality across the line disappears. A quarter later the manager can no longer answer whether the customer is better off.
- The bot's scores go into the shared line quality report, not into a separate AI presentation
- The bot's misses are reviewed exactly like an agent's and lead to an edit in the source of knowledge
- During an incident window the same parameters leave the scoring base as for people
The second loop: an assistant for agents
Alongside the autonomous agent I built a real-time AI assistant: it suggests an answer from the knowledge base to a live agent while the conversation is running. Coverage is over ninety per cent of routine contacts.
That loop is usually underrated, and it removes a newcomer's main pain: not "where is it written" but "which part of what is written applies to this customer". Together with the AI conversation simulator it shortened the time to get an agent onto the line.
Economics rather than enthusiasm
While the effect of AI is measured in the percentage of conversations closed, the conversation with finance does not work. It works when the effect is expressed in the same units the budget is built from: cost per contact and cost per minute of departmental work.
For me that is 213 roubles per contact and about 10 roubles per minute on the high-volume line, both within the same calculation boundaries, including the manager's payroll. From there payback is a matter of arithmetic, not an argument about how promising the technology is.