How we shipped a support agent that resolves 62% of tickets

Six months ago a subscription software client asked us for a support agent that could close tickets on its own, not just suggest replies to a human. Their team of nine was handling roughly 4,000 tickets a month, and median first response had crept past eleven hours. Today the agent fully resolves 62% of inbound tickets with a customer satisfaction score within two points of the human team. Getting there took three false starts, and those are more instructive than the final architecture, so this post covers both.

The architecture that actually shipped

The production system is deliberately boring. A ticket arrives from the helpdesk via webhook. A small classifier model tags intent and risk. If the ticket is in one of the fourteen intents we have approved for automation and the risk tag is low, it goes to the agent. Everything else goes straight to a human queue with a drafted summary attached.

The agent itself is a single model with a narrow toolset:

• Knowledge search over help center articles, internal macros and resolved tickets from the last twelve months, served from a retrieval index rebuilt nightly.
• Account lookup, read-only, returning plan, billing status, recent invoices and feature flags for the customer who wrote in.
• Four write actions: resend an invoice, extend a trial by up to 14 days, reset two-factor enrollment after identity checks, and apply a pre-approved credit of up to $25.
• Escalate, which hands off to a human with a structured summary and the reason the agent stopped.

Each write action is wrapped in server-side policy checks. The model can ask for a credit of $500; the tool will refuse and log the attempt. We treat the model as an untrusted caller of our own API, which removes an entire category of risk from the prompt.

We built this as a custom AI agent rather than configuring an off-the-shelf helpdesk bot because the value lived in the write actions and the account data, and no packaged product let us enforce policy at the tool boundary.

The three mistakes we made first

Mistake one: starting with the prompt

Our first week went into a long, carefully worded system prompt. It read well and demoed well. When we finally ran it against 300 real historical tickets, it resolved 31% correctly and confidently gave wrong answers on another 12%. We had no way to tell which prompt changes helped, because we had nothing to measure against. We threw away the week and built the evaluation set first.

Mistake two: letting the agent handle everything

Version two had no intent gate. The agent saw every ticket and was told to escalate when unsure. In practice, models are poor judges of their own uncertainty on edge cases. Refund disputes, legal threats and data deletion requests were occasionally answered with cheerful, plausible nonsense. Adding the classifier gate in front, with an explicit allow-list of intents, dropped harmful responses in our eval set from 4.1% to 0.3% and made the remaining failures far easier to reason about.

Mistake three: optimizing deflection instead of resolution

Our early dashboard counted a ticket as handled if the customer did not reply within 48 hours. That metric rewarded vague answers that made people give up. We switched to a resolution definition that required either a positive reply, a completed action, or no reopen within seven days combined with a sampled human review. The headline number dropped from 71% to 48% overnight, and then climbed honestly to 62% over the following three months.

A deflection number that goes up while customers get angrier is not a success metric. It is a slow-moving incident.

Evals and guardrails that keep it honest

The evaluation suite now holds 1,150 historical tickets with graded reference outcomes. Every prompt, retrieval or model change runs against it in CI, and a change that lowers correct resolution by more than one point, or raises the harmful-response rate at all, blocks the merge. Grading uses a mix of exact checks (did the agent call the right tool with the right arguments) and a rubric-based model grader that we calibrated against two human reviewers until agreement passed 90%.

In production we run three guardrails on every response before it is sent:

1. A grounding check that confirms any factual claim about pricing, limits or policy appears in a retrieved source.
2. A tone and commitment check that blocks promises the company has not authorized, such as refunds outside policy or delivery dates.
3. A sampling queue where 5% of agent-resolved tickets are reviewed by a human each day, with disagreements added to the eval set.

That last loop is the most valuable piece of the system. The eval set grows from real failures, not from what we imagined might go wrong.

What the numbers look like now

After six months, the agent handles about 2,480 tickets a month end to end. Median first response on automated tickets is under two minutes. The human team did not shrink; it moved to the 38% of tickets that need judgment, and their median handle time on those dropped because every escalation arrives with a summary and the relevant account data already pulled.

Running costs are modest. Model and retrieval spend averages about $0.09 per ticket, including the classifier and guardrail calls. The larger ongoing cost is people: roughly six hours a week of review and eval maintenance, which we consider non-negotiable. If you are weighing a simpler customer support chatbot against a full agent, that review budget is the real difference, not the model bill.

A few decisions we would make again without hesitation:

• Keep write actions few, small and enforced server-side.
• Gate by intent before the agent ever sees a ticket.
• Build the eval set before the first prompt, and never stop adding to it.
• Measure resolution with reopen windows and human sampling, not silence.

The retrieval layer deserves its own post, and we covered much of our approach in our write-up on building RAG for small teams. The short version is that indexing resolved tickets, not just help articles, was responsible for roughly a third of the resolution gains.

Planning an agent of your own

If your support queue looks like this client's did, the first step is a two-week scoping exercise: pull a sample of tickets, tag intents, and estimate which ones are safe and valuable to automate. We run that as a fixed-price engagement, and you will know the realistic resolution ceiling before committing to a build. Tell us about your queue and we will send a fixed-price quote within 24 hours.