An agentic order-issue assistant for e-commerce
Concept case study. The company, data and numbers are fictional; the thinking is how I'd run it for real.
Summary: Basketly, a fictional European marketplace with third-party sellers, gets most of its support contacts about orders gone wrong. I designed an assistant that resolves them end-to-end, including taking actions like refunds, and escalates to a human exactly when it should. The core idea: measure true resolution, not containment.
The problem
- Most contacts are a few repeat issues: where is my order, return or refund, damaged or wrong item, and disputes with a third-party seller.
- Today's FAQ bot "contains" chats by deflecting them. Customers come back a few days later, angrier, and cost more to serve.
- Human agents spend their time copy-pasting tracking info instead of handling the cases that need judgment.
Users and what they need
| User | Need | What success looks like |
|---|---|---|
| Shopper | My issue fixed in one go | Resolved without contacting again |
| CS agent | Only cases that need a human, with full context | No re-asking the customer |
| Seller | Fair disputes, fewer escalations | Disputes settled with evidence |
| Business | Lower cost per contact without refund leakage | Cost down, refunds within policy |
Goals and metrics
- North star: true resolution rate. A chat counts only if it's closed by the agent and the customer doesn't contact again about it within 7 days.
- Supporting: correct-action rate, escalation precision and recall, CSAT, cost per contact.
- Guardrails: refunds outside policy, wrong actions taken, and time to a human when one is requested.
The design
The system is an orchestrator that routes each conversation to a specialist agent. Each specialist owns one job, its own tools and its own limits.
Only the returns agent can move money, and only up to its limit; anything beyond goes through the orchestrator to a person.
The decision that matters most: when to hand off
The assistant always hands off when:
- The customer asks for a human. No retention attempts.
- The action exceeds its authority, for example a refund above the auto-approve limit or an account with fraud signals.
- It's the customer's third contact about the same order.
- Safety or legal topics come up, such as a hazardous product.
- Its confidence in the intent or the answer is low after one clarifying question.
A handoff includes a summary, the order data and what was already tried, so the human never re-asks.
The routing decision and the handoff rule it triggers will show here.
Evals: the spec for the assistant
Before building, I'd write the eval set, because it defines "good" more precisely than a PRD can.
- Test cases drawn from real (anonymized) conversations across the main issue types, plus adversarial cases: refund-fraud attempts, prompt injection through order notes, and policy edge cases.
- Scoring: an LLM judge with a written rubric (resolved correctly, right action, policy followed, right escalation call, tone), calibrated against human ratings on a sample before we trust it.
- Escalation is scored both ways: a missed handoff is a failure, and so is an unnecessary one.
Release gates
Illustrative thresholds.
| Stage | Scope | Gate to move on |
|---|---|---|
| 1. Shadow mode | Agent drafts answers, humans send them | Judge and human ratings agree on 90% of cases |
| 2. Read-only pilot | 5% of order-status chats | True resolution beats the FAQ bot; zero policy violations |
| 3. Actions with limits | Returns and small refunds added | Correct-action rate of 98% or more; refund leakage within budget |
| 4. Scale | All order issues, all traffic | Stable for 4 weeks; escalation recall of 95% or more |
What I'd watch after launch
- Repeat contacts within 7 days, which reveal false containment.
- Refunds by agent versus by human, to catch leakage early.
- The handoff queue: are humans getting the right cases, with enough context?
- A weekly review of the worst-rated conversations, which feed back into the eval set.
What this shows about how I work
I start from the decision the agent has to get right, write evals before code, and roll out in stages where each stage has to earn the next.