Customer email in, approved reply out — and a log of everything between

The problem#

Customer email looks like a text problem. It's a workflow problem. Something has to decide what the message is about, find the one policy that applies, check what the customer actually bought, draft an answer in the company's voice, and know when to stop and hand it to a person.

Wire an LLM straight to the inbox and you get confident replies with no idea what's true: a refund policy invented, a customer's plan guessed, an escalation that never happened. The model isn't the hard part. The hard part is everything around it.

So this is the deliberately boring version: a production-style workflow with approvals, monitoring, retries, testing and cost visibility — the things that make an AI system something a business can actually run.

How does it work?#

email arrives → classify → retrieve knowledge → CRM lookup → external API → draft response → human approves or escalates → send (or simulated send) → audit log

One pass, one direction, and a human in the middle of it — not at the end as a rubber stamp.

intake

Customer email arrives, gets an ID, and starts a run.

classifier

Works out what the message is about, and how urgent it looks.

retrieval

Pulls the knowledge that applies — with a source reference attached to every claim in the draft.

crm_lookup

Checks who this customer is and what they own, instead of asking them again.

external_api

Where the answer lives outside the CRM, the run goes and gets it.

drafter

Writes the reply from the retrieved sources, not from memory.

approval_gate

A person approves, edits or escalates. Low confidence means escalation, never a guess.

audit_log

Input, category, sources, tools called, draft, decision, outcome — written down.

That last pair is the point of the whole design: the system is allowed to be slow and careful, and it is never allowed to be quietly wrong.

What did I build?#

1. Phase 1 — the core workflow. Synthetic customer emails, classification, knowledge retrieval, CRM lookup, an external API call, response drafting, human approval, send or simulated send, and the audit log.
2. Phase 2 — reliability. Retries, timeout handling, a fallback path, malformed-input handling, a simulated failed API, and queue or retry state where it's warranted.
3. Phase 3 — trust. Human approval, permissions, clear source references, an action log, and a safe fallback whenever confidence is low.
4. Phase 4 — evals. A small fixed test set with the expected category, expected escalation and expected data source for each case, plus the outputs that are simply not acceptable. It runs automatically and reports pass or fail.
5. Phase 5 — monitoring. One dashboard: runs, successes and failures, latency, token and model cost where it's measurable, escalations, retry counts.
6. Phase 6 — demo assets. A hero GIF, a 30-second video, an architecture diagram, a monitoring screenshot, an eval screenshot, and one failure-and-recovery example.

Synthetic data is fine. Fake performance claims are not.

Before and after#

None of these steps is exotic. Together they're the difference between a demo and something you'd let near real customers.

Technical depth#

• workflow orchestration
• structured outputs
• retries + timeouts
• eval harness
• monitoring dashboard
• audit log

The architecture is deliberately ordinary: a workflow layer that owns the state, models that only do the language steps, and deterministic code for everything else — retries, permission checks, logging and the send gate.

Model choice is a configuration detail rather than an architecture decision, because the eval set is what decides whether a change ships.

Production readiness#

This is the case study that only exists because of its safeguards. Without them there's nothing here worth showing.

1. Nothing sends without a person. Approval or escalation is a required step in the run, not a setting someone can turn off.
2. Sources travel with the draft. Every claim in a suggested reply points back to the knowledge or record it came from.
3. Low confidence escalates. The safe fallback is a human, and it's the default rather than the exception.
4. Failures are expected, not discovered. Retries, timeouts, malformed input and a failing external API are all in the test plan.
5. Everything is logged. Input, decision, tools, draft and outcome — so any reply can be explained afterwards.

The result#

The honest answer: a specified operating design and a phased build plan. No production traffic, no customers, no numbers. What exists is the part most AI projects skip — the failure paths, the approval gate, the eval set and the dashboard.

Those three are what the monitoring phase will report — written as what will be measured, because nothing has run in production yet.

When the demo assets land they'll show the architecture, the monitoring view, the eval run and a failure being caught and recovered — the four things a page like this normally can't prove.

30-second walkthrough#

Why this matters to an employer#

FAQ#