AI field-service recovery where the AI only ever proposes, deterministic policy decides what is allowed, a human approves anything risky, and the customer fixes everything — pick a slot, share location, approve a quote, pay — inside one RCS chat thread. No app, no phone call.
Last updated 2026-10-20.
Built with Python, FastAPI, LangGraph, LlamaIndex, pgvector, MCP, RCS Messages API, Salesforce Field Service, RabbitMQ, Razorpay, Groq, PostgreSQL, Next.js 15, Logfire.
Field-service recovery is what happens when a booked service visit goes wrong — and it goes wrong constantly. Take home-appliance repair as the running example: the technician runs late, the part is not on the van, the unit turns out more broken than booked. The same shape repeats across any field-service business — HVAC, utilities, telecom, medical equipment — a scheduled visit becomes an exception that someone has to recover by hand. Today a human agent does that one case at a time, by phone and SMS, checking stock in one system and rebooking in another. It is slow, it drops cases, and it costs money on every exception. FieldFlow AI runs that whole recovery automatically — and lets the customer fix it from inside one chat thread.
When an appointment becomes at risk, FieldFlow AI detects it from an enterprise event, reasons over the full situation — customer, asset, warranty, technician, live stock, and the service manuals — proposes the fixes, removes every option that is not legally allowed, asks a human about anything risky, and then negotiates the fix with the customer entirely over RCS (Rich Communication Services — the upgraded SMS that carries rich cards, carousels, buttons, location and photo upload, pay links and PDFs). The customer picks a slot, shares location, photographs the unit, approves a quote and pays. The system then executes the real changes across Salesforce, inventory and payments.
The whole point — and the reason it is worth building rather than another chatbot — is where the AI is allowed to act and where it is not. The clever part (the model) is fenced in by the safe part (deterministic policy plus a human gate). That fence is what makes an AI system trustworthy enough to touch real operations and real money, and everything else on this page follows from it.
The reason the full toolkit earns its place — rather than being buzzword stacking — is that the system works by collecting live data from several enterprise systems and acting back on them. The facts are spread across Salesforce (the service domain: customer, asset, warranty, technician) and a separate e-com inventory service (live stock and price), so the AI needs a discoverable tool surface over more than one backend — that is MCP (Model Context Protocol, a standard way to expose tools a model can find and call). The reasoning needs knowledge that is in no database — which part fits which model, what the warranty covers, the repair procedure — so it needs RAG (retrieval-augmented generation, grounding the model in the actual service manuals). And because it acts on real systems and real money, it needs guardrails — a deterministic policy engine — plus a human in the loop for anything risky. Each piece is here because the problem put it there, not because it looked good on a diagram.
It is built as seven layers, and the governing rule is that each one has to answer "why am I here" in a single line or it gets cut. The anti-goal, named in the project's own principles, is technology soup. That rule is why this reads as a system and not a pile of impressive logos.
This is a portfolio POC. It fires end to end and delivered a real RCS card to a real Android phone on 9 October 2026, with the live orchestrator on Render, the e-com inventory service on Vercel, and the full AI pipeline (retrieval → model → policy → RCS) running. It never carried production traffic and every customer is fake, so this page carries no volume, uptime or latency figures. The Status tab says exactly what was and was not exercised.
Field-service recovery is the expensive, invisible half of a repair business. The booking is the easy part. The money leaks out of what happens when the booking goes wrong, and recovering each exception is manual, slow, and done by a human improvising across four systems.
A repair appointment has a dozen ways to go sideways. The technician is stuck in traffic. The part the job needs is out of stock at the nearest depot. The unit, once someone looks at it, needs a specialist. Each of these turns one scheduled visit into a recovery: find a new slot the customer accepts, confirm the right technician has the right skill, check whether a part exists and where, decide whether the repair is under warranty or chargeable, and keep the customer informed through all of it.
Today a human does that, one case at a time, by calling the customer, texting, checking stock in one system, rebooking in another, and chasing a payment in a third. It is slow, it drops cases when someone is busy, and the cost of the exception is paid on every single one. The booking flow gets all the software; the recovery flow gets a person with a phone.
The information needed to recover a case is real and it exists — it is just scattered. The customer and the asset and the technician live in Salesforce. The stock lives in a commerce or inventory system. The knowledge needed to reason — which part fits which model, what the warranty actually covers, the repair procedure — lives in PDF manuals nobody queries at 9am on a Tuesday. The money lives in a payment gateway. Nobody holds all four at once, so the human in the middle becomes the integration layer, by hand.
| System of record | Owns | Blind to |
|---|---|---|
| Salesforce Field Service | Customer, asset, warranty, appointment, technician and skills | Whether the part is actually on the shelf right now |
| Inventory / e-com | Product, price, and live stock at each location | Anything about the service visit or the customer |
| Service manuals (PDF) | Fault codes, part compatibility, warranty wording, repair SOPs | Anything live — it is static knowledge, not current state |
| Payment gateway | What was charged and captured | Why, or against which repair |
The tempting version of this project is a large language model with access to the Salesforce API, told to go fix appointments. That is exactly the version that cannot be allowed near real operations. A model that can reschedule, reserve stock and move money on its own judgement is one confident hallucination away from double-booking a technician, overselling a part, or charging the wrong customer. The failure is not a wrong sentence, it is a wrong action on a real system with real money.
So the actual problem is not "can a model talk to a customer." It is: how do you get the genuine value of a model — reading a messy situation, picking a sensible fix, explaining it like a human — without ever letting it be the thing that decides what happens? The answer this project is built around is a strict separation of proposing from deciding, and it is the subject of the next tab.
A recovery that still ends in a phone call has not removed the bottleneck, it has moved it. If the customer has to install an app to pick a new slot, most of them will not. The bet here is that RCS — rich, interactive messaging that already lives in the customer's default texting app on Android — is enough of a surface to carry the entire recovery: choosing a slot, sharing where they are, sending a photo, approving a cost, paying. If that holds, the human agent is removed from the loop entirely, not just assisted. Making RCS carry real actions rather than notifications is the second thing this project had to prove.
What this actually buys a service operation: the human agent stops being the integration layer between four systems, and the customer stops waiting on a callback. The agent's time moves from every exception to only the risky ones; everything else recovers itself, end to end, in the customer's chat thread.
The problem is that recovering a failed appointment is manual work spread across disconnected systems, paid on every exception. FieldFlow AI solves it by doing the cross-system work automatically — reading the customer, asset, warranty and technician from Salesforce, the live stock and price from the e-com inventory service, and the repair knowledge from the manuals — then proposing valid fixes, filtering them through policy, and handing the customer a set of choices they can act on themselves over RCS. The agent is not copying data between tabs any more; the system does that, and only surfaces the cases a human actually needs to judge.
| The recovery step | Today, by hand | With FieldFlow AI |
|---|---|---|
| Spot the at-risk appointment | An agent notices, or a customer complains | Detected automatically from a Salesforce event |
| Gather the context | Open Salesforce, the stock system, the manuals, one by one | Assembled in one pass across all three, grounded in the manuals |
| Work out valid options | Agent reasons it out, risks a wrong call | AI proposes, deterministic policy removes every illegal option |
| Agree a slot with the customer | Phone tag and SMS, several round-trips | A carousel of valid slots the customer taps once |
| Confirm location, part, quote, payment | More calls, manual data entry, chase the payment | Share-location, photo, quote card and pay link — all in the thread |
| Judge the risky cases | Every case competes for the agent's attention | Only safety, warranty and low-confidence cases are escalated to a human |
The saving is structural, not a trick. Three things that used to cost the agent time are removed. The cross-system lookup is gone — the system reads all four sources itself. The round-trips with the customer are gone — the customer self-serves the slot, location, photo, quote and payment in one thread instead of waiting for a human to call back. And the judgement load is cut — instead of handling every exception, the agent only sees the ones policy flags as risky or low-confidence, which is the small fraction where a human genuinely adds value.
Manual recovery gets worse as volume grows — more exceptions, proportionally more agent time. This moves the common cases to an automated flow, so success stops punishing the team. Growth costs compute, not headcount.
Every decision carries a full trace — the reasons, the knowledge sources, the tools used, the confidence and the policy result. A manager, an auditor or a compliance review can see exactly why the AI did what it did. That is rare for an AI system touching operations.
The model never acts on its own: policy decides, a human approves risk, and payments cannot be double-charged. The thing that usually blocks AI from real operations — the fear of a confident wrong action — is engineered out.
The reliability spine — idempotency, dead-letter queue, reconciliation, tracing — is built and live-verified, so this is a POC that could be piloted against real volume once the gated edges are armed, not a demo that falls over off the happy path.
The honest limit on this tab: I am not quoting minutes-saved or a rupee figure, because this ran as a POC on fake data and never carried production volume — any number would be invented. What is real is the mechanism of the saving (the cross-system work, the round-trips and the judgement load that are removed) and the structural wins above. Quantifying the ROI is exactly what a pilot against real cases would measure, and the decision trace plus the metrics are already wired to produce those numbers.
The AI proposes. Deterministic policy decides. A human approves risk. RCS is the customer's control plane. If you remember only that sentence you can predict where any feature in the system belongs — and why it is safe.
Every decision in the system runs down three levels of authority, and the order is fixed. The clever level sits in the middle, sandwiched by two levels it cannot overrule.
LEVEL 1 — DETERMINISTIC Can this action legally / technically happen?
permission · SLA · inventory · price · appointment · technician · state · idempotency
LEVEL 2 — AI What is probably happening? Which fix is best? What is missing? How do I explain it?
LEVEL 3 — HUMAN Should we allow a high-risk or low-confidence decision?
Level 2 — the model — only ever runs after Level 1 has said an action is possible, and its risky outputs are gated by Level 3 before anything executes. The model proposes candidate slots and drafts the message to the customer. It never reschedules, never reserves, never charges. Those are mutations, and a mutation is never the model's call.
This is the invariant the whole architecture exists to protect. The model emits a Proposal — a list of candidate options, a self-rating, and a short explanation. That Proposal is treated as untrusted input to a deterministic pipeline. The policy engine re-validates every option against hard rules. The action tools that actually touch Salesforce or inventory are plain validated functions, not an API handed to the model; the model may ask one to run, but the function decides the outcome and can refuse.
The practical test the project holds itself to: anyone who wires "let the model reschedule directly" has violated the one rule, no matter how good the prompt is. The authority lives in the function and the policy, never in the prose.
Because a human (or an auditor, or a sceptical manager) has to be able to trust this, every AI decision is recorded as a decision trace: the decision, the reasons, the knowledge sources it was grounded in, the tools it used, the confidence with its full breakdown, and the policy result. It rides on the live trace and is stored durably in Postgres. It is the "show your work" artefact, and it is the single strongest-built thing in the system — every live case carries the complete trace.
decision trace (per case)
{
decision: "offer 2 slots, same technician",
reasons: [ "technician_delay", "SLA window 240min", ... ],
knowledgeSources: [ { source: "Daikin XYZ-492 manual", score: 0.81 }, ... ],
toolsUsed: [ "salesforce.get_appointment", "inventory.find_part", ... ],
confidence: 0.88,
confidenceBreakdown: { factors: {...}, weights: {...} },
policyResult: { policyResult: "APPROVED", removed: [...] }
}
The reason this one idea matters beyond this project: it is the general shape of a trustworthy AI system. The value of the model is real, and it is kept. What is removed is the model's authority to act. Every hard engineering decision on the Engineering tabs is downstream of drawing that line and holding it.
RCS here is not a notification channel. It is where the customer acts. The rule the build held to: if a step can be done over RCS, it is done over RCS — the customer is never pushed to an app or a call for anything the thread can carry.
RCS (Rich Communication Services) is the carrier-grade successor to SMS. On a supported device — Android with Google Messages — the message thread becomes interactive: rich cards, swipeable carousels, tappable actions, inbound image and location, in-thread web views, calendar and dial actions, and, in India on Google Messages, PDF cards. A cloud RCS Messages API is how the system sends and receives all of that, and how it gets delivery and read callbacks and the fall back to plain SMS.
| RCS primitive | What the customer does with it |
|---|---|
| Rich card + action chips | View the appointment, add it to their calendar, ask to change the slot |
| Carousel (2–10 cards) | Pick one of several valid recovery slots by swiping and tapping |
| Share location (inbound) | Their phone sends GPS so the backend can work out technician ETA |
| Inbound image | Upload a photo of the unit's model label for vision + retrieval |
| Quote card + Open-URL web view | Approve and pay on the secure Razorpay page — RCS never sees card data |
| PDF rich card (India) | Open the service report inside the thread |
| RCS → SMS failover | Still receive the message if RCS is unavailable on their device |
at-risk event
→ AI investigates context + SLA
→ policy removes illegal options
→ carousel of valid slots (customer taps one)
→ share location (ETA confirmed)
→ upload a photo of the label (vision + RAG identify model / part)
→ stock reserved atomically
→ quote card
→ pay via web view (Razorpay test mode)
→ confirmation + calendar + PDF service report
The strongest moment in that chain is share-location. It is not us tracking the customer — it is the customer's phone choosing to participate in the workflow with one tap, feeding a real computation on our side. That is the clearest demonstration that RCS is a control plane and not a billboard.
Three things on this tab are POC stand-ins, and the project labels them as such rather than hiding them. The photo's OCR is simulated — the code returns the asset's known model and marks it matched, with a comment saying a real vision call swaps in at the gated live step. The share-location ETA is a chip the customer can tap, but the feasibility maths behind it was not built. The PDF service report is a placeholder link to a page, not a per-case generated document.
The distinction the project insists on is between the primitive and the payload. The RCS primitives that carry those three steps — the inbound image, the share-location chip, the PDF card — are real and were verified on a device. What rides on them is, in three specific places, a stand-in. Keeping that line honest is the whole reason the RCS story is believable at all.
What was built, what actually ran, and what I will not claim.
| Status | |
|---|---|
| The orchestrator | Built and deployed live on Render — FastAPI, LangGraph, the policy engine, the AI pipeline, the reliability spine. |
| The e-com inventory service | Built and deployed live on Vercel — Next.js + Supabase, 86 parts seeded, its own MCP server answering. |
| Salesforce | A real Field Service org: an Apex record-trigger fires the at-risk event as an HTTP callout; reads and reschedule run over Apex REST; 8 read-only Apex copilot tools deployed. |
| RCS on a device | A real carousel delivered to a real Android and the tap resumed the case (2026-10-09). The ongoing billed send is gated to one test device. |
| Production | Never. It ran end-to-end against real external systems with fake data; it never carried customer traffic. |
| Verification | 118 tests passing, linter clean, at the audited commit. |
One honest artefact the audit surfaced: the live /version endpoint 404s while the route exists in the code, which means the running deploy is behind the latest commit. It is listed here rather than hidden because "you cannot fully trust what is running" is exactly the kind of thing a case study should admit — a known gap, alongside the missing retry tier on the Reliability tab.
Seven layers, each with a one-line reason to exist. The constraint that kept it honest: if a layer cannot answer "why am I here" in a sentence, it is cut.
CUSTOMER (Android · Google Messages · RCS)
│ ▲ taps a card / carousel / PDF
▼ │
RCS Messages API (RCS send + real inbound & status webhooks) · FastAPI intake
│
RabbitMQ (retry · backoff · DLQ · idempotency)
│
▼
LangGraph Orchestrator (stateful · interrupt · resume)
┌──────────────────────────┼───────────────────────────┐
▼ ▼ ▼
Policy Engine MCP Tools (read + action) LlamaIndex RAG
(deterministic authority) │ (manuals · warranty · SOPs)
┌─────────┴─────────┐ │
▼ ▼ pgvector
Salesforce Field Service · e-com Service-Commerce
(service domain) (products+stock) (quotes/orders) ──▶ Razorpay (Test)
Underneath: PostgreSQL (state · idempotency · audit · AI decisions · pgvector)
Logfire (live request / case traces — the god-eye view)
| # | Layer | Tech | Why it is here |
|---|---|---|---|
| 1 | Experience | RCS | The customer acts in the moment without an app or a call. |
| 2 | Communication | RCS Messages API | Sends and receives RCS, reports delivery, fails over to SMS. The only layer that cannot be self-hosted. |
| 3 | Orchestration | LangGraph | A recovery case is long-running and pauses for the customer — it needs checkpoint, interrupt and resume. |
| 4 | Knowledge | LlamaIndex + pgvector | Salesforce has the data; retrieval supplies the knowledge — fault codes, SOPs, part compatibility. |
| 5 | Tools | MCP | Business systems as a small, discoverable, controlled tool surface, not fifty raw APIs in a prompt. |
| 6 | Enterprise | Salesforce FS · e-com · Service-Commerce · Razorpay | The service domain, product and stock, and a thin commerce seam for quotes and pay. |
| 7 | Reliability | RabbitMQ · idempotency · DLQ · reconcile · Logfire | What makes it credible as production rather than a demo. |
The sharpest design decision in the data model is that three systems own three disjoint things, and nothing is mixed. Salesforce owns the service domain — customer, asset, warranty, appointment, technician. The e-com service owns product data — image, price, stock. Our own PostgreSQL owns only the conversation and the AI — what we asked, what we decided, what we already processed.
The test for where a fact belongs is a one-liner the team used constantly: a "current appointment time" is Salesforce's; a "this part costs ₹4,800 and 7 are in Noida" is the e-com's; a "we sent the customer three slots at 09:30 and they have not replied" is ours. Mixing those up is the most likely early design mistake, and keeping them apart is why the inventory race condition later has a clean home to live in.
Every case-related record in the system is keyed by one correlation id — in the demo it equals the work-order id for legibility. It is stamped at the webhook or event gateway, carried through RabbitMQ, LangGraph, every tool call, every message and every audit row. The rule the code enforces: if you cannot answer "which case does this belong to?", you are missing the correlation id. It is what makes a single case one filterable trace from the inbound event all the way to the captured payment.
Everything runs on managed free tiers — Supabase for Postgres and pgvector, CloudAMQP for RabbitMQ, Groq and Gemini free tiers for the model, Render and Vercel for hosting. The whole system costs nothing to run, which is a deliberate POC constraint, not an accident. The one layer that genuinely cannot be self-hosted is RCS itself, which is exactly why a hosted RCS Messages API is the one managed layer in the stack.
A recovery case is not one request and one response. It is a workflow that spans minutes to hours, pauses for the customer to tap something, and must resume exactly where it stopped. That is the entire reason LangGraph is here rather than a web handler.
The naive version of this is a function that handles the at-risk event, decides, sends a card, and returns. But the customer's tap arrives seconds or hours later, on a different request, maybe after the server restarted. A plain handler has nowhere to keep "this case is waiting for a slot choice at version 3". You would end up hand-rolling a durable state machine with a wait-and-resume — which is precisely what LangGraph already is. Choosing it was choosing not to rebuild it badly.
LangGraph (a framework for stateful, multi-step agent workflows) checkpoints the graph state to Postgres, interrupts when it needs the customer or a human, and resumes on the next event. The case's position in the flow, what it is waiting for, and the conversation version all persist across a crash.
appointment.at_risk
│
▼
load_context (customer · appointment · technician · asset · warranty · inventory)
│
▼
retrieve_knowledge (RAG: manual · warranty policy · SOP)
│
▼
evaluate_sla
│
▼
generate_options ← LEVEL 2 (AI proposes candidate resolutions)
│
▼
policy_validate ◀──────────── LEVEL 1 (deterministic; removes illegal options)
│
├── all auto-safe ─────▶ offer_to_customer (RCS) ← interrupt; wait for the tap
│
└── risky / low-conf ──▶ human_approval ← LEVEL 3 (interrupt; wait)
│
▼
resume_on_reply ──▶ (reserve part · quote · payment · confirm · report)
│
▼
verify ──▶ close
There are exactly two places the graph pauses. It waits for the customer after offering options, requesting a photo, or sending a quote. It waits for a human when policy says the decision needs one. Both are real LangGraph interrupts, and both persist — the live app checkpoints to Postgres on the same Supabase database, so a paused case survives a restart and resumes instead of starting over. Tests and offline runs use an in-memory checkpointer so they need no infrastructure; the two sit behind one seam.
The sharp edge of a durable wait is: what if the saved state is gone — pruned, expired, or lost to a restart that predated the durable checkpointer? A resume that re-enters the graph with empty state would crash deep inside, reading an event that is not there. So every resume path first fetches the snapshot and, if there is no saved state, returns a clean 409 that says the case cannot be resumed, rather than throwing a 500 from the middle of the graph.
snapshot = await graph.aget_state(config)
if not snapshot.values:
raise HTTPException(409,
"Cannot resume: this case's paused state was lost or expired. Start a fresh case.")
The graph nodes never touch the database. A separate service layer owns durable state — the case row, the append-only audit trail, the decision trace — so the nodes stay pure and unit-testable. That split is why the decision logic can be tested offline with no infrastructure at all, and it is a large part of why there are 118 passing tests.
The deterministic authority. Plain code, no model. Given the options the AI proposed and the current context, it removes every option that cannot legally or technically happen — each with a reason — and no mutation in the system is allowed to skip it.
The engine is small on purpose — under a hundred lines — because it is the part that must be obviously correct. A candidate option is a plain dictionary, and only the keys it declares are checked, so a bare slot passes everything and a richer slot is held to every rule it names. It walks the checks in a fixed order and returns the first rule a candidate fails: terminal case state, technician skill, SLA window, working hours, warranty entitlement, part stock.
CHECKS = ["skill", "sla", "working_hours", "entitlement", "inventory", "state"]
# first rule the candidate fails, or None if fully valid
if case_state in TERMINAL_STATES: return f"case state {case_state} is terminal"
if skills and required_skill not in skills: return f"technician lacks required skill ..."
if start_minutes > sla_window: return "start breaches SLA window"
if not (9 <= hour < 18): return "start hour outside working hours"
if requires_warranty and warranty != "active": return "asset not under active warranty"
if part_stock <= 0: return "part not in stock"
The result is one of APPROVED (nothing removed), PARTIAL (some removed, some survive) or DENIED (nothing legal is left). The customer is only ever shown options that survived. An option the model loved that happens to be illegal — wrong skill, outside the SLA — is gone before it reaches the carousel.
One rule is softer than it looks, and the reason is a real bug it fixed. The skill check only fires when the technician's skills are actually known. With no assigned technician — an org that has not set up Field Service skills and territories — claiming a skill mismatch would be wrong, and worse, it would strip every skill-requiring slot and leave an empty offer. An empty offer is a 500 when the system tries to send a carousel with no cards. So the rule refuses to invent a mismatch it cannot prove.
# Only enforce the skill match when we actually KNOW the technician's skills.
# Without this guard, an appointment with no technician loses EVERY skill-requiring
# slot → empty offer → a 500 on send.
skills = set(context.get("technician", {}).get("skills", []))
if skills and (rs := cand.get("requiredSkill")) and rs not in skills:
return f"technician lacks required skill '{rs}'"
That is the difference between a rule that is strict and a rule that is correct. The strict version looks safer and quietly breaks a whole class of real org. The guarded version says less, and says only what it can prove.
A separate function decides when a human must approve, and it is not a single threshold. A human is required when any of three things is true: the reason is on a hard floor of always-human cases, or confidence is below 0.7, or policy left nothing legal to offer. The floor is the important one — a safety risk or a warranty dispute is never auto-approved on a model's confidence, no matter how high the score. That is how real systems tier decisions: some categories are human-only by law or by risk, regardless of how sure anything is.
def needs_human_for(reason, confidence, policy_result, always_human, threshold=0.7):
if always_human and reason in always_human: # safety_risk, warranty_dispute
return True # never auto-approved on confidence
return confidence < threshold or policy_result == "DENIED"
Everything here — the working-hours bounds, the confidence threshold, the always-human list — is tunable, and a calibration test locks the routing contract: a delay or a parts case routes to auto, a complex case routes to a human. The policy is not just code, it is code with a test that asserts which kinds of case reach a person.
The model's whole job is to propose and explain. Three things make that safe: a fallback ladder that cannot be taken down, a grounding step that strips anything the model invented, and a confidence score the model can lower but never inflate.
The proposer tries providers in a fixed order — Groq first, then Gemini, then a deterministic floor — and the first that returns a valid proposal wins. The deterministic floor never calls the network and never fails, so the decision loop cannot be taken down by a model outage or a rate limit. If a live provider is rate-limited, the note rides onto whatever rung eventually answers, so the trace shows the fallback happened and when a full retry is possible.
for name, provider in self.providers: # groq, then gemini (each only if its key is set)
try: return provider(reason, context, knowledge)
except RateLimited: note = "..."; continue # surfaced in the trace
except Exception: continue # any failure → next rung
return deterministic_proposal(reason, context) # the floor: no network, never fails
With no API keys at all — tests, offline runs — the ladder is just the deterministic floor, so the entire decision pipeline runs with zero external dependencies. That is why the AI logic is fully covered by tests that need no model.
The prompt asks the model to use only real part numbers from the asset's catalogue and never to attach a part to a pure scheduling delay. But asking is not enough — the parser enforces it. When the model's proposal comes back, a delay archetype has any part stripped, and any part number that is not in the asset's catalogue is stripped as invented. Only then does policy validate stock and skill on whatever survived.
# grounding ENFORCED in the parser, not just asked for in the prompt
if part and (archetype == "delay" or (valid_parts and part not in valid_parts)):
item.pop("partNo", None) # delay needs no part; an unknown part was invented
The line the project repeats is that the model can never smuggle in authority. It proposes; the parser grounds it out; policy validates it; the action function executes or refuses. Four independent gates, and the model is upstream of all of them.
The confidence that drives the human gate is not a single number the model emits — that would be un-auditable and gameable, since a model can simply sound sure to win. It is a weighted blend of five checkable factors, four of them deterministic signals already in the system and one from the model, and the model's slice is clamped.
| Factor | Weight | What it measures |
|---|---|---|
| Archetype prior | 0.40 | How hard the reason is — an easy delay scores 1.0, a complex job 0.40 |
| Policy headroom | 0.20 | How much legal room validation left — APPROVED 1.0, PARTIAL 0.6, DENIED 0.0 |
| Grounding | 0.15 | Strength of the retrieved knowledge |
| Data completeness | 0.10 | Fraction of the key case fields that are present |
| LLM self-rating | 0.15 | The model's own score, clamped to min(self, prior) |
The clamp is the point. The model can lower confidence but never raise it above what the job's difficulty warrants, so it cannot brag a risky case past the gate. And the prior for a complex job was tuned to 0.40 for a concrete reason worth seeing: at 0.40 the blended ceiling is 0.45 + 0.55 × 0.40 = 0.67, which stays under the 0.70 gate across the entire grounding range. At 0.50 the ceiling would be 0.725 — which would let a well-grounded complex job auto-send without a human. So the number is not arbitrary; it is set exactly so that the maths cannot route a complex case around the human.
The full breakdown — every factor, every weight, the final — rides in the decision trace, so the panel can explain why a confidence is what it is rather than just showing a number. A score you can decompose is a score you can trust; a bare 0.88 is not.
Two layers that are easy to confuse and were deliberately kept apart. Retrieval answers "what is generally true?" — what the manuals say. Tools answer "what is true right now, and may I change it?" — live data and validated actions.
RAG (retrieval-augmented generation — fetching relevant documents and feeding them to the model so it reasons over facts instead of its own memory) is the knowledge layer. Salesforce knows the data: this asset is a Daikin AC, warranty active. It does not know the knowledge: the service manual, the fault codes, which part fits which model, the warranty wording, the repair SOP. LlamaIndex chunks and embeds the manuals; pgvector (a vector-search extension inside the same Postgres, so there is no separate vector database to run) stores them; one retrieval per decision pulls the top matches for the current asset.
A retrieved chunk becomes a knowledge source in the decision trace with its score, so the demo can show the AI concluding "likely control-board failure, part PCB-492 compatible, warranty covered" grounded in a named manual and a warranty policy version — not asserted from nowhere. Retrieval is bounded to one call per decision; a chain of lookups would reintroduce the latency risk the whole design avoids. The live store holds 209 chunks and the ingest is incremental — add five pages anywhere and only those five re-embed.
MCP (Model Context Protocol — a standard way to expose tools to a model so it can discover and call them) exposes the business systems as a small, controlled surface instead of fifty raw APIs in a prompt. The split that matters is read versus action. Read tools are safe, idempotent lookups the model may call freely — get the customer, the asset, the technician, find a part, search a manual. Action tools are validated functions that mutate — reschedule, reserve, quote, capture payment — and the model may call one but the function decides the outcome.
READ (safe, idempotent — call freely) ACTION (validated; the function decides) salesforce.get_asset(assetId) reschedule.confirm(appointmentId, slotId) atomic salesforce.get_technician(resourceId) inventory.reserve(productItemId, qty) atomic; may fail inventory.find_part(partNo) commerce.create_quote(workOrderId, lines) amount from the price book knowledge.search_manual(model, query) commerce.capture_payment(paymentId) idempotent; no double-charge
There are two tool-use modes in the system and they are deliberately not the same mechanism. The deterministic recovery pipeline calls tools by name through an in-process toolbox backed by Apex REST and the e-com HTTP API — no model chooses the tool, policy decides. The admin copilot — a separate chat on the control panel where an operator asks open-ended questions across Salesforce and inventory — is the real MCP client, because an open-ended agent genuinely has to discover and pick tools by description. That is the one place MCP's discovery earns its keep, and it is why MCP is real here rather than decorative.
The guardrail holds across both: the copilot reads freely but any write stays human-confirmed, and the recovery pipeline never fires an unsupervised mutation. The e-com MCP server is live on Vercel; the Salesforce hosted MCP server and the copilot client are built with eight read-only Apex tools deployed, with the live arming of the hosted server deferred as named debt.
The classic mistake this separation avoids: putting live inventory in RAG, or putting the manuals behind an action tool. "How many PCB-492 are in Noida right now, reserve one" is a tool call. "Does PCB-492 fit this model, what is the SOP" is a retrieval. One is live and mutable, the other is static knowledge, and conflating them is how these systems rot.
The layer that makes it credible as production rather than a demo, and the part I would defend hardest. The contract, in one line: nothing is lost, nothing happens twice, and everything is explainable.
The failure that matters with money and bookings is doing something twice. A webhook retries, a customer double-taps, an event is redelivered. The system guards against it at two independent layers. At the envelope layer, every event and every payment carries a unique id, and the first thing the service does is check whether that id was already processed — if so, it is dropped and a duplicate counter ticks. At the gateway layer, the payment capture is itself idempotent, keyed by the payment reference: the first call charges, a repeat returns the same receipt marked already-captured and charges nothing.
# layer 1 — envelope dedupe (the event / payment id)
if await session.get(IdempotencyKey, event.eventId):
telemetry.events_duplicate.inc()
return {"status": "duplicate", ...}
session.add(IdempotencyKey(key=event.eventId))
# layer 2 — the gateway, independently (keyed by payment ref)
if payment_ref in self._captured:
return {**self._captured[payment_ref], "alreadyCaptured": True} # no double-charge
Two layers, because they fail differently. The envelope layer stops the graph from re-running at all; the gateway layer is the last line even if a bug somehow got two captures through. Fire the same payment event twice and it is charged exactly once — proven by a test and visible as a duplicate counter on the live server.
Every event rides RabbitMQ (a message queue), so a downstream outage — Salesforce temporarily down — never drops the work. When a handler crashes, the message dead-letters to a DLQ (dead-letter queue — a holding pen for messages that failed) rather than vanishing. A read-only peek shows what is parked; a replay reprocesses it once the downstream is back. The reconciliation pass flags any case stuck in a non-terminal state past a threshold. A parked message is not a failure the system hides — it is the proof that it did not lose the work.
When the graph crashes mid-case, the service does something deliberately more than dead-letter it. It first rolls back to drop the uncommitted idempotency key — so a replay can genuinely reprocess rather than being rejected as a duplicate — then writes a FAILED case row carrying the error and its type, then re-raises so the message still dead-letters. The result is that a failure shows up in the case list and the panel with the reason attached, instead of being only a silent entry in a queue you have to think to check.
except Exception as exc:
await session.rollback() # drop the uncommitted idempotency key so a replay can reprocess
await _persist_failed(session, cid, exc) # a FAILED row, error + type, VISIBLE in /cases
await session.commit()
raise # still dead-letters for replay
If the RCS card does not reach the customer — a delivery-status callback says failed, rejected or undelivered — and the case is still waiting for their tap, the same options are resent as a plain SMS listing each slot id so the customer can reply through the normal path. SMS has no carousel, so the fallback degrades the format, not the function. The message still lands and the workflow continues.
Observability is Logfire — live tracing built on OpenTelemetry (the vendor-neutral standard for traces), which fits a Python, FastAPI and Pydantic stack with almost no glue. One configure call plus auto-instrumentation gives, per case, the full trace tree: webhook intake, the queue publish, each graph node, every tool call, every SQL query, the model call — each a timed span, all tagged with the correlation id so one case is one filterable trace end to end. The decision trace rides on the span as attributes, so "why did the AI choose this" sits directly on the timeline. Prometheus counters sit beside it, fed by real callbacks, so the numbers are true rather than staged.
The one honest gap worth naming here: there is no retry-with-backoff tier before the DLQ, so a transient Salesforce blip dead-letters immediately rather than retrying a few times first. The DLQ, the replay and the reconcile are real; the gentle retry in front of them is the half of the requirement that was not built.
The unglamorous, accurate half of the RCS story: what it actually took to make RCS deliver a carousel to a real phone. None of this appears in a design document, and it is where a day disappears.
When the customer taps a slot in the carousel, the webhook that fires has to know three things with no other context: which case, which slot, and which version of the offer they were looking at. The design encodes all three into the suggestion chip's postback data as a single string, which the RCS API echoes back verbatim on the inbound webhook. So the webhook alone can resume the right case at the right slot and reject a stale tap — the versioning that guards against a customer replying to options that have since changed.
# each carousel card's reply chip
{"type": "reply", "text": "Choose",
"postback_data": f"{correlation_id}|{slot_id}|{card.version}"}
# the RCS API echoes that string back on the inbound webhook, so the webhook alone
# knows which case, which slot, and which offer version (the stale-reply guard).
The first real send failed, and the reason is the kind of thing you only learn against the real API. The RCS channel accepts only four suggestion chip types — reply, dial, open_url, share_location. Sending a generic action type is a 422 with error 1020, "Invalid suggestion." An open_url chip additionally requires a description field or it is a different 1020. A carousel card with no image is a 422, and the image must be a static, directly-addressable URL — Google's RCS backend fetches and validates the media itself and will not reliably render a dynamic placeholder, which is why an early carousel arrived with blank images.
So the card builder is careful in ways that look fussy until you have hit each wall: it maps each internal action to only the chip types RCS accepts, caps suggestions at four (the RCS limit), truncates text to the 25-character chip limit, converts phone numbers to E.164 for the dial chip, and drops the calendar and photo chips from the real send — there is no flat chip the POC data can form for them — so they render in the panel but never 422 the send.
# calendar + photo have no flat RCS chip we can form from POC data, so they # render in the panel but are OMITTED from the real send rather than 422 it. # share_location + dial are the headline "control plane" chips and go out for real.
RCS requires a different shape for each message kind, and getting one wrong is a 422. A standalone card — the payment card, the confirmation — needs a title, text, a media URL and height, and an rcs object with image alignment and orientation. A carousel needs card width instead. An empty card 422s, and critically, an empty carousel 422s too — which is the real reason the policy engine refuses to strip every slot and why a confirmation with no options is sent as a single standalone card, not a one-element carousel. Two separate build-time 500s traced back to accidentally sending an empty carousel.
RCS senders are tied to a registered application, so Basic auth cannot identify the agent — the real client signs a short-lived RS256 JWT with the application's private key for every call. And the real client only takes over from the in-memory fake when the send is fully armed: application id, private key, agent id and a test recipient all present. Real credentials can sit in the environment during development without a billed send ever firing, because the recipient is the last switch, set deliberately right before a demo. When the API rejects something, the client logs its exact complaint and the payload it sent rather than swallowing it — which is the only reason the 1020s above were diagnosable at all.
Every one of these is a real wire-level fact about RCS, learned by sending to a real device and reading the rejections. It is the least glamorous tab and the most useful one, because it is the part a demo hides and a second engineer inherits.
A thin service-commerce seam — parts, quotes, orders, pay, refund — not a storefront. The amount is set by a deterministic price book, never the model, and the one guard that matters is that a payment cannot be captured twice.
Money is integer paise everywhere — ₹1 is 100 paise — and never a float, because float money is how rounding bugs become charges. The quote amount comes from a deterministic price book, which is the authority on what a repair costs. The model never sets a price; it proposes which slot and whether a part is needed, and the price book turns that into a number. This is the authority ladder applied to money: the clever part suggests, the deterministic part decides the amount.
The capture is keyed by the payment reference. The first call charges and records a receipt; a repeat returns the same receipt marked already-captured and charges nothing more. That is the gateway-level half of the no-double-charge guarantee, independent of the envelope dedupe in the service layer — two separate guards, as the Reliability tab describes. The guarantee is backed by a runnable self-check in the file itself: capture the same reference twice, assert the second is marked already-captured and the ledger still holds one entry.
first = rp.capture("pay_1", 370000)
second = rp.capture("pay_1", 370000)
assert first["alreadyCaptured"] is False and second["alreadyCaptured"] is True
assert len(rp._captured) == 1, "a duplicate capture must not charge twice"
The payment gateway sits behind an interface so the real Razorpay swaps in for the in-memory fake with one wiring line. But the boot does something stronger than default to the fake: if a key is present and it is not a test key, the boot aborts. A POC literally cannot fire a real charge, because a non-test key raises on startup rather than quietly going live.
if not key_id.startswith("rzp_test_"):
raise ValueError("Razorpay: refusing a non-test key. POC is Test-Mode only.")
Payment reaches the customer as an Open-URL web view to the Razorpay page — the RCS "Approve & Pay" button opens it, and RCS never sees card data, which is the honest answer to "can you pay inside a text message" (you cannot, and you should not want to). Completion is confirmed by polling Razorpay rather than an inbound webhook — an explicit POC simplification — and in the real client the capture verifies the payment by fetching the link status, treating Razorpay as the source of truth rather than trusting the caller.
Live cases on the server carry a real test-mode short link and a captured payment id, so the commerce path is live-verified, not mocked — the only gated step is the single on-device tap-to-pay, which is the billed action held behind the arming switch like the real RCS send.
The decisions that shaped the system, each with the alternative it beat. The discipline, borrowed from the project's own decision log: a decision is recorded with the thing it rejected, and never silently rewritten.
| Decision | Why | Rejected alternative |
|---|---|---|
| AI proposes, policy decides, humans approve risk | The only shape that gets a model's value without its authority over real systems | A model with direct write access to Salesforce and payments |
| Three systems of record, no overlap | Stock realistically lives in a commerce system, not the CRM, and it matches the code's existing seam | Folding inventory into Salesforce; building our own domain tables |
| LangGraph as the orchestrator | A recovery case is long-running, interruptible and resumable — exactly what it is for | Hand-rolling a durable state machine; using an automation tool as the core engine |
| RAG and MCP kept separate | Knowledge and live-data-plus-actions are different problems | Using the retrieval framework as the orchestrator |
| MCP read / action split | Keeps dangerous operations as validated functions | Raw Salesforce API handed to the model |
| Live SF trigger = Apex record-trigger → HTTP callout | The orchestrator is already public, so a simple callout beats standing up a gRPC event subscriber | The heavier Salesforce Pub/Sub gRPC stream (built as a seam, deferred as debt) |
| Managed free tiers everywhere | Ship fast, ₹0, nothing to babysit | Self-hosting the queue, the database and the models locally |
| Fake everything | Removes privacy and compliance concerns whole | Real customer data in a POC |
The named enemy of the whole project is technology soup — impressive layers that cannot justify themselves. The system deliberately does not build Kubernetes, multiple clouds, a full storefront, a technician mobile app, real route optimisation, multi-tenancy or a real billing engine. None of those improve the thing being proven, which is the combination of RCS, a bounded AI, Salesforce, MCP, RAG, commerce, payment and reliability. If one started creeping in, the rule was to stop and re-check it against that list.
The one decision reversed mid-project is worth noting because it was reversed honestly: inbound webhooks were first skipped as a POC-speed call, then un-skipped once the orchestrator went public — the RCS inbound and status callbacks became real webhooks and the Salesforce trigger a real callout. The decision log marks the old entry superseded rather than deleting it, which is the same append-only discipline the whole project runs on.
Five things this project taught me that I would carry into the next AI system that has to touch something real.
I went in expecting the interesting work to be the prompting. It was not. The interesting work was the fence around the model: the policy engine, the grounding step that strips invented parts, the confidence clamp, the human gate, the decision trace. The model's contribution is real and it is kept — reading a messy case, picking a sensible fix, explaining it warmly — but almost everything that made the system safe to show someone is deterministic code the model cannot overrule. If I had to brief someone building their first AI system that acts on real data, this is the thing I would tell them to design first.
Asking the model in the prompt to only use real part numbers is necessary and completely insufficient. The thing that actually works is the parser that strips any part the model invented before policy ever sees it, backed by policy that validates stock anyway. Every important constraint in an AI system needs a deterministic enforcement point downstream of the model, because the prompt is a request and the code is a guarantee.
The RCS design was the easy half. Making RCS actually deliver a carousel to a real phone — the four accepted chip types, the 1020 errors, the static image requirement, the empty-carousel 422, the E.164 phone format — was a sequence of specific wire-level facts you can only learn by sending to a device and reading the rejection. I would budget for that discovery time explicitly next time rather than treating the integration as wiring.
The decision I am most confident about on the reliability side is the cheap one: when the graph crashes, write a FAILED case row with the error attached so it shows up in the case list, then still dead-letter it for replay. It is far less clever than an auto-healing reconciler, and it is worth more, because a failure a human can see and replay is better than a failure the system quietly tries to fix. The gap I would close first is the matching one — a retry-with-backoff in front of the DLQ, so transient blips never reach it.
The thing I am proudest of is not a feature, it is the production-readiness audit — a pass that treats the code as truth and the live server as the tiebreaker, scores mocked features at zero, and publishes the gaps with the fixes. It is what lets me put a real number on the system (about 79 percent, with the reliability spine at 85 to 90) instead of a vibe, and it is why every stand-in on this page is labelled rather than hidden. A system you can describe honestly is one you actually understand, and the audit is how I made sure I did.