Payment routing an agent can operate — evidence in the hot path, AI at the edges
A routing engine that decides which payment service provider should receive a card authorization, and what happens next if it declines — on empirical approval evidence, at a fee tolerance the operator states in percentage points. The thesis: the decision that moves money stays deterministic and auditable, and the language model lives only at the edges.
Skip the read — open the live demo or read the code
A card authorization is about to leave. It can go to any of four providers, each with a different fee and a different history of getting this kind of transaction approved — and if it comes back declined, something has to decide whether to retry, where, and when. That is not an infrastructure detail: there is money on both sides, approval on one and fees on the other, and the trade-off is a product decision that should be stated in units a human can argue about. This is the engine I built for it, and the thesis it demonstrates: evidence in the hot path, AI at the edges. It started from a technical exercise I generalized as a personal project; the data is synthetic and the generator ships with the repo.
The engine decides, deterministically
decide(txn, config) -> Decision is a pure function over pre-materialized tables. Five choices carry the design.
Offline/online split. The engine never reads raw payment events. A batch step reconstructs one row per authorization attempt and aggregates it into approval tables; the online path loads those tables once and decides in microseconds, no database in the request path. That makes the decision testable without infrastructure and auditable after the fact — the boundary you would draw in production between a reconciliation layer and a routing layer.
Segment hierarchy with fallback. The finest segment is channel × funding × issuer × amount band (L0); the coarsest is channel alone (L3). Support is resolved per provider, dropping one dimension at a time — amount band, then issuer, then funding — until the cell clears min_support (200 by default), and the level used is reported with the decision. The channel is never dropped: user-present and off-session are different worlds, and averaging across them is how a routing table starts lying.
Wilson lower bound, not the raw rate. A provider with 9/10 approvals in a cell is not a 90% provider. The bound shrinks toward zero as support thins, so a well-evidenced 870/1000 beats a lucky 9/10 without a confidence rule bolted on — “the best we are sure of” rather than “the best we have seen”.
cost_bias as a knob in real units. The trade-off is expressed as percentage points of approval the operator will give up for a cheaper provider: tolerance = cost_bias × 10 pp, and the cheapest provider within that tolerance of the best approver wins. A blended score would let a fractions-of-a-point fee difference silently outvote a double-digit approval gap; a tolerance filter cannot.
Retry by error class, not by a blind counter. insufficient_funds is an account problem, not a provider problem, so off-session it retries the same provider on the next billing window. bank_auth_required off-session cannot be satisfied without the customer, so the chain stops and asks for a reschedule to a user-present channel. fraud_risk and invalid_card_info stop permanently — low marginal recovery, real cost per attempt. generic_decline fails over by score, and an unrecognized class degrades to that policy and says so. Attempt caps differ by channel: 8 user-present, 20 off-session.
The receipt: an out-of-sample replay on 84,011 test transactions (days 22–31, tables trained on days 1–21) puts expected approval at 72.02% against 66.13% actually observed at cost_bias=0 — +5.89 pp. The knob is priced too: 71.56% / +5.43 pp at 0.5, 69.57% / +3.44 pp at 1.0. Directional numbers, not an A/B result — historical routing was not randomized, capacity is not modeled, and “expected approval” is a train-period Wilson bound applied to test volume. Those caveats ship in the summary file, next to the headline.
Where AI belongs — and where it does not
There is no LLM inside decide(), and there will not be. Money should not move on a sampled token: latency, auditability and the post-mortem all argue the same way. The model is confined to the two edges where language really is the problem. And even there, hallucination is treated as a certainty to design for, not a bug to hope away: the output schema constrains the answer to the enum, an explicit gate rejects anything outside it, low confidence falls to a safe default the retry machine already understands, and the eval counts every violation — the number has to be zero for the build to pass.
In — the decline normalizer. Each provider declines in its own dialect: ISO 8583 numerics, a Stripe-like decline_code, an Adyen-like refusalReason, or raw bank prose with no code at all. The retry state machine keys on one enum of six classes, so the dialects have to collapse before the engine sees them. A deterministic table handles the codes that carry volume — confidence 1.0, no latency, no cost. Only a table miss reaches the model chain (Gemini first, Mistral second), which answers under a schema that constrains the output to the enum. Anything outside it, or below 0.6 confidence, is discarded in favour of generic_decline — the retry policy’s own safe default. With no API keys set, the repo still runs green.
Out — the MCP server. Six tools — route_transaction, explain_decision, simulate, segment_evidence, normalize_decline, backtest_summary — let an analyst operate the engine in English while the decision stays deterministic. The useful shape is an incident: “psp-b just paged us, it’s down — where does a 250-unit debit checkout land instead, and what does that cost?” is simulate plus explain_decision with psps_down=["psp-b"]. “Is recurring routing to psp-c a real signal or three lucky transactions?” is segment_evidence, which answers with n and approvals per provider — a count, not an opinion. The agent interrogates every decision and changes none of them.
Same discipline as the remote MCP server and the eval harness on nutri.: give the model a bounded job, keep correctness in a layer it cannot reach, and measure the part that can be wrong in ways tests will not catch.
Evals as the gate
The normalizer is exactly that part, so it gets measured. The golden set is 48 declines: 28 table-route cases (58%) and 20 off-table (42%) — a misspelling from a changelog typo, verbose bank prose, unusual codes — including three deliberately ambiguous ones where generic_decline is the correct answer. The runner reports accuracy by route and a per-class confusion table, asserts zero hallucinations (a class outside the enum would reach the state machine as an unrecognized string; the eval asserts that gate still holds), and exits non-zero if accuracy drops more than 2 pp below the committed baseline.
Two baselines are recorded, and they are scored against themselves, never against each other. Table-only, keys off: 32/48 = 66.67% overall, 28/28 on the table route, and the safe generic_decline default on the 20 that fall through — the floor with the model switched off, not a measurement of the model. LLM, keys on, run against the deployed API so the keys never leave the server: 48/48 = 100% — 28 by the table, 19 by gemini-3.6-flash, one by the low-confidence fallback on a case where generic_decline was the expected answer. Forty-eight cases and one run is a regression gate, not a claim about the long tail; publishing it as a ceiling would be the same mistake as calling a model good because its output looked fine.
What the demo gives a reviewer
For a recruiter with two minutes: eight transactions, each sitting on a decision boundary — a hierarchy fallback, a cost knob that flips the route, an off-session retry that returns to the provider that just declined, a fraud hard stop, a provider marked down mid-incident, an unseen BIN, an amount exactly on a band edge. One click each, and the decision comes back with its reasoning lines.
For someone who owns payments: the controls are live. Move cost_bias and watch the fee-tolerance sweep; mark a provider down and watch the pool re-score; open the evidence panel and read level, n, approvals, raw rate and Wilson bound per provider. The same functions sit behind the HTTP API (/api/decide, /api/simulate, /api/evidence, /api/normalize, /api/backtest) and the MCP config — both thin wrappers over one shared module, not two implementations that can drift.
How it was built
Inside Claude Code, end to end. One orchestrating session planned the redesign and delegated the work to subagents — one for the engine’s runtime and the AI edge layer, one for the web UI, one to publish repo and deployment, one for this write-up — and verified each result against the tests, the evals and the live endpoints before accepting it. It is the same operating system I run nutri. on: convention files the agent must load every session, structural guardrails instead of guidelines, and a passing test suite as the only definition of done. The thesis of the project — deterministic system in the hot path, model at the edges, evals as the gate — is also how it was made.
The guardrails earned their keep on this project, and the incidents are the evidence. Post-deploy verification found the Vercel rewrite dropping every API path — every endpoint answered 404 — and it was fixed before the URL was shared. UI verification against the engine’s own segment resolution caught the demo mislabeling five of the eight demo issuers as “unseen”. A subagent handed a page-count rule that did not apply to these PDFs refused to cut content and reported that the premise was wrong. Another, told that DNS lived on Vercel, checked, found Cloudflare, and stopped rather than touch account settings. None of that came from a model being asked to be careful; it came from checks that run whether or not anyone is careful.
Limitations
Synthetic data, built to make routing non-trivial rather than to reproduce any real portfolio. No live provider connectors: the engine decides, it does not send. No 3DS orchestration, network tokens, or scheme retry-rule enforcement. The backtest is directional, for the reasons above. And one asymmetry worth naming: the tables pool all attempts while the backtest trains and replays on first attempts only — first-attempt-only production tables are the next fix.
Run it yourself
Live demo: orchestrator.vryahn.com · Repo: github.com/vryahn/payment_orchestrator · MCP setup: MCP.md
Locally, the eight boundary cases run in one line: python cli.py --txn-file demo_transactions.json.
The PM takeaway: “where does AI belong in this product?” is answerable only once you can say which decision must stay deterministic, and why. Here the answer is the architecture — an auditable engine in the hot path, a model on the two edges where the input really is language, and an eval standing between the model and the state machine.