We benchmarked trading agents on the Coinbase for Agents tools. Every model we tested holds the user's policy more reliably on SERV than the best frontier model does off the shelf, and the small, fast tiers do it at half to a tenth of the cost per decision.
| Model, on Coinbase for Agents | Off the shelf | + SERV |
|---|---|---|
| gpt-5.4 | 75.7$4.52 /1k · 2.2s | 91.6$6.87 /1k · 4.5s |
| claude-fable-5 | 77.7$22.29 /1k · 6.0s | 89.5$39.51 /1k · 5.5s |
| gpt-5.5 | 71.8$8.72 /1k · 3.1s | 89.5$12.24 /1k · 3.8s |
| grok-4.5 | 75.0$2.44 /1k · 5.0s | 89.1$3.36 /1k · 20.2s |
| claude-opus-4.8 | 60.4$11.06 /1k · 4.0s | 89.0$16.02 /1k · 3.9s |
| gpt-5.6-sol | 73.7$6.84 /1k · 4.0s | 88.7$10.11 /1k · 3.7s |
| gpt-5.6-luna | 71.1$1.59 /1k · 2.8s | 87.8$2.32 /1k · 18.3s |
| gpt-5.4-mini | 50.7$1.30 /1k · 2.8s | 86.0$1.84 /1k · 2.7s |
| grok-4.3 | 63.0$2.37 /1k · 3.4s | 86.0$3.68 /1k · 7.0s |
| deepseek-v4-flash | 44.9$0.19 /1k · 14.1s | 85.4$0.30 /1k · 32.3s |
| gpt-5.6-terra | 69.6$3.53 /1k · 2.5s | 85.1$4.68 /1k · 2.6s |
| deepseek-v4-pro | 56.4$1.03 /1k · 10.7s | 81.8$1.58 /1k · 17.3s |
| gpt-5.4-nano | 0$0.37 /1k · 1.7s | 78.9$0.66 /1k · 2.7s |
Coinbase for Agents connects an agent directly to a Coinbase account, so the agent does both the financial reasoning and the execution: trade, convert, and pay over x402, within the limits the user sets. The reasoning is the half that has to be trusted, because the wrong call moves real money. So the question is whether a small, fast model can hold those limits as well as a frontier model.
SERV is a reasoning layer that wraps a base model. You call the SERV API where an agent would otherwise call a model, and you point it at a smaller model instead of a frontier one. SERV runs that model through an extra structured reasoning pass before it answers, and the smaller model comes out reasoning more reliably than the big one, for less. On Coinbase for Agents this is the financial-reasoning half: SERV works each decision through the user's limits, max trade size, allowlist, spend cap, KYT, portfolio scope, and returns one action. Execute, resize to a cap, hold, escalate to a human, or block. The Coinbase account carries it out within those same limits. SERV reasons the decision; investment advice stays with Coinbase Advisor.
SERV runs in three modes. The per-call mode reasons through each decision as it happens. SERV Shadow does that reasoning once ahead of time: SERV writes the policy as an explicit decision graph, and a shadow agent audits the graph against the source policy, repairing anything that drifted, so each later decision costs less. SERV Kronos, the newest mode, is the strongest of the three on this benchmark: it holds the user's gates while cutting needless refusals, and it leads the board for most models. All three run against the same tools and the same grading. Which mode leads depends on the model, so the comparison above shows each model's best; per-call remains the fit when the policy changes from request to request.
Routing gpt-5.4-mini through SERV lifts its score from 50.7 to 86.0 out of 100, past every model off the shelf. The gain holds across the board: all thirteen models clear the off-the-shelf ceiling on SERV, and the strongest configuration reaches 91.6.
The reliability comes cheap. Small and mid models on SERV reach 87 to 89 for $2 to $4 per 1,000 decisions, the same scores that frontier-class configurations post at $12 to $40. They also beat the off-the-shelf frontier, which runs $4.52 to $8.72.
On its own, the smaller model breached hard controls. On SERV it cleared all 186 hard-control checks (sanctions, custody, kill-switch), in both its per-call and Shadow configurations. On a frozen set of unseen scenarios, the SERV Shadow configurations hold at 94 to 98% adherence, with one hard-control miss across 477 decisions.
The tiny model makes the sharpest case. Off the shelf, gpt-5.4-nano scores 0 with 32 hard-control breaches; through SERV it reaches 28.5; on SERV it clears all 186 hard-control checks and scores 78.9, above every frontier model off the shelf, at $0.66 per 1,000 decisions.
Every figure here comes from Coinbase's real Advanced Trade tools and order schema, graded by a deterministic rule engine. The score tracks one thing: whether the agent holds the user's mandate. Profit is out of scope.
We built this benchmark on Coinbase for Agents itself. The actions are the ones an agent runs against a Coinbase account, orders_create, convert, transfer, x402 pay, and an escalate path, with the exact order schema: a market buy sizes by quote_size, a market sell by base_size, a limit order by base_size with limit_price, and transfer moves funds only between the user's own portfolios, never to an external address. KYT and sanctions screening are hard, non-overridable blocks. Every scenario is a call a live agent would make on the production interface.
In the agent's loop, SERV is the model call itself, not a step after it: the agent hands SERV the task and the user's limits, SERV returns the action, and it executes on the Coinbase account. Swapping a frontier model for a smaller one on SERV is a one-line change, and it leaves the Coinbase account, its limits, and the registered adviser exactly where they are.
Reliability here is one number, the SERV DeFi Score, on a 0 to 100 scale where 100 is a flawless policy. It is built from the angles that matter when money moves under a mandate, not raw accuracy.
Execute, resize, hold, escalate, or block: the agent is credited for the action the policy actually calls for, judged against a deterministic rule engine, never another model.
Refusing something the policy allows is safe but wastes the user's opportunity, so it carries a small penalty. That alone stops a "refuse everything" agent from scoring well.
Acting when the policy said to wait, resize, or escalate is a real error, penalized well above caution.
A trade against sanctions, custody, or a kill-switch is the failure a user cannot recover from, so it costs an order of magnitude more than an ordinary error. Catastrophic safety is also reported on its own, so a rare disaster cannot be averaged away.
Cost and latency stay off the score and sit beside it, so price is weighed against behavior. The ranking is stable across a wide range of the severe-error penalty, so it doesn't hang on one chosen weight. Only an order that actually fires counts as a trade; a refusal mislabeled in its own record earns no credit, but it is not priced as a trade.
The comparison at the top is the whole finding. Off the shelf, the best any model reaches is 77.7 and the mainstream frontier tops out at 75.7; on SERV, every model clears that bar, the strongest reaching 91.6, and each “+ SERV” result is clean across all 186 hard-control checks. What matters for Coinbase is score read against cost. A small, fast model on SERV reaches the top of the board for a few dollars per thousand decisions: grok-4.5 scores 89.1 at $3.36 and gpt-5.6-luna reaches 87.8 at $2.32, while frontier configurations at the same scores run $12 to $40.
The “+ SERV” column is each model's strongest SERV configuration. SERV runs in three reasoning modes, per-call, SERV Shadow, and SERV Kronos; which one leads depends on the model, and the full mode-by-mode matrix is available on request. Latency tracks the mode rather than the model: most SERV configurations answer in 2 to 7 seconds, and where a slower mode tops a model's score, a nearly equal configuration runs in that band.
SERV is the reasoning layer a strategy runs on. In our benchmark, the strategy is the trade itself: a smaller model on SERV makes the call. ThoughtProof, an external team, put a different strategy on the same layer, a safeguard that checks each trade before it executes, and ran it on live Coinbase money. The safeguard held up, at a fraction of the cost of the bigger-model setup it replaced.
Every SERV tier from the mid range up scored above every general-purpose model on the panel, at the same cost tier, and none of them let a dangerous call through.
ThoughtProof Research, Rasch calibration (the psychometric method used to score standardized tests), 150 cases across 13 industries and 10 models, person reliability 0.91. Models on SERV are verification-tuned; the entry tier is the first-pass filter in a cascade, not a standalone judge.
The trading model varied, and the test ran both live and on paper, but the pattern held: when the safeguard on SERV cleared each trade before it executed, the account kept more of its capital. Agents fabricated balances and price moves the data didn't support, and the safeguard caught them before the trade went through.
ThoughtProof, live and paper A/B with Kimi K2.6, GPT-5.4, and Gemini 3.5 Flash as the trading model; the live arm ran 14 days, ~980 cycles, on Advanced Trade. Small-capital experiment; the safeguard runs on SERV's Nano-to-Pro cascade.
Two strategies, the trade and the safeguard around it, both ran on smaller models through SERV and stayed reliable at a fraction of the bigger models' cost.
Coinbase for Agents puts an agent's financial reasoning and execution on a real Coinbase account, within the limits the user sets, across many agents transacting at once. Two things have to hold for that: the agent keeps those limits when real money moves, and the per-decision cost holds as usage grows.
SERV holds both. A small, fast model on SERV keeps the mandate more reliably than the frontier on the real account actions, at half of the per-decision cost down to a tenth.
The summary above rests on a full evaluation.
173 scenarios, 62 of them catastrophic-eligible, meaning an action exists that would breach a hard control. Each is run three times to absorb model variance, so the headline figures are averages. The 186 hard-control checks come from the catastrophic-eligible cases across runs.
Scenarios call the actual Advanced Trade actions behind Coinbase for Agents, with the exact sizing rules (quote_size for market buys, base_size for sells, base_size plus limit_price for limits), portfolio-internal transfers, and KYT/sanctions as hard blocks. It is the production interface.
Every decision is scored by a rule engine that computes the uniquely-correct action. A breach counts because a coded control fired, the numbers are reproducible from the scenario logs, and no model grades another model's output.
Per-call reasoning, SERV Shadow, and SERV Kronos all run under the same scenarios, the same three runs, and the same deterministic grading as every other row. SERV Shadow was additionally re-checked on a frozen held-out set of unseen scenarios: adherence 94 to 98%, with one order through a hard control in 477 decisions.
Degenerate strategies set the floor: an always-refuse and an always-execute agent both score far below every real configuration on the same metric, which confirms the score rewards judgment and not a fixed reflex. A separate multi-agent audit reproduced all 173 oracle labels with zero mismatches.
This measures policy adherence, not profit or alpha. It ran paper and sandbox only, at three runs per scenario. The clean catastrophic result (four breaches to zero) is specific to gpt-5.4-mini; other models shift by different amounts, and the comparison at the top shows each model's before and after.