Skip to content
AI Automation

Decision Models Explained: One Question, One Answer, One Number. When Can A Trading Firm Trust The Confidence Score?

Between 1 and 10 October 2026 Cloudflare, OpenAI, Microsoft and the Strands Agents project each shipped a decision model, a small AI model that answers one bounded question with a probability per option in well under a second. The three largest Hacker News threads on the subject gathered 1,479 points and 807 comments. The argument running through them is whether the confidence number can be trusted: one test found a frontier model was right 68% of the time when it claimed 99% or more confidence. This is the plain-English guide for founders, product heads and compliance leads: what a decision model is, where a trading business would use one, and what to insist on before anyone acts on its number.

AlchmAI Editorial9 min read

68%

How often GPT-6 Luna was right when it claimed 99% or more confidence, across 2,672 answers (Anthus, 1 October 2026)

0.7009

Accuracy in the top bin of a home-built decision model that claimed 0.9855 before calibration, 0.9541 after (Nish Tahir, 10 October 2026)

38.8 ms

Median latency of Cloudflare's Clef-flash, against 209.3 ms for Clef and 524.1 ms for Jev (Cloudflare, 1 October 2026)

1,479

Hacker News points across the three largest decision-model threads, 1 to 9 October 2026, with 807 comments

Decision models are small AI models that answer one bounded question, such as "is this client message a complaint?", with a probability for each possible answer, in well under a second and for a fraction of a cent. Between 1 and 10 October 2026 Cloudflare, OpenAI, Microsoft and the Strands Agents project each shipped one, and on 9 October 2026 TypeSafe AI, maker of the first, Jev, announced it had raised $870 million at a $7.5 billion valuation. The argument running through the Hacker News threads on all of them is the one a trading firm should settle first: can the confidence number be trusted?

The evidence from the same ten days says: not until you have measured it. Anthus reported on 1 October 2026 that GPT-6 Luna, the model under OpenAI's Decisions API, was right 68% of the time when it claimed 99% or more confidence. That is not a reason to avoid decision models. It is a reason to treat the number as a reading from an instrument nobody has calibrated for your data.

What Is A Decision Model, And What Is It Not?

Three properties define the category. The question is bounded: you supply the options and the model cannot answer outside them. The answer is a distribution: 0.62 complaint, 0.31 query, 0.07 praise is the whole output. And it is fast and cheap because it skips text generation: at the published latencies and prices a client email of a few hundred tokens is classified in well under a second for a few thousandths of a cent.

It is not a general-purpose brain at a discount, and the gap is where the risk sits.

  • Not a chatbot. It produces no sentences, so nothing in its output can drift into advice.
  • Not an analyst. It does not reason through your rulebook; Anthus ran its test with reasoning off, which is how these models operate.
  • Not a judgement. A 0.94 for "escalate" means the model's probability concentrated on that option, not that a person would have escalated.
  • Not advice. It sorts, routes and gates. It must never say what a client should do, and its output must never reach a client as a view on an instrument.

What Launched In The First Ten Days Of October 2026?

Jev was the first hosted decision model. In ten days it acquired four rivals, two with open weights.

  1. 011 October 2026: Cloudflare released Clef and Clef-flash with open weights under an Apache 2.0 licence, as adapters on Qwen3.8-27B and Qwen3.5-9B. On BANKING77, a public benchmark of customer banking queries, Cloudflare reported macro-F1 of 94.20 for Clef against 79.74 for Jev. Hacker News: 642 points, 218 comments.
  2. 021 October 2026: the Strands Agents project released Strands Decider 2B, a 2 billion parameter open-source model, with an example that uses its yes or no answer to decide whether an agent's tool call proceeds.
  3. 036 October 2026: OpenAI's Decisions API entered public beta on gpt-6-luna, the only supported model, at $0.10 per 1M input tokens with no charge for output. Hacker News: 393 points, 233 comments.
  4. 049 October 2026: Microsoft released Microsoft-Decision-1 at $0.042 per 1M input tokens, reporting that it changes its decision on 1.3% of perturbations on average and makes zero flips when options are paraphrased, reversed or shuffled. The same day Cloudflare added Clef-omni, which accepts audio and video.
  5. 059 October 2026: TypeSafe AI announced it had raised $870 million at a $7.5 billion valuation, a company claim that drew 444 points and 356 comments on Hacker News, much of it asking what the moat is.

Beneath the vendors sits an open clone. Kev, published by Jared Palmer, is a family of Jev-like decision models you can train and run yourself, with 9.1k GitHub stars on 11 October 2026, and Nish Tahir's tutorial of 10 October 2026 built one from scratch on Qwen3-1.7B. Having a decision model now takes an afternoon; having a trustworthy one is the harder part.

Where Would A Trading Business Use One?

Anywhere a person makes the same small classification hundreds of times a day and one wrong case is recoverable. Six candidates stand out.

  • Client-message triage: complaint, withdrawal request, KYC query or sales lead? Route it, with complaints going to a person first.
  • KYC document checks: passport, driving licence, utility bill or none of those? A person or a verification service still verifies.
  • Surveillance alert triage: which overnight alerts match known false-positive patterns and which need an analyst first? The model orders the queue. It does not close alerts.
  • News and filing tagging: is this headline about a watch-list instrument, and is it earnings, guidance, regulatory or corporate action?
  • Gating an agent's actions: before an internal agent places, amends or cancels anything, a separate decision model answers "does this match the approved pattern?" and a person confirms anything above a size limit.
  • Classifying chart images: candlestick, line or not a chart? Classification only. Anything that reads a chart to a client goes through descriptive, signed-off analysis, never a decision model's number.

Every item has the same shape: the model classifies, and a person, a rule or a confirmation step acts. That division gives you somewhere to send the cases the model is unsure about.

Can You Trust The Confidence Score?

Start with what the number is. TypeSafe's Jev documentation says the confidence property "collapses that shape into a single number from 0 to 1": 1 when all probability sits on one outcome, 0 when it is spread evenly. That is a concentration measure, not a hit rate. The property that would tell you that is calibration: a model is calibrated when, of the answers it gives at 90% confidence, about 90% are correct. The October evidence shows how far apart the two can be.

  • Anthus, 1 October 2026: testing GPT-6 Luna through the standard API with reasoning off, Ryan Porter found it right 68% of the time when it stated 99% or more confidence, across 2,672 answers, with an expected calibration error of 0.32. Jev was right 98.9% of the time on 1,667 such answers, with errors of 0.04 and 0.03.
  • Maximum Effort, 2 October 2026: Dylan Black found Jev's average error on probability distributions, measured as total variation, was 0.518 against 0.546 for a flat guess.
  • Red Hat, 2 October 2026: on a prompt-injection benchmark a pre-trained DeBERTa-v3 classifier scored 89.01% at a 54.1 ms median against 86.35% at 348.1 ms for Jev. The authors "did not find that decision models produced faster, cheaper, or higher-quality answers compared with LLM-as-a-judge".
  • Nish Tahir, 10 October 2026: a model built on Qwen3-1.7B and tested on 1,221 held-out CommonsenseQA items claimed 0.9855 in its top bin and scored 0.7009. After temperature scaling, a one-parameter calibration fitted to the model's own accuracy, the top bin claimed 0.9333 and scored 0.9541.

Read together, the four are consistent. A general-purpose model asked to act as a decision model is badly over-confident. A purpose-built model can be well calibrated on one task, as Anthus found, and poorly on another, as Black found, because calibration is a property of a model on a particular distribution of questions, not of the model alone. TypeSafe's own example routes anything below 0.5 confidence to a human and still asks for confirmation on approve_transfer above 0.9, and its documentation says thresholds "depend on your domain and the performance of the model for your use case".

What To Insist On Before Anyone Acts On A Decision Model

Our view is that decision models belong in trading operations, and that most firms deploying them will skip the step that makes the number mean something. Put this list to any vendor, internal team or AI Automation London supplier first.

  1. 01A labelled test set of your own cases: a few thousand real messages, alerts or documents, labelled by your staff and held back from training. BANKING77 shows a model can separate someone else's categories, not yours.
  2. 02A calibration curve on that set, with expected calibration error as one figure. Anthus got 0.32 from a frontier model and 0.04 from a purpose-built one on the same questions.
  3. 03Thresholds per risk level. Jev's example uses 0.5 for low stakes and 0.9 with confirmation for a transfer. Set yours from the curve and record who set them.
  4. 04Abstain-to-human below the threshold. If the system has no path for "not sure", the threshold is decorative.
  5. 05Shadow mode first. Log every decision the model would have taken beside what people did, then let it act only on the categories where it earned it.
  6. 06A fine-tuned classic classifier as the benchmark. Red Hat's DeBERTa-v3 beat Jev on accuracy and was six times faster.

“A confidence score is a claim the model makes about itself. Until you have checked it against a few thousand of your own labelled cases, it is a claim, not a measurement.”

What It Means For Trading And Fintech Firms

  • Brokers and CFD providers: start with client-message triage in shadow mode against your support team's own routing for a month, have the compliance lead sign the thresholds, and keep complaints and withdrawal requests on the abstain-to-human path.
  • Prop trading firms: use a decision model to order the overnight rule-breach and payout-dispute queue, not to decide it. Your dispute history is the labelled test set; log the model version beside the rule version on every case.
  • Fintech and trading startups: if an agent in your product calls a broker API, put a separate decision model in front of order placement, amendment and cancellation, with a person confirming anything above a size limit. Due diligence will ask for the shadow-mode log.
  • Research and quant teams: decision models tag news, filings and chart images cheaply, and open weights from Cloudflare and Kev keep the data on your infrastructure. Benchmark against a fine-tuned classic classifier on your own labels first.

The Bottom Line

Decision models answer one bounded question with a probability per option, fast and for a few thousandths of a cent, and between 1 and 10 October 2026 Cloudflare, Strands, OpenAI and Microsoft each shipped one while TypeSafe AI announced an $870 million raise for the first. For a trading business they suit triage, tagging and gating, and nothing a client sees or that acts alone. The confidence score is a concentration measure, not a probability of being right: a frontier model was right 68% of the time at 99% claimed confidence, and a home-built model claimed 0.9855 while scoring 0.7009 until calibrated. Measured on a labelled set of your own cases, with thresholds by risk and an abstain path, it becomes a figure you can defend to an auditor. That evaluation and controls work is what AlchmAI, a London trading technology and AI studio, builds around AI features near clients and markets.

References & Further Reading

AI Automationdecision modelsAI Automation LondonTrading Workflow automationCompliance & Regulatory SystemscalibrationClef
Share Email
AI

AlchmAI Editorial

Research and analysis, London

The AlchmAI team writes about the markets, technology and regulation we work with every day. We build trading platforms, real-time charts and AI analysis tools for brokers, prop firms and fintech teams from our office in Mayfair, London. Every article lists its sources. Nothing we publish is investment advice.

This article is general information and commentary. It is not investment advice or a recommendation to buy or sell any investment. Important information

Keep reading

All articles