The OpenAI Decisions API Needs a Confidence You Can Trust

October 1, 2026Say you send an agent's next step to a decision model, and you only let it act on its own when it's at least 99% sure. Everything else goes to a person. That rule is only as good as the 99%. We gave GPT-6 Luna, the model behind OpenAI's new Decisions API, 3,600 reasoning problems and asked for exactly that kind of answer. When Luna said it was 99% sure or more, it was right 68% of the time. OpenAI announced the Decisions API at DevDay on September 29, in limited preview. It picks one answer from a list you define, so software can branch on it: classify a message, route a request, choose what an agent does next. The Decoder and Axios report it runs on "a version of GPT-6 Luna," OpenAI's low-cost model, and OpenAI's own chart puts it at 150 milliseconds a decision. It's OpenAI's answer to Jev. We'd already run Luna that way. In Hard-Decisions, our benchmark of decision models on multi-step logic, Luna answered the same problems as Jev, Kev and Laya, one request each, from a fixed list of options. That makes it a preview of a Luna-based decision model. It isn't the Decisions API itself: OpenAI uses a specialized version of Luna, and there's no documentation yet to test against. Why we care We've been routing on a model's confidence since 2023. For Call Criteria, LLMs score calls against hundreds of client scorecards, and people review the calls the system is unsure about. Everything depends on knowing which calls those are. When we read that confidence from the token log-probabilities of OpenAI's models, the numbers were so concentrated that they looked certain about nearly every answer. OpenAI's models have had confidence problems for as long as we've used them, and not the shy kind. Newer reasoning models often don't expose log-probabilities at all. So we built the confidence ourselves: extract the probability behind each label, check it against labeled answers, calibrate it, and only then route on it. Classification with Confidence shows that pipeline on GPT-4o-mini. That's the pain the Decisions API could end. What we're hoping for most is that OpenAI now treats calibration as a product: a confidence you can set a threshold on, documented and tested, instead of something withheld from the API, which we suspect, without proof, is partly about making the models harder to distill. A decision endpoint is where that would have to happen. So we wanted to know what the model underneath does today. Accuracy by the number of inference steps the answer needs, both ProofWriter tasks pooled. The test The problems come from ProofWriter, a dataset from the Allen Institute for AI. Each one is a short list of facts and if-then rules in plain English, and a statement to judge. Some answers can be read straight off the page. Others need five rules chained together. The dataset labels every problem with that number, its proof depth, which makes difficulty something you can measure instead of argue about. There are two versions of the task. In the open-world one the answer is true, false or unknown: unknown when the rules settle neither the statement nor its opposite. In the closed-world one, anything you can't prove is false. We sampled 1,800 problems for each, about 300 at every depth, and recomputed every gold answer with an independent solver. Luna got the same treatment as everyone else. One request per problem. The problem, the question and the meaning of each option, word for word what Jev receives. A strict JSON schema that only allows the three options. Reasoning off, which is the setting a fast decision API implies. We wrote our predictions down before any model answered. Accuracy: fine on the page, coin flips five steps in On problems you can answer by reading, Luna is nearly as good as Jev: 93% against 98% on the open-world task, and 98% against 99% on the closed one. Then it loses ground with every step. By five chained inferences Luna gets 46% and 45% right. On the closed-world task, that's a coin flip. Jev gets 81% and 89% at the same depth, on the same problems. Across all depths, Jev scored 83.8% and 89.3%; Luna, 64.1% and 65.0%. The gap at depth 5 is 35 and 45 points, with 95% intervals from 27 to 43 and from 38 to 51. That isn't a close call. You might be thinking that reasoning off is what sank Luna. Turning it on might help, but it would also make every decision slower and costlier than the one-shot job a decision API exists to do, and every model here got the same single shot. The open models we tested are stuck at depth 5 too: every model except Jev was at coin-flip accuracy by depth 5. Wrong when the answer is on the page Most of Luna's misses come from long chains, but not all of them. At depth 0, where the statement or its opposite is written in the text, Luna got 20 of 300 open-world problems wrong and 6 of 300 closed-world ones. At depth 1, one rule away, it missed 67 of 302 and 70 of 300. Here are the shortest of those misses at each depth, in each version of the task, with the confidence Luna gave when we asked again for log-probabilities and it gave the same answer: Depth 0, open world. The text says "The lion is not big." Is "The lion is big" true, false or unknown? It's false; the text says so.

When Luna says 99% One example proves nothing, so we asked for log-probabilities on all 3,600 problems with exactly the benchmark's request. It cost about 14 cents, and every request and response is kept verbatim. We preregistered four predictions first. All four held. Each point is a band of stated probability; the dashed diagonal is where a perfectly calibrated model would sit. Both tasks pooled. Luna stated 99% or more on 2,672 of the 3,600 answers, and 68% of those were right. Below that it's flat: whether Luna says 70%, 90% or 97%, it's right about half the time. Jev stated 99% or more on 1,667 answers and got 98.9% of them right. Its line runs close to the diagonal the whole way. The other numbers say the same thing: Wrong answers stated at 95% or more: 78% of Luna's, against 7% and 17% of Jev's on the two tasks. Expected calibration error, the average gap between stated and actual: 0.32 for Luna on both tasks; 0.04 and 0.03 for Jev.