The Economics of Decision Models

If you like this blog post, do subscribe to my RSS feedTo Make, or to Buy: The Economics of Decision Models First Published: 18/10/26 It is the maxim of every prudent master of a family, never to attempt to make at home what it will cost him more to make than to buy. The tailor does not attempt to make his own shoes, but buys them of the shoemaker. The shoemaker does not attempt to make his own clothes, but employs a tailor. The farmer attempts to make neither the one nor the other, but employs those different artificers. All of them find it for their interest to employ their whole industry in a way in which they have some advantage over their neighbours, and to purchase with a part of its produce, or, what is the same thing, with the price of a part of it, whatever else they have occasion for. - Adam Smith in The Wealth of Nations (1776) In the past few weeks, Jev and decision models in particular have been the talk of the town. As things are cooling down a bit, I thought it would be fun to systematically explore and forecast where all of this leads to. The magic, as we all would agree, seems to be in the output, of which there can be three kinds: choice, score and noul. With choice, we provide a fixed list of options and the output is a probability for each option, summing to 1. The score is for rating something against an ordered list of levels, using a probability-weighted average of those levels. And finally, the noul, which some say is named after the Bernoulli trial, returns a number between 0 and 1: the probability that a given statement is true. Here is an example using Jev 1.13 from OpenRouter: REQUEST: { "model": "typesafe/jev-1.13", "state": "Some damn good curly fries - seasoning was something.", "questions": {"answer": {"type": "score", "instructions": "How enthusiastic is the reviewer?", "criteria": ["Lukewarm", "Happy", "Thrilled"] }}} RESPONSE: { "model": "typesafe/jev-1.13-20260917", "answers": {"answer": {"type": "score", "score": 1.48, "legend": {"0": "Lukewarm", "1": "Happy", "2": "Thrilled"}, "probabilities": {"0": 0, "1": 0.51, "2": 0.49}, "confidence": 0.27}}, ..... } There are a couple of interesting consequences as a result of this model. First, it's a well-defined output! We state the possible answers before asking the question, and the model has to pick among them. Second, every answer carries odds. As we will see later, we can leverage this to do some interesting arithmetic on them. Third, as the expense is in reading the state and consuming the input tokens, the output tokens are essentially free. Case in point: I did some napkin math on the above example. Assuming that a regular LLM would require 250 input tokens and 50 output tokens to produce this response, and that we have 400,000 such queries to process, the resulting 100M input tokens and 20M output tokens would cost $13 on GPT-5 nano ($8.5 with cached tokens) vs. $4.2 via Jev. And finally, it is fast! The decision hierarchy Perhaps the most exciting revelation I had, is that this gives rise to a new form of memory hierarchy, which I have labeled as decision hierarchy. As a refresher, pick your favorite textbook and flip to its memory management page. Here is what Tanenbaum has to say about memory hierarchies in his Modern Operating Systems book: Over the years, people discovered the concept of a memory hierarchy, in which computers have a few megabytes of very fast, expensive, volatile cache memory, a few gigabytes of medium-speed, medium-priced, volatile main memory, and a few terabytes of slow, cheap, nonvolatile magnetic or solid-state disk storage, not to mention removable storage, such as DVDs and USB sticks. It is the job of the operating system to abstract this hierarchy into a useful model and then manage the abstraction. These learnings can be applied in the realm of LLMs too. Assume that you have an L-cache consisting of a mixture of hand-written rules, and exact/semantic matching of past answers to the same question and its state. If so, Jev and Decision models can be imagined as the RAM, LLM as disk, and magnetic tapes as human beings. Each level handles what its confident about and passes the misses down to the level below. Figure 1: Memory hierarchy v.

decision hierarchy.

But how do we decide on a confidence factor and pick a threshold? This is where Chow's rule from 1970 comes in. Let us motivate the idea first. Imagine a classifier \(C\) with \(n\) classes. For each input, the classifier generates a probability for each of the \(n\) classes. When the largest class has a probability close to 1, the model is sure about its decision, and mistakes are rare. However, when all the classes have equal probability, say \(1/n\), any answer is close to a guess. So, the errors come from the uncertain inputs. And to mitigate it, we need a higher form of intelligence - from decision models, to LLMs, and in the worst case humans. As I like to put it, the more “philosophically-NP” (not in a formal sense) a question is, the higher form of intelligence it needs to make a non-ambiguous judgement call.

Answer v.

Abstain But how do we decide whether we should give a response at a particular level or abstain from doing so? A simple way would be using threshold \(T\). For each input, we look at the top probability, say \(p_{\text{max}}\). If \(p_{\text{max}} \leq T\), we reject and hand the input on to the next level. If \(p_{\text{max}} \gt T\), we answer with the top probability outcome. With this approach, we get a dial that moves from \(T = 1/n\) to \(T = 1\). The higher the dial, the more rejects we get and fewer errors on what we keep. However, this simple threshold approach does not take into consideration the price of the outcome. Particularly, what is the cost of us answering a question wrong at a level, and the cost of declining to answer at a level. Say, a wrong answer costs 1, rejecting costs a fraction \(F\), and a right answer costs 0. If we answer with class \(k\), we are wrong with probability of \(1 - p_k\). The probability of being wrong would be at its smallest for the top class, \(1 - p_{\text{max}}\). Therefore, a second approach would be to answer only if we beat the price of rejecting. In other words, answer when \(1 - p_{\text{max}} \lt F\). Take an example where \(F=0.2\) and \(p_{\text{max}}=0.6\). As answering costs \(1 - p_{\text{max}} = 0.4\) on average and rejecting costs 0.2, we reject. Next, we need to evaluate what rejection (\(F\)) really costs. When we are sending a decision to the next level, we need to take into consideration the cost of an error in that level (\(\text{NX}_{\text{errcost}}\)), the cost we pay to evaluate any input in that level (\(\text{NX}_{\text{price}}\)), and finally the error rate on the next level itself (\(\text{NX}_{\text{err}}\)). Deriving Chow's rule Consider this - say we have 100 software tickets to evaluate and make decisions upon. And we are working at the Jev and LLM layers. Imagine that for all our tickets, Jev is 60% sure about its decision (\(p_{\text{max}} = 0.6\)).

A wrong routing costs $2.

The LLM charges $0.01 per ticket (\(\text{NX}_{\text{price}}\)) and is wrong 5% of the time (\(\text{NX}_{\text{err}}\)). The LLM-level also incurs a $2 cost (\(\text{NX}_{\text{errcost}}\)) when an error is produced at its level. What is the cost incurred in each extreme scenario? Jev answers all 100 tickets: Jev is wrong on 40 of them. 40 x $2 = $80. Send all 100 tickets to LLM: LLM is wrong on 5 tickets. The cost of LLM evaluation is 100 x $0.01 = $1 in fees. On the 5 wrong tickets, the total cost incurred is 5 x $2 = $10. All in all, $11. Per ticket, that's $0.80 for answering versus $0.11 for handing on to the next level, so we hand on. Notice that we did not take Jev's fees into account. This is because it's paid in both options, so it can't change which one is cheaper. Now let us derive \(F\) so that we can apply this in a more general scenario. Remember that handing on to the next level charges us twice: we pay the fee and absorb the LLM's mistakes. Error rate on the next level (\(\text{NX}_{\text{err}}\)): 5 wrong answers per 100 tickets = 0.05 wrong answers per ticket. How many wrong answers is a fee worth in the next level: $0.01 per ticket / $2 cost for a wrong answer = 0.005 wrong answers. Or \(\text{NX}_{\text{price}} / \text{NX}_{\text{errcost}}\). Therefore, \(F = \text{NX}_{\text{err}} + \text{NX}_{\text{price}} / \text{NX}_{\text{errcost}}\). Substituting it in the original equation, we get Chow's rule: \[1 - p_{\text{max}} \lt \text{NX}_{\text{err}} + \text{NX}_{\text{price}} / \text{NX}_{\text{errcost}}\] For the above example, \(1 - p_{\text{max}} = 1 - 0.6 = 0.4\). And \(\text{NX}_{\text{err}} + \text{NX}_{\text{price}} / \text{NX}_{\text{errcost}} = 0.05 + 0.005 = 0.055\). As \(0.4 \gt 0.055\), we hand on to the next level, which is the same decision as $0.80 versus $0.11.

What is it telling us?

Notice that when errors are expensive, LLM's price barely matters. The only question is who's more accurate on this input. This is because \(\text{NX}_{\text{price}} / \text{NX}_{\text{errcost}}\) shrinks toward zero, so \(F \sim \text{NX}_{\text{err}}\). In other words, when errors are expensive, a cheaper LLM barely helps, but a more accurate one helps a lot. However, when errors are cheap, \(\text{NX}_{\text{price}} / \text{NX}_{\text{errcost}} \gt \text{NX}_{\text{err}}\). Re-arranging it gives us - \(\text{NX}_{\text{errcost}} \lt \text{NX}_{\text{price}} / \text{NX}_{\text{err}}\). In other words, LLM's fee only matters more than its mistakes when a wrong answer costs under a certain amount. For our above example, that's $0.01 / 0.05 or 20 cents.

Our tickets cost $2 per wrong answer, well above that, so accuracy decides.