Why reasoning models should not have to read everything
InfoWorld ·
The AI industry has been expanding how much information an AI model can consume at once. In just a few years, context windows have grown from thousands of tokens to millions. Anthropic increased Claude’s context window from 9,000 to 100,000 tokens in May 2023 , reaching 200,000 tokens by November 2023 . Google’s Gemini 1.5 Pro reached one million tokens in February 2024 , then two million by June 2024 . By March 2026 , Anthropic made the million-token window generally available across its Claude models, and thirteen or more hosted frontier models now offer one-million tokens or more. Reasoning systems now work through longer problems, call on more tools, and run for extended stretches with little human input. But the more widely AI is used, the more expensive it gets. Tokens are getting cheaper, but AI systems are consuming more of them. Ramp data shows the price US businesses pay per million tokens has fallen 41% from its peak in March 2026 to September 2026, yet Goldman Sachs expects AI agents to drive a 24-fold increase in token consumption by 2030. As frontier models grow more capable and complex, lower token prices will only go so far. In data-intensive industries, making the context window larger risks being counterproductive. Although models may be able to process more information, more data drives up costs faster than token prices can fall. But a reasoning model should not have to read every data point in order to reason about the data, or reprocess the same underlying dataset every time a new question is asked. Larger context is not the same as better architecture Picture a trader analyzing 10 years of market data, or a scientist working through millions of genomic observations. One approach is to place the underlying data inside the model’s context and let it work through everything directly. Technically, this may work, but architecturally, it means an expensive reasoning system has to process enormous quantities of information before it can begin understanding the question, determining the appropriate analysis, and interpreting the result. But a quantitative analyst doesn’t manually read hundreds of thousands of observations and do the arithmetic by hand. They decide what needs calculating, hand it to a computational tool, and reason with what comes back. There’s no obvious reason AI needs to work differently. Reasoning and computation can be separated, with the model determining what needs to be calculated, while conventional systems perform the numerical work and the model reasons over the output. Those are two different skills, and treating them as one can create unnecessary costs. The goal should be to spend reasoning where it creates value and let conventional computation handle the rest. The cost of the gap An Opetek benchmark using its proprietary reasoning architecture tested this with a simple task, calculating realized volatility from 10 years of EUR/USD data. Passing the raw time series directly into the LLM’s context cost roughly $0.80 per query. Keeping the computation outside the reasoning layer, however, brought the cost down to about $0.07, showing a reduction of approximately 90%. That’s not a claim that every AI workload gets 90% cheaper by restructuring this way. Rather, it shows that where computation happens can materially change the economics of a reasoning system. When a calculation is deterministic and can be run cheaply outside the reasoning model, there is little value in paying the more expensive model to ingest every observation needed to perform the calculation. Context windows cannot substitute for system design There is good reason for the industry’s enthusiasm about larger context windows. Giving models access to more information can make them significantly more capable. But the fact that a model can read everything doesn’t mean an application should ask it to do so every time. Determining which information should enter the reasoning model will be an important contributor to AI efficiency as reasoning models become more integrated in production workflows. AI efficiency has three layers. The first is model efficiency, or how cheaply the underlying model can produce intelligence. The second is retrieval efficiency, the system’s ability to find the information the model actually needs. The third is computational architecture, the discipline of deciding what reaches the model at all. Together, these three layers determine how efficient an AI application can be. Capital markets are only an early warning The design problem is particularly visible in financial markets because the datasets are enormous, the questions are highly quantitative, and the same analytical tasks are performed repeatedly. Asking a reasoning model to process all of the underlying data, including information that could be handled by conventional computation, therefore becomes expensive very quickly. But the same challenge applies in disciplines where sophisticated reasoning meets large numerical datasets, from genomics and physics to engineering, climate modelling, and medicine. AI systems in all these fields are gradually combining foundation models with specialized tools and very large datasets. As those systems take on larger and more complex workloads, the amount of information they process will grow. With that, the distinction between which questions require reasoning and which can be handled through conventional computation becomes more important. The next breakthrough in AI efficiency will come from deciding what a model never needs to read. — New Tech Forum provides a venue for technology leaders—including vendors and other outside contributors—to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to doug_dineley@foundryco.com .