A model that performed at GPT-3.5 level cost about $20 per million tokens in November 2022. By October 2024, the same level of performance cost about $0.07. That is a drop of more than 280 times in under two years, according to the Stanford AI Index 2025.

Source: Generated using AI

Yet many enterprise AI bills are growing, not shrinking. Both things are true at once, and understanding why is the difference between a sensible AI cost strategy and a surprising invoice.

This article looks at how fast the cost of AI inference is falling, what is driving it, where the numbers can mislead you, and what engineering and finance teams should do about it. The title says “exponentially”. That word is fair for the price of a fixed level of capability, which falls in a roughly straight line on a log chart. It is not fair for every model or every price list, and the difference matters.

Three different things people call the cost of AI

Most confusion about AI pricing comes from mixing up three separate measurements.

• Cost per token: what you pay to process or generate one token. This is the number on the pricing page.
• Cost per task: what you pay to finish a piece of work, such as summarizing a contract or resolving a card dispute. It depends on how many tokens the task consumes and how often it has to be retried.
• Cost per unit of intelligence: what you pay to reach a given quality bar, for example a score on a benchmark. This is the number that tells you whether AI itself is getting cheaper.

The first has fallen sharply at every tier. The third has fallen even more dramatically. The second depends on your workload, and it is the one your CFO actually sees.

What the price lists show

OpenAI’s history gives a clean timeline because the company has published list prices for every generation. The table uses standard rates in USD per million tokens, taken from OpenAI’s pricing page and launch-time list prices.

Astra reached the API on September 3, 2026. Sol and Luna followed in late September at half the price of their GPT-5.6 predecessors, with OpenAI attributing the cut to improvements in caching and inference.

The arithmetic is striking. GPT-4 to GPT-4o made input 6 times cheaper and output 4 times cheaper. GPT-4o to GPT-4.1 added another 2.5 times on input and about 1.9 times on output. GPT-4 to GPT-6 Luna is 300 times cheaper on input and 120 times cheaper on output at standard rates.

Two warnings apply before anyone puts those numbers on a slide.

First, these are different capability tiers, not the same model getting cheaper. GPT-6 Luna is not GPT-4 at a discount.

Second, look at the top of the table. Astra’s input price is 3 times lower than GPT-4’s launch price, but its output price is only about 1.2 times lower. The frontier price has barely moved. What collapsed is the price of yesterday’s capability. The newest and strongest model still costs real money, and it is priced in the same range as the top tiers from other labs. Anthropic’s top tier sat at $10 and $50 for Claude Fable 5.1, then dropped to $8 and $40 for Fable 5.2.

[Suggested visual: log-scale chart of OpenAI list prices by model release date, with the frontier tier and the small tier plotted as separate lines]

The same pattern appears across vendors

Anthropic’s published pricing shows the same shape when you compare tiers across generations.

The figures come from Anthropic’s pricing documentation. Google showed the earlier version of this behavior when it cut Gemini 1.5 Pro prices by 64% on input and 52% on output for prompts under 128K tokens in October 2024.

The tokenizer note in the table is easy to miss and worth remembering. Anthropic states that Claude 4.7 and later models produce roughly 30% more tokens for the same text than earlier ones. A lower price per token does not automatically mean a lower price per document. Comparing vendors on price per million tokens is only a first approximation.

Intelligence is getting cheaper, not just tokens

A price list compares different models. A better question is how much it costs to buy a fixed level of capability over time. Several independent efforts have tried to measure that.

The Stanford AI Index 2025, using data from Epoch AI and Artificial Analysis, found that the cost of querying a model at roughly GPT-3.5 level on the MMLU benchmark fell from $20 to $0.07 per million tokens between November 2022 and October 2024. Stanford attributes the drop to increasingly capable small models. On a harder benchmark, GPQA, the cost of reaching above 50% fell from $15 per million tokens in May 2024 to $0.12 in December 2024, which is about 125 times in seven months.

Epoch AI’s analysis of inference price trends puts the rate of decline between 9 times and 900 times per year, depending on the performance threshold. For example, the price of matching GPT-4’s score on PhD-level science questions fell by about 40 times per year.

There are good reasons to hold these numbers loosely:

• The extreme end of the range rests on few data points. Epoch itself notes that the fastest declines happened recently and it is less clear they will persist.
• Benchmarks saturate. Once many models pass a test, the price of “passing” stops telling you much about harder work.
• Price per token is not cost per solved problem. An MIT FutureTech study, The Price of Progress, points out that reasoning models can have a lower per-token cost but a larger overall cost because they generate more tokens. It also cites separate work finding that the cost to solve benchmark problems fell much more slowly on harder tests, by about 24.5 times per year on MATH-500 but only about 3.2 times per year on AIME 2024.

So the honest summary is this. At a fixed, well-defined capability level, cost is falling very fast. For the hardest tasks, where the frontier is the only option, it is falling much more slowly.

Why the cost keeps falling

No single breakthrough explains the trend. Several independent improvements stack, and competition pushes providers to pass some of the savings on.

Compute per token and bytes per token

During generation, a large model spends much of its time moving weights and cached attention data through memory rather than doing arithmetic. That is why several of these techniques work.

A mixture-of-experts model routes each token through a small subset of its parameters. Compute per token falls, though the whole model still has to sit in memory. Quantization stores weights, and sometimes the attention cache, in fewer bits. Less data to move means more tokens per second on the same chip. The price is a small but real risk of quality loss that only shows up when you test against your own tasks.

Batching and prompt caching

A GPU is cheapest to run when it is busy. Providers group many requests together so the hardware stays utilized. The tradeoff is latency, which is why speed is now a priced feature. OpenAI lists a Fast mode for Astra at $20 and $100, double the standard rate. Anthropic prices fast mode for Claude Opus 5.5 at twice standard as well.

Prompt caching works because transformer models compute intermediate attention state for every token they read. If the next request starts with the same long prefix, a provider can reuse that state instead of recomputing it. This describes the general mechanism, not any one vendor’s implementation. The price effect is public: cached input for Astra is $1 per million tokens versus $10, and on Claude Sonnet 5.5 a cache read costs 5% of the base input price.

What this looks like in real money

Take a hypothetical application that processes 1 million input tokens and 250,000 output tokens. Input-heavy workloads are typical. Snowflake’s AI research team has observed roughly 10 to 1 ratios between prompt and generated tokens across many enterprise workloads.

Choosing the right tier matters as much as choosing the newest model. Astra costs 100 times more than Luna for the same token volume, and the model released today at the top of the range is not cheaper than GPT-4.1 from 2025.

Now scale it up. Consider a bank that uses an LLM to process debit-card disputes. Assume 10 million claims a year, with 5,000 input tokens and 1,000 output tokens per claim. That is 50 billion input tokens and 10 billion output tokens annually. These are illustrative assumptions, not a forecast.

Standard rates are assumed throughout. Flex and Batch processing roughly halve every figure for work that can wait, so Astra drops to about $500,000 and Luna to about $5,000.

Caching changes the picture further. Suppose 80% of each prompt is a stable policy manual and examples that hit the cache on Sol. Input cost falls from $100,000 to about $24,000 (10 billion uncached tokens at $2, plus 40 billion cached tokens at $0.10 per million). The total becomes about $124,000. Output, now $100,000, dominates the bill. Once input is cached, the length of the answer is the main cost lever.

Cheaper tokens do not always mean cheaper outcomes

A $0.001 claim sounds like a bargain until the cheap model gets the answer wrong.

The metric that matters to a business is cost per successful outcome:

cost per successful outcome = total AI operating cost / successfully completed tasks

total AI operating cost = token cost + retries + human review + rework + incident cost

Here is a worked example, with assumptions stated. A small model costs $0.001 per claim and resolves 70% of claims correctly. A frontier model costs $0.10 per claim and resolves 98%. Each failed claim needs a human to fix it, at an assumed $8.

• Small model: $0.001 + 0.30 × $8 = about $2.40 per claim.
• Frontier model: $0.10 + 0.02 × $8 = about $0.26 per claim.

The frontier model costs 100 times more in tokens and is about 9 times cheaper per claim. Across 10 million claims that is roughly $24 million versus $2.6 million. The result flips if the cheap model’s accuracy is close to the expensive model’s, or if human review is cheap. The only way to know is to measure accuracy on your own tasks.

Where the arithmetic gets messy

• Reasoning tokens. Reasoning models produce hidden thinking tokens that are billed as output. A cheap per-token price can still produce an expensive answer, as the MIT study above notes.
• Agent loops. An agent that makes 20 model calls, each re-sending 30,000 tokens of context, consumes about 600,000 input tokens for one task. At $2 per million that is $1.20 for the task, against a cent or two for a single claim. With 90% cache hits it falls to about $0.18, but it never reaches the cost of a single call.
• Tokenizers. The same document can be a different number of tokens on different models, as the Claude tokenizer note showed.
• Long context. Some price lists change above a prompt-length threshold. Claude Haiku 5.5 lists five times the standard rate above 100,000 tokens.

The deeper point is Jevons-style. When a resource gets cheaper, people use more of it. Cheaper tokens encourage more agents, longer contexts and more automated workflows. Unit cost can fall 100 times while total spend rises, which is why many teams see bills grow during a period of falling prices.

Engineering for a falling cost curve

If unit prices keep falling, the right architecture is one that can move work between tiers as prices and quality change. The practical pattern is a model cascade: try a cheaper model first, validate the result, and escalate only when needed.

The cascade’s expected cost is easy to model. The following Python sketch is illustrative, not production code. It assumes a validator that reliably detects failures and that each tier’s success rate applies to the tasks it actually receives. In practice, tasks that escape a small model are usually harder than average, so measured success rates on escalated cases will be lower.

from dataclasses import dataclass

@dataclass(frozen=True)
class Tier:
    name: str
    input_per_m: float    # USD per 1M input tokens
    output_per_m: float   # USD per 1M output tokens
    success_rate: float   # measured on your own evaluation set

def token_cost(tier: Tier, tokens_in: int, tokens_out: int) -> float:
    return (tokens_in * tier.input_per_m + tokens_out * tier.output_per_m) / 1_000_000

def expected_cost(tiers: list[Tier], tokens_in: int, tokens_out: int, human_cost: float) -> float:
    """Expected cost per task when tiers are tried in order and failures escalate."""
    cost, p_reach = 0.0, 1.0
    for tier in tiers:
        cost += p_reach * token_cost(tier, tokens_in, tokens_out)
        p_reach *= 1 - tier.success_rate      # share of tasks that fail this tier
    return cost + p_reach * human_cost         # leftover failures go to a human

small = Tier("small", 0.10, 0.50, 0.70)
frontier = Tier("frontier", 10.00, 50.00, 0.98)

print(expected_cost([small], 5_000, 1_000, 8.0))            # about 2.40
print(expected_cost([frontier], 5_000, 1_000, 8.0))         # about 0.26
print(expected_cost([small, frontier], 5_000, 1_000, 8.0))  # about 0.079

With the earlier assumptions, the cascade costs about $0.08 per claim. That beats both single-model options, but only because the validator is trustworthy. A weak validator turns a cost optimization into a quality problem.

Caching deserves the same care. The stable part of the prompt must be identical from one request to the next, so the variable part goes last. This pseudocode shows the idea.

# Pseudocode: keep the shared prefix byte-identical so it can be cached
def build_prompt(claim):
    return [
        {"role": "system", "content": POLICY_MANUAL + FEW_SHOT_EXAMPLES},  # same on every call
        {"role": "user", "content": render(claim)},                         # changes on every call
    ]

Providers differ in how they enable this. Anthropic documents a cache_control field, and OpenAI’s docs have a guide on keeping the prompt prefix unchanged. Check each vendor’s current documentation for the exact behavior.

Which tactics fit which situation

Token efficiency is becoming an engineering metric in its own right. Useful numbers to track are tokens per task, cache hit rate, escalation rate, retries, and cost per successful outcome. Without them you cannot tell whether a price cut helped.

Risks and limits of the cost curve

Prices are not fixed, and promotions end. OpenAI’s page notes that GPT-5.6 Sol’s promotional pricing runs only until November 30, 2026. Models are deprecated and replaced, and a forced migration is an engineering cost even when the new model is cheaper.

A cheaper model is a different model. Swapping tiers changes output style, failure modes and sometimes refusal behavior. Without a regression suite you will learn about the difference from customers. For regulated work such as banking, a cost-driven model change may need its own review.

Hidden multipliers. Regional processing for newer OpenAI models carries a 10% uplift, and Anthropic applies a 1.1 times multiplier for US-only inference. Speed premiums double the rate. These adjustments can erase part of a headline discount.

Efficiency gains are not guaranteed to reach you. The savings from better hardware and serving can be passed on, kept as margin, or spent on larger models. Stanford’s cost-per-capability data shows the net effect so far, but it does not guarantee the future.

Budget governance lags adoption. Because usage tends to grow faster than price falls, teams that budget by token price alone are surprised by total spend.

Where the curve may go next

It helps to separate what is demonstrated from what is hoped for.

• Demonstrated: Orders-of-magnitude declines in the price of fixed capability since 2022, as measured by Stanford and Epoch AI. Frontier-tier prices that stay roughly flat while older capability collapses.
• Deployed: Prompt caching, batch discounts, small and mid-tier models for high-volume work, and routing patterns in production systems.
• Research: Algorithmic efficiency gains, cheaper long-context handling, and better methods for making reasoning use fewer tokens. The Price of Progress study estimates how much of the decline comes from algorithms and how much from hardware and economics.
• Speculative: That current rates of decline continue for many more years. Epoch’s own caution applies. The fastest drops were recent, and rates tend to slow as a capability level ages.

For the strongest claims to come true, hardware efficiency must keep improving, small models must keep closing the gap on hard tasks, and competition must stay intense enough that providers pass savings on. Any of those could weaken.

What enterprises should do differently

The opportunity is bigger than saving money on today’s workloads. As the cost of a capability falls, tasks that were uneconomic to automate become viable. Document review, claims triage and internal search might not have cleared a business case three years ago and might clear it now.

A practical approach is to keep a short list of workflows that failed on cost and re-run the numbers every quarter. Build routing and evaluation first, so a cheaper model can be adopted in days rather than months. Negotiate contracts around capability and outcomes instead of fixed per-token rates that will look expensive within a year.

Key takeaways

• Token prices are falling, and the price of a fixed level of capability is falling much faster. Stanford documents a drop of more than 280 times for GPT-3.5-level performance in under two years, and Epoch AI reports declines of 9 to 900 times per year depending on the threshold.
• The frontier is not getting cheap at the same rate. The newest top-tier models still cost around $10 and $50 per million tokens.
• Cost per token, cost per task and cost per unit of intelligence are different measurements. Cost per successful outcome is the one that belongs in a business case.
• A model cascade with validators, prompt caching and batch processing turn falling prices into real savings. Without measurement they can also hide quality problems.
• Lower unit cost does not guarantee lower total spend. Cheaper tokens invite more usage, and total consumption can grow faster than prices fall.

The price of intelligence is dropping quickly, but the price of being wrong is not. The teams that benefit most will be the ones that can measure outcomes, switch models safely, and spend the savings on problems that were previously out of reach.