What a generative AI system costs to run after launch

A build price covers the work of making the system. Running it costs money every month afterward, and the amount depends on use. This post lists the line items, shows how to estimate the largest one, and describes the levers that reduce it.

The line items

• Model usage. Providers price by the token, the unit a model reads and writes, often a word or part of a word. Input and output are priced separately. On Anthropic's published price list in September 2026, output tokens cost five times as much as input tokens on each current model in the list (Anthropic pricing). Other providers set their own ratios, so check the list for the model you use.
• Retrieval. A RAG system pays to turn documents into embeddings, to store them, and to search them on every request.
• Hosting. The application, its database, and any self-hosted model need servers. Our post on hosted and open-weight models compares the two routes.
• Monitoring and evaluation. Every evaluation run calls the model, and an automated judge adds a second model call for each answer it scores.
• People. Someone reviews flagged answers, keeps the documents current, and handles the changes that come with new model versions.

Estimating model usage

Cost per request is the input tokens times the input price, plus the output tokens times the output price. Multiply that by the requests you expect each month.

Here is an illustration with made-up traffic. Suppose the model costs $2 per million input tokens and $10 per million output tokens, and a typical request sends 3,000 input tokens (the question plus the retrieved passages) and receives 500 output tokens. The input costs $0.006 and the output costs $0.005, so a request costs $0.011. At 100,000 requests a month, that comes to $1,100. These prices are examples, not a quote, and your figures will differ. The method is what carries over.

In this example the input is more than half the cost, so retrieval quality affects cost as well as accuracy. Suppose five retrieved passages of about 500 tokens each make up 2,500 of those 3,000 input tokens. Cutting to the two best passages removes 1,500 tokens, which saves $0.003 per request, about a quarter of the cost, or $300 a month at 100,000 requests. These are assumptions for the illustration.

What moves the number

• Model choice. The FrugalGPT paper reviewed the cost of querying popular LLM APIs and found heterogeneous pricing in 2023, with fees that can differ by two orders of magnitude. Its cascade approach, which tries cheaper models first and escalates only when needed, matched the best single model's performance in the authors' experiments with up to 98% lower cost (Chen et al., 2023). Results depend on the task, so test on your own examples.
• Prompt length. Instructions, examples, and retrieved passages all count as input.
• Output length. Output is the pricier side, so ask for concise formats where concise is enough.
• Caching. Some providers, Anthropic among them, price repeated prompt prefixes at a fraction of the standard input price. On Anthropic's list, a cache hit costs 10% or less of the standard input price on the models listed. Writing to the cache costs more than normal input, so caching pays off only for prefixes that many requests reuse.
• Batching. Anthropic's list shows a 50% discount on input and output tokens for its batch API, which suits work that does not need an immediate answer.
• Number of calls. An agent makes a model call at each step, so it usually costs more per task than a single call. See our post on agents and workflows.

Lowering the bill, and checking quality holds

Try the changes in this order, from least to most risk to answer quality:

1. Cache the parts of the prompt that never change.
2. Batch offline work, such as nightly document processing.
3. Improve retrieval so fewer, better passages reach the prompt. Our post on why RAG answers fail covers how.
4. Cap output length where a short answer is enough.
5. Send easy requests to a smaller model and escalate only the hard ones.

Test every change against your evaluation set. A cheaper setup that lowers answer quality can cost more in the end, through rework and lost trust.

Set budgets before launch

Track cost per request and cost per completed task for each feature, and set an alert on each. Usage can grow as more people adopt the system, so revisit the estimate after launch. Provider prices change, so read the current list instead of relying on a figure from a document written last year, including this one.

What to ask a vendor for

Ask for a running-cost estimate with its assumptions written out: requests per day, tokens per request, which model, what retrieval and hosting cost. An estimate without those numbers cannot be checked. Our post on how to choose a generative AI development company lists the other pricing questions, and our pricing page explains what drives the cost of a build.