Hosted or open-weight LLM: how to choose

Every generative AI project has to decide where the model runs. A hosted model is one you call over an API from a provider. An open-weight model is one whose trained weights are published, so you can run it on your own servers or in your own cloud account. Open-weight is not always the same as open source, because licenses differ, so read the license before you build on one.

What decides it

• Data rules. If your rules require data to stay in your environment, that narrows the choice. Our industry pages describe how such rules shape the build in healthcare, fintech, and legal work.
• Quality on your task. Only a test on your own examples answers this.
• Volume and unit cost. A hosted model charges per token. A self-hosted model has server costs that do not fall when traffic is low.
• Latency and reliability. A hosted API depends on the provider's uptime. A self-hosted model depends on yours.
• Team capacity. Someone has to run the infrastructure behind an open-weight model.
• Cost of switching. How hard it would be to change models later.

Do not choose from a leaderboard

Chatbot Arena ranks models from pairwise votes by human users. Its authors report more than 240,000 votes and good agreement between the crowd's votes and expert raters (Chiang et al., 2024). That makes it a reasonable signal of general chat quality. It says little about how a model performs on your contracts or your support tickets.

The HELM project evaluated 30 models on 42 scenarios and, where possible, measured seven things: accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency, so that trade-offs show up instead of hiding behind one number (Liang et al., 2022). Accuracy is one measure among several, and the right model depends on the scenario.

So shortlist two or three candidates, run your own evaluation set on each, and compare quality, latency, and cost per request. For a RAG task, our post on evaluating a RAG system shows how to build that set.

Open-weight models: what you gain and take on

You gain control. Your data stays in your environment. You decide when the model version changes, because the weights you deployed do not change underneath you. You pay no per-token fee, though you still pay for the servers.

You take on the work: serving infrastructure, scaling, security updates, monitoring, and an evaluation of each new release you consider.

Open models can be competitive. The Mistral 7B paper reported that it outperformed Llama 2 13B on all the benchmarks its authors evaluated, and released it under the Apache 2.0 license (Jiang et al., 2023). The Llama 2 paper released models from 7 billion to 70 billion parameters, and its authors said human evaluations of helpfulness and safety suggested the chat models may be a suitable substitute for closed-source models (Touvron et al., 2023). Both results date from 2023, and the field moves quickly. They show that self-hosting is viable. They do not tell you which model to pick today.

Hosted models: what you gain and take on

You gain a fast start and access to the strongest models a provider offers, with no infrastructure to run.

You take on some dependence. Your data goes to the provider, so read the terms on how long it is kept and whether it can be used for training. The provider sets the price and decides when model versions change or retire, so plan to re-run your evaluation when they do.

Providers offer some control over where processing happens. Anthropic's price list, for example, shows a 1.1 times multiplier for US-only inference on its Claude 4.6 and later models, and it says Claude models are available on Amazon Bedrock and Google Cloud. Where a model is sold through your own cloud platform, the traffic can fall under your existing cloud agreement, but check the terms for your case.

Design so you can switch

The choice does not have to be permanent. Keep your prompts and evaluation sets in your repository. Put a thin layer between the application and the model API, so that changing models means changing one place. Before you switch, run the evaluation on the candidate. With those three habits in place, moving from a hosted model to an open-weight one, or the reverse, becomes a project with a test, not a rewrite.

An illustration

This is a made-up example, not a client result. A regional health system wants to summarize referral letters, and its rules keep patient data inside its own cloud account. It shortlists a self-hosted open-weight model and a hosted model offered through its cloud platform. It builds an evaluation set from de-identified letters, runs both candidates on it, and compares summary quality, response time, and cost per letter. If the two are close on quality, cost and the team's capacity to run servers decide. If one is clearly better, quality decides.

A short decision guide

• Rules require data to stay in your environment: evaluate an open-weight model and a hosted model deployed in your own cloud account, if that platform's terms meet your rules.
• You need the best quality quickly and your data rules allow it: start hosted.
• Traffic is high and steady, and you have a team to run infrastructure: price the open-weight route against per-token fees.
• You are unsure: start hosted, keep the switching layer, and revisit once you have quality and cost data from real use.

Our MLOps and production deployment page describes the deployment work behind either route, and our consulting page covers the decision itself.