How to choose a generative AI development company

Before you sign with a generative AI development company, ask these six questions. Each comes with what a good answer sounds like. Use them on us too.

Ask what is running in production

A demo shows that a model can do something once. Production means doing it repeatedly, under real traffic, with real data and messy inputs.

Ask which systems the team has taken to production, who uses them today, and what broke in the first month. A vendor that can only describe demos has not met the hard part yet. Ask for a reference you can call. If confidentiality rules that out, ask for the architecture in outline and the failure they had to fix.

Ask how they measure quality

Generative AI output varies from one run to the next, so "it looks right" is not a test. A serious team builds an evaluation set from your real examples, runs it on every change, and shows you the scores. Our post on evaluating a RAG system shows what such a set contains.

Ask to see an evaluation set from a past project with the client details removed. Ask what score counts as good enough to launch and who decides. If the answer is a demo review with your team, plan for surprises after launch.

If the system uses retrieval, ask whether they score the retrieval step and the answer step separately. The Ragas paper describes three things worth measuring on their own: whether retrieval returns relevant passages, whether the model uses them faithfully, and the quality of the answer (Es et al., 2023). If they use a language model as an automated judge, ask how they check it. Research on LLM judges documents position, verbosity, and self-enhancement biases (Zheng et al., 2023), so a sample of judged answers should still be read by a person.

Ask who does the work

Agencies often sell with senior people and staff with junior ones. Ask who will write the code, how much relevant experience each person has, and whether the person you meet on the first call stays on the project.

Ask where your data goes

Every model call sends data somewhere. Ask which model providers the system will use, whether your data can train their models, where it is stored, how long it is kept, and whether the system can run inside your own cloud account.

For regulated data, ask which controls the vendor has in place today and request the documents, not a summary of them.

Ask how the price works

A fixed price hides unknowns and a time-and-materials price hides scope. Ask which one you are getting and what happens when the scope changes.

Ask separately about running costs. Model usage, hosting, and monitoring continue after launch. A build price with no running-cost estimate leaves out a cost you will keep paying.

Ask what you will own

The code, the prompts, the evaluation sets, and any tuned model weights should belong to you at the end. Ask for that in writing, and ask how long a handover to another team would take.

Signs to walk away

• A guaranteed accuracy figure before anyone has seen your data.
• A price with no discovery phase behind it.
• No plan for what the system does when it is wrong.
• A recommendation to fine-tune a model so it "learns" your documents, with no test of retrieval first. In one comparison, retrieval consistently beat unsupervised fine-tuning for adding knowledge (Ovadia et al., 2023).
• Case studies with no client, no date, and no method behind the numbers.
• A ranking list that includes the vendor that wrote it.

A note on "best" lists

Ranking lists rarely say how they ranked. Some are written by vendors that rank themselves, and some list whoever paid for a profile. Use a list to collect names, then run the questions above on each name.

If you want to test the list on us, bring your hardest question to a call. To see what we build, start with our services, each of which says when it is the wrong choice. For background on the work itself, read what generative AI development involves.