How to interview an AI vendor (without being fooled)
Five questions that separate the engineers from the slide-makers — based on demos we've sat through.
We've sat through more vendor demos than we can count. The signal-to-noise ratio is grim, but a few questions reliably separate real engineering teams from polished sales orgs.
Ask: "what happens when the model returns garbage?" The answer should be specific, not "we have eval suites" — they should describe the fallback behavior and how often it triggers.
Ask: "show me your most recent on-call ticket about an LLM issue, what caused it, and what you changed." If they can't answer, they don't run anything in production.
Ask: "what model are you on, and when did you last upgrade?" If the answer is "we use whatever is best", they're flexible. If it's "we're on GPT-3.5 because we evaluated newer ones and they hallucinate more", they're engineers.
Keep reading
More from the blog.
A practical AI readiness checklist for 50–500 person teams
The questions to ask before any vendor demo, and the data work nobody wants to do first.
Apr 02, 2026 · 8 min read
Vendor selection: the AI tools we actually use
A snapshot of our current AI stack — what we deploy for clients, why, and what we avoid.
Feb 12, 2026 · 5 min read
The unsexy work that makes AI projects succeed
Data cleaning, schema design, edge-case mapping. The 60% of an AI project that no one wants to talk about.
Jan 29, 2026 · 6 min read
Get started
Want this in your inbox?
We email occasionally — when there's something genuinely useful to share. No spam.