Since generative AI took off, nearly every IT vendor has added “AI” to its homepage. Some are excellent, and some wrapped an API last month. A demo will not tell you which is which. These questions will.
Experience
1. Show me something running in production, and tell me what broke. Real experience comes with war stories. If everything went perfectly, they have not shipped much.
2. Who exactly would work on my project? Meet the engineers, not only the sales team.
Quality
3. How do you measure quality? Look for evaluation sets, regression tests and tracked metrics. “We test it thoroughly” is not an answer.
4. What happens when a model is updated? They should describe a process: run evaluations, roll out in stages, keep a fallback.
5. How do you handle wrong answers? The honest answer admits they will happen, and explains how they are reduced, detected and escalated.
Security
6. Where does my data go? Expect a clear data-flow diagram, and a firm position on training and retention.
7. How do you defend against prompt injection? A blank look here is a serious warning sign for any agent project.
Commercials
8. Who owns the code, prompts and evaluation data? It should be you.
9. Can we change model provider later? Good architecture keeps this cheap.
10. What would make you tell us not to do this? The best partners turn work down. It shows that they care about the outcome and not only the invoice.
Red flags
- Accuracy promises before seeing your data
- A proposal with no evaluation budget
- No mention of running costs
- One-model-fits-all recommendations
- Demos only on data the vendor chose
How to run the selection
Shortlist three. Give each the same paid mini-task on a sample of your real data, and compare the thinking as well as the output. A week of this will tell you more than a dozen pitch decks.
We built the AI Vendor Scorecard to make this easy, and yes, you can use it on us. When you are ready, book a strategy call.