Why AI vendors need their own review
Standard third-party risk questionnaires cover access control, encryption and business continuity. They miss four risks specific to AI products:
- Your data may train the model. Prompts, inputs and uploaded files can be retained or used for training unless the contract or plan says otherwise.
- Outputs can be wrong in ways a demo won't show. Accuracy shifts with model updates, prompt wording and data the vendor never tested on.
- The product changes underneath you. Vendors update or swap foundation models, sometimes without notice, which can change behavior, data flows and sub-processors.
- There is a hidden supply chain. Many AI tools call a third-party model API, so your data may pass through a provider you never assessed.
Treat an AI vendor review as your existing third-party risk process plus these questions, not a separate program.
How to assess AI vendor risk in 6 steps
- 1Classify the use case. Note what data the tool will touch (public, internal, confidential or regulated) and whether its outputs inform decisions about people. That sets how deep the review goes.
- 2Map the data flow. Ask where inputs go, who processes them, how long they are kept, and whether they train any model. Request the sub-processor list, including model providers.
- 3Send the questionnaire. Use the six categories below, and ask for evidence rather than yes/no answers.
- 4Verify independently. Check claims against third-party audits (SOC 2, ISO 27001, ISO/IEC 42001), published CVEs, incident history and independent evaluations such as huby's reports.
- 5Pilot with your own tasks. Run a short trial on representative, non-sensitive work. Record accuracy, failure modes and which admin controls actually exist on your tier.
- 6Decide and set review triggers. Approve, approve with conditions (for example enterprise tier only, no regulated data), or reject. Re-review after model changes, incidents, or at least once a year.
AI vendor risk assessment questionnaire
30 questions across huby's six evaluation categories. Copy them freely or download the PDF (no sign-up).
| # | Category | Question | Evidence to request |
|---|---|---|---|
| 1 | Quality | What accuracy or performance benchmarks do you publish, and were any run independently? | Benchmark reports with method and test date |
| 2 | Quality | How do you test for errors and regressions before releasing a model update? | QA process, release notes |
| 3 | Quality | How will we be notified of model or behavior changes? | Changelog, notification policy |
| 4 | Quality | What are the known failure modes and limitations for our use case? | Model card or limitations doc |
| 5 | Quality | What defect and resolution-time data can you share? | Support SLAs, incident metrics |
| 6 | Privacy | Are our inputs, outputs or files used to train or improve any model? Can we opt out by contract? | DPA, plan terms |
| 7 | Privacy | How long are inputs and outputs retained, and can we delete them on request? | Retention schedule, deletion procedure |
| 8 | Privacy | Which sub-processors, including model providers, process our data, and where? | Sub-processor list, data residency |
| 9 | Privacy | Which certifications and regulations apply (SOC 2, GDPR, CCPA/CPRA, HIPAA)? | Audit reports, signed DPA, BAA if needed |
| 10 | Privacy | How do users exercise access, export and deletion rights? | Privacy policy, admin docs |
| 11 | Security | When was your last independent penetration test, and what was remediated? | Pen test summary, remediation log |
| 12 | Security | Do you support SSO, MFA and role-based access? | Admin documentation |
| 13 | Security | How is data encrypted in transit and at rest, and who holds the keys? | Security whitepaper |
| 14 | Security | How do you defend against prompt injection, cross-customer data leakage and model abuse? | AI security controls, red-team results |
| 15 | Security | Which security incidents or CVEs affected the product in the past 24 months? | Incident history, CVE records |
| 16 | Use Cases & Pricing | Which of our use cases are supported today, and which are roadmap? | Product docs, dated roadmap |
| 17 | Use Cases & Pricing | Which integrations and APIs are included on the tier we would buy? | Tier comparison |
| 18 | Use Cases & Pricing | How is usage metered, and what happens at limits? | Pricing page, overage terms |
| 19 | Use Cases & Pricing | Which data-protection features are gated to higher tiers? | Tier feature matrix |
| 20 | Use Cases & Pricing | What does exit look like: export format and transition support? | Exit and termination terms |
| 21 | Sustainability & Reliability | What uptime do you commit to, and what was actual uptime over the past 12 months? | SLA, status page history |
| 22 | Sustainability & Reliability | What happens to our service if your model provider changes terms or fails? | Continuity plan |
| 23 | Sustainability & Reliability | What evidence of financial stability can you share? | Funding status, references |
| 24 | Sustainability & Reliability | What is your incident response and customer notification timeline? | Incident response policy |
| 25 | Sustainability & Reliability | Which customers of similar size use the product today? | Named references, case studies |
| 26 | Impact, Ethics & Safety | Do you publish an ethics or acceptable-use policy, and how is it enforced? | Policy, enforcement examples |
| 27 | Impact, Ethics & Safety | How do you test for and reduce bias in outputs that affect people? | Bias testing results |
| 28 | Impact, Ethics & Safety | What human oversight do you recommend for consequential outputs? | Usage guidance |
| 29 | Impact, Ethics & Safety | How do you handle copyright in generated content, and do you offer IP indemnity? | Terms, indemnity clause |
| 30 | Impact, Ethics & Safety | How have you responded to past safety or ethics incidents? | Incident reports, public statements |
What independent scores show
Security varies widely within the same product category, so verify each vendor rather than the category. Across the 24 AI products huby scores in three categories, as of October 2026:
| Category | Products | Avg Security (1.0–5.0) | Security range | Strong overall (4.0+) |
|---|---|---|---|---|
| AI chatbots | 6 | 3.6 | 2.6-4.2 | 3 |
| Coding assistants | 9 | 3.5 | 2.8-4.1 | 3 |
| AI detectors | 9 | 3.0 | 2.3-3.7 | 0 |
Six of the 24 products (25%) sit in the Weak band (2.0–2.9) on Security, and only five reach Strong (4.0+) on Security. See each leaderboard for a product's evidence.
Red flags
- The vendor won't state in writing whether your data trains models.
- No sub-processor list, or one that leaves out model providers.
- Data-protection controls exist only on a tier you aren't buying.
- Self-attestation only, with no independent audit.
- No policy for notifying customers of model changes.
Frequently asked questions
- How do you evaluate AI vendors?
- Start with the use case and the data the tool will touch, then map where that data goes, including any model providers underneath. Send a questionnaire that asks for evidence across quality, privacy, security, pricing, reliability and ethics. Verify claims against independent audits and evaluations, pilot with your own tasks, and set triggers for re-review when the vendor changes models.
- How do you perform an AI risk assessment?
- Identify the AI system and its purpose, list who and what it affects, and rate risks by likelihood and impact across accuracy, privacy, security, fairness and misuse. Record existing controls, decide what is acceptable, and assign owners for the rest. The NIST AI RMF (Govern, Map, Measure, Manage) and ISO/IEC 42001 both give a structure to follow.
- How do you perform a vendor risk assessment?
- Tier vendors by the data and processes they touch, send a security and privacy questionnaire scaled to that tier, and verify answers with evidence such as SOC 2 reports and penetration tests. Score the remaining risk, agree contract terms that close gaps, and monitor the vendor over time. For AI vendors, add questions on training data use, model providers and output testing.
- Can AI write a risk assessment?
- AI can draft one: it can summarize vendor documents, map answers to a framework and flag missing evidence. It can't verify claims or own the decision. Treat AI output as a first draft a reviewer checks against source evidence, and never paste confidential vendor documents into a tool whose data terms you haven't assessed.
Check a vendor's independent scores on huby before you send the questionnaire.
See the leaderboards