A 2026 Ncontracts survey of third-party risk management found that 72% of organizations are only partially aware of which vendors use AI, while 16% haven't assessed vendor AI usage at all. If you're about to sign an AI-enabled sales, marketing, support, or forecasting contract, the standard security questionnaire won't give you enough protection.
We recommend treating AI vendor risk as a concentration and change-management problem. Score the vendor across governance, data, model behavior, and operational resilience. Put training-data restrictions, model-swap notice, sub-processor disclosure, and exit rights in the contract. Then reopen the assessment whenever the vendor changes the system underneath you.
Table of Contents
- The AI Vendor Risk Problem Nobody Has Mapped
- The Weighted Scorecard Tied to NIST AI RMF
- Pressure-Testing the Model and Data Layer
- Contract Clauses That Have to Be in Writing
- The Concentration Risk Most Checklists Miss
- Quarterly Re-Review and Change Triggers
- Run the Scorecard Against One Vendor This Week
The AI Vendor Risk Problem Nobody Has Mapped
A vendor can pass your SOC 2 review and still create an unacceptable AI risk. Traditional SIG and SIG-Lite questionnaires usually cover encryption, access control, business continuity, and audit reports. Those controls matter, but they rarely force a vendor to disclose training-data provenance, model replacement rights, fourth-party AI dependencies, or drift monitoring.
That gap creates a familiar scenario for growth-stage B2B companies. A sales AI tool receives customer call transcripts, fine-tunes its system on those transcripts without clear disclosure, then changes the underlying large language model during the contract term. Your team sees the same interface. The system's behavior, data flows, and liability position may have changed.
An AI vendor failure can combine three problems at once:
- Data exposure: Customer inputs, outputs, or embeddings may move through undisclosed AI providers or remain in systems longer than your privacy team approved.
- Model behavior change: A model swap, fine-tune, prompt change, or new automation layer can alter output quality and risk.
- Contractual weakness: Your agreement may give the vendor broad rights to change subprocessors or models without notice, remediation rights, or an exit path.
Four buckets, four owners
A practical AI vendor risk assessment needs four separate buckets. Each bucket belongs to a different buyer, which is why a single security review tends to collapse under its own scope.
Governance belongs to legal, compliance, and executive risk owners. Ask who owns the vendor's AI policy, whether an AI risk committee exists, how the vendor maps controls to NIST AI RMF or ISO/IEC 42001, and who takes charge during an AI incident. NIST's framework is voluntary and includes third-party software, data, system components, and supply-chain risks, as explained in this overview of the NIST AI Risk Management Framework.
Data belongs to privacy and security. Require answers about training-data rights, customer-data isolation, PII handling, retention, deletion, geographic processing, and the full subprocessor chain. “Your data isn't used for training” means little unless the DPA covers inputs, outputs, embeddings, logs, support systems, and every AI subprocessor.
Model belongs to the technical buyer. Review model provenance, fine-tuning rights, evaluation methods, hallucination testing, guardrails, model versioning, and the process for replacing the foundation model. Request evidence, not a product page.
Operational resilience belongs to operations and the process owner. Review uptime and latency commitments, drift monitoring, rollback capability, manual fallback, data portability, and exit assistance. A service can remain available while its outputs become unusable.
Each bucket needs an owner, a score, and an evidence file. Your security team shouldn't be expected to validate training-data rights, and your legal team shouldn't be left to infer model dependencies from a marketing brief.
For baseline context on what conventional software reviews typically examine, SOC2Auditors' SOC 2 software review is useful. Use that material as a starting point, then add AI-specific controls instead of assuming a general assurance report answers them.
The Weighted Scorecard Tied to NIST AI RMF
A vendor questionnaire collects statements. A weighted scorecard turns those statements into a decision.
We use a 1 to 5 raw score. A score of 1 means the vendor has no meaningful policy or evidence. A score of 5 means the control is documented, tested, and independently audited. Apply a risk weight from 1 to 3. Weight 1 represents a low blast radius. Weight 3 applies when the tool is customer-facing, mission-critical, or trained on customer data.
The category allocation should reflect where AI creates exposure:
- Governance, 20%
- Data, 25%
- Model, 30%
- Operational Resilience, 25%
| NIST AI RMF Function | Category | Weight (%) | Raw Score (1-5) | Weighted Score | Pass Threshold |
|---|---|---|---|---|---|
| Govern | Governance | 20 | 4 | 16 of 20 | 60% of category maximum |
| Map | Data | 25 | 4 | 12 of 25 at risk weight 3 | 60% of category maximum |
| Measure | Model | 30 | 3 | 18 of 30 | 60% of category maximum |
| Manage | Operational Resilience | 25 | 4 | 20 of 25 | 60% of category maximum |
The arithmetic matters. A vendor scoring 4 on data with a risk weight of 3 contributes 12 of 25 in the example. A score of 4 on governance with a risk weight of 2 contributes 8 of 20. Keep the raw score, risk weight, evidence reference, owner, and remediation decision in the same audit trail.
Our stop-deal rule is direct. Any weighted category below 60% of its maximum blocks signature. An overall score below 70% triggers renegotiation, even when no individual category falls below the stop-deal line. A sales AI vendor might reach 78% overall and still fail because its Model category scores 54%. That vendor stays blocked until the model evidence and contract protections improve.
Map the scorecard rows to NIST's Govern, Map, Measure, and Manage functions. You can also map the same evidence to ISO/IEC 42001 controls. For teams managing broader control debt, the recommended risk technical debt approach from Faberwork LLC offers a useful way to record exceptions rather than allowing them to disappear after procurement.
Use the Stimulead guide to AI governance best practices when assigning internal ownership and escalation paths.
Pressure-Testing the Model and Data Layer
Treat the vendor's model claim as unverified until the procurement lead can answer five questions in writing:
- Where did the base model come from?
- What data was it trained on?
- What will be fine-tuned on your data?
- Who owns the weights, embeddings, prompts, and derived artifacts?
- Which foundation model sits underneath the vendor's wrapper?
The answer “we use proprietary models” is incomplete without provenance documentation. A refusal to name fourth parties such as OpenAI or Anthropic is a procurement blocker when those parties can process your data. A clause saying “your data is never used for training” also needs a DPA addendum that names the training-data opt-out and applies it across subprocessors.
| Question to Ask | Red-Flag Answer | Proof Required |
|---|---|---|
| What is the base model and its version? | “We use proprietary models.” | Model card, system card, version history |
| What data trains or fine-tunes the system? | “We use industry data.” | Data lineage, rights statement, training policy |
| Which providers receive inputs or embeddings? | “Our cloud partners are confidential.” | Named subprocessor and fourth-party register |
| How do you test output quality and misuse? | “The model is tested continuously.” | Evaluation suite and red-team report |
| How are model changes controlled? | “Swaps happen with no impact.” | Change process, notice record, rollback evidence |
Request the model card, system card, evaluation suite, and a sample red-team report. Ask for named dependencies covering vector databases, embedding providers, inference hosts, cloud regions, and monitoring services. The NIST AI RMF Playbook for generative AI supports a use-case-based supplier review that examines provenance, anomalies, unauthorized changes, value-chain risk, and legal compliance.
Reject “no impact” as a model-swap answer. The vendor may believe the interface remains stable, but you need evidence about output quality, latency, token cost, refusal behavior, and data handling. Deduct points in both the Model and Data columns when the vendor can't prove equivalence.
For a practical testing sequence, use the Stimulead AI testing framework.
Contract Clauses That Have to Be in Writing
The security questionnaire is evidence for negotiation. It isn't a control until the obligation appears in the signed MSA, DPA, or order form.
| Clause | Must-Have Language | Typical Pushback | Walk-Away Trigger |
|---|---|---|---|
| Training-data opt-out | Customer inputs, outputs, prompts, embeddings, and derived data won't train or improve any model across the vendor or its subprocessors. | “We need broad rights to improve the service.” | Vendor refuses an explicit opt-out covering subprocessors. |
| Model-change notification | Vendor gives 30 days' notice before a material model, provider, or processing change, with remediation and termination rights. | “We can't predict research or infrastructure changes.” | Vendor can swap a material model with no notice or remedy. |
| Subprocessor disclosure | Current list, change notice, objection right, and named AI, embedding, vector, inference, and cloud providers. | “Our standard DPA only lists material subprocessors.” | Vendor refuses to identify AI subprocessors with data access. |
| AI-specific indemnity | Indemnity covers AI-specific data misuse and agreed output-IP claims outside the generic liability ceiling. | “All indemnities share the standard cap.” | No meaningful remedy for prohibited training or covered IP claims. |
| Exit and portability | Return data and embeddings in a documented portable format, with transition assistance and deletion certification. | “Export is available through standard APIs.” | No usable export, deletion confirmation, or transition support. |
Push for a model-change clause that defines “material.” Include a foundation-model replacement, fine-tuning process, new training use, new data location, new inference provider, or material change to guardrails. Your fallback can narrow the notice period for low-risk changes, but preserve a remediation right for customer-facing or regulated workflows.
For liability, separate AI-specific exposure from the generic SaaS cap where the vendor controls the relevant conduct. Review the practical implications of EU liability rules for AI and software, then have counsel map the contract to your actual use case and jurisdictions.
The Stimulead explanation of AI consulting gives executives useful context on where vendor evaluation ends and implementation oversight begins. Those responsibilities should be clear before signature.
The Concentration Risk Most Checklists Miss
The dangerous dependency may be a capable model shared by several vendors. A Cloud Security Alliance survey on AI provider concentration risk found that 91% of senior executives didn't fully understand their AI dependencies, while 81% said a seven-day vendor outage would cause severe or critical disruption.
Map the dependency graph behind each critical workflow. One foundation model may sit under the chatbot, sales copilot, call summarizer, enrichment tool, and content assistant. One vector database may hold embeddings from several systems. One inference provider may serve multiple applications in the same region.
Start with each vendor's subprocessor list. Trace the named providers back to foundation models, embedding services, cloud infrastructure, and data brokers. Then group your business workflows by underlying dependency, not by the visible software brand.
Practical rule: A vendor inventory tells you who you bought from. A dependency map tells you what can fail together.
Set a concentration ceiling for revenue-generating and customer-facing workflows. The exact ceiling should reflect your tolerance, but the policy must prevent one underlying model from becoming the default for every important process. Name an alternate provider for each critical path, document the manual fallback, and test whether your data and prompts can move.
The same review should cover fourth-party change rights. If your direct vendor can add a model provider without consent, your concentration exposure can increase after signature. Require disclosure when a new provider becomes material to the service, then re-score the affected category.
A vendor can be secure in isolation and still be the wrong choice for your stack. The buyer's job is to find shared failure points before they become operational dependencies.
Quarterly Re-Review and Change Triggers
Annual vendor reviews are too slow for AI systems that can change through model swaps, prompt updates, new providers, or added automation. Set a quarterly review cadence and reopen the scorecard whenever a material trigger appears.
The four triggers are:
- Foundation-model change: The vendor announces a model swap or a new fine-tune.
- Subprocessor change: A new AI, embedding, vector, inference, or cloud provider enters the DPA.
- Performance breach: Drift or bias metrics cross the thresholds set during onboarding.
- External event: A regulatory action or public incident affects the vendor or a material dependency.
Require a quarterly evidence packet. The vendor should return an updated model card, current subprocessor register, SOC 2 bridge letter, drift report, and a redlined diff of any terms-of-service changes. If the vendor won't provide the packet, record the missing evidence as a score deduction rather than accepting a verbal assurance.
Your buyer-side owner should monitor the system in production. Track sampled output quality, latency outliers, unexplained cost spikes, refusal patterns, and material changes in workflow completion. The product owner can review output samples, while security owns incidents and procurement owns contractual notice compliance.
Ncontracts reported that 72% of financial institutions were only partially aware of which vendors use AI, and no organization in that survey felt extremely confident managing AI-related vendor risk, according to its 2026 survey report. That makes ownership and recurring evidence more important than another annual questionnaire.
Every trigger reopens the weighted scorecard. Apply the same below-60% category threshold used at signing. Don't allow a vendor to pass once and remain approved forever.
Run the Scorecard Against One Vendor This Week
Pick one AI vendor already on your books. Choose a sales-intent tool, chatbot, call-analysis system, enrichment platform, or forecasting model that touches customer data or a critical workflow.
Pull the latest DPA, subprocessor list, security questionnaire, model documentation, incident terms, and service description from the vendor portal. Assign one owner for each bucket, score every line item from 1 to 5, apply risk weights from 1 to 3, and record the evidence URL or document name beside each score.
Independent research summarized in AI vendor due diligence guidance found that 40% of organizations were adding AI-usage language to vendor contracts, 32% were using questionnaires, and 38% were using verbal communication to monitor vendor AI use. The same source reported that 14% collected vendor AI documentation, while 23% weren't monitoring vendor AI usage. Your scorecard should force the evidence collection that verbal conversations miss.
| Weighted Score | Decision | Required Action This Week | Owner |
|---|---|---|---|
| Above 80% | Keep | Schedule the quarterly review and close minor evidence gaps. | Vendor owner and security |
| 60% to 80% | Renegotiate | Add training-data, model-change, subprocessor, indemnity, and exit protections before renewal. | Legal and procurement |
| Below 60% | Escalate | Take the vendor to the security committee and build a replatform shortlist. | Executive sponsor and security |
A benchmark cited in the same due-diligence source reported that 59% of organizations had updated their inherent vendor risk assessment within the previous year, while 55% had updated vendor questionnaires and due-diligence document requirements during that period. Use that as a reason to refresh your process, not as permission to wait for a company-wide program.
Run the exercise against one vendor first. If the score exposes a shared foundation model, repeat the mapping across the other tools that depend on it. The CSA finding that 81% of executives expect severe or critical disruption from a seven-day AI vendor outage makes the next action clear: decide whether to renegotiate, replatform, or escalate before the dependency becomes business-critical.
Choose the AI vendor with the greatest access to customer data or the greatest effect on pipeline, complete the four-bucket scorecard this week, and send any category below 60% to legal and the security committee before renewal. Stimulead can help your leadership team map the dependency chain, negotiate the control gaps, and establish the quarterly review process without adding another unmanaged program to your team's workload.
