Comprehensive framework for evaluating AI vendors and solutions to avoid costly mistakes.
Version 1.0 | October 2025 | Based on $1.2M average AI spend analysis
This skill provides a systematic framework for evaluating AI vendors and solutions to avoid the costly mistakes that plague 95% of AI projects. Use when conducting vendor due diligence, evaluating proposals, negotiating contracts, or making strategic AI purchasing decisions.
Key capabilities:
Start here to determine which references to read:
What stage are you in?
├─ Early exploration (multiple vendors being considered)
│ └─ Read: evaluation-criteria.md, use-case-fit.md
│ Use: scorecard-template.xlsx
│
├─ Evaluating specific proposal or demo
│ └─ Read: red-flags.md, technical-assessment.md
│ Check: pricing-models.md for pricing reasonableness
│
├─ Contract negotiation
│ └─ Read: contract-checklist.md, pricing-models.md
│ Reference: red-flags.md for problematic terms
│
├─ Build vs Buy decision
│ └─ Read: build-vs-buy.md, use-case-fit.md
│ Consider: Total cost of ownership from pricing-models.md
│
└─ Post-purchase review or audit
└─ Read: evaluation-criteria.md, technical-assessment.md
Assess: Whether vendor is delivering on promises
Trigger scenarios:
Goal: Eliminate obviously problematic vendors before deep evaluation
Key questions:
Read: references/red-flags.md for disqualifying signals
Read: references/use-case-fit.md for domain fit assessment
Goal: Assess vendor capabilities systematically across all dimensions
Evaluation dimensions:
Read: references/evaluation-criteria.md for comprehensive framework
Read: references/technical-assessment.md for technical evaluation
Read: references/pricing-models.md for pricing analysis
Use: assets/scorecard-template.xlsx to score vendors systematically
Goal: Secure favorable terms and avoid costly traps
Critical areas:
Read: references/contract-checklist.md for essential terms
Reference: references/red-flags.md for problematic contract patterns
Characteristics: Claims to solve everything, vague on technical details, aggressive sales tactics
Red flag: "Our AI can handle any use case"
Response: Demand specific technical explanations and verifiable references
Characteristics: Long feature lists, complex pricing, unclear core value proposition
Red flag: Can't explain what problem they actually solve
Response: Force clarity on primary use case and success metrics
Characteristics: Software license + mandatory professional services
Red flag: Professional services cost more than software
Response: Assess true cost of ownership, consider if you're buying software or consulting
Characteristics: Thin layer over OpenAI/Anthropic APIs with high markup
Red flag: No proprietary technology, just API access + UI
Response: Calculate cost of building similar solution in-house
Full pattern library: See references/red-flags.md
When to read this section: Before committing to vendor evaluation, determine if building in-house is better option.
Key factors:
Read: references/build-vs-buy.md for detailed decision framework
The vendor scorecard enables structured comparison across vendors.
To use:
assets/scorecard-template.xlsxCustomization: Adjust weights based on priorities for your specific use case.
Comprehensive scoring framework across all vendor evaluation dimensions. Includes specific questions to ask, what constitutes good/bad answers, and how to weight criteria for different use cases.
Use when: Conducting systematic vendor evaluation
Catalog of warning signs indicating problematic vendors. Organized by category: technical red flags, business red flags, pricing red flags, contract red flags, and behavioral red flags.
Use when: Initial vendor screening or reviewing proposals
Guide to AI vendor pricing models (per-seat, usage-based, platform fees, etc.), fair market rates, what drives costs, and how to negotiate. Includes pricing red flags and total cost of ownership analysis.
Use when: Evaluating vendor pricing or negotiating contracts
Framework for assessing technical capabilities: architecture review, model evaluation, integration complexity, scalability, security, and data handling. Includes specific technical questions to ask.
Use when: Deep technical evaluation of vendor capabilities
Essential contract terms for AI vendor agreements: performance guarantees, data rights, pricing protection, exit terms, liability, and support commitments. Includes negotiation guidance.
Use when: Contract review or negotiation
Framework for assessing whether vendor solution actually fits your use case. Includes questions to ask yourself, questions to ask vendor, and warning signs of poor fit.
Use when: Initial vendor screening or use case definition
Decision framework for whether to build AI capability in-house vs purchasing vendor solution. Includes total cost analysis, capability assessment, and strategic considerations.
Use when: Before committing to vendor evaluation process
Structured spreadsheet for vendor comparison with:
Customize: Adjust criteria weights and add company-specific requirements