How to Evaluate a DeployCo: A Checklist for Engineering Leaders Choosing an AI Deployment Partner
Choosing an AI deployment partner is a procurement decision with unusual stakes: the vendor will sit inside your systems, build software your operations depend on, and - if it is lab-backed - has a commercial interest in which model you standardize on. This page turns the public descriptions of how DeployCos work, and the published critiques of them, into a checklist. It is neutral between lab-backed, independent and hyperscaler options; the right answer depends on your constraints.
Background reading: How DeployCos Work, Types of DeployCos, Risks and Debates.
The five evaluation criteria
Category descriptions converge on five things a CTO should measure a DeployCo engagement against 1:
| Criterion | What to measure | Red flag |
|---|---|---|
| Production accuracy and exception handling | Eval score on your data, not the vendor's demo set; what happens to the cases the agent cannot handle | No eval suite; "accuracy" quoted without a denominator |
| Integration depth | Which systems of record the agent reads and writes; whether it works through your APIs or screen-scrapes | Integration deferred to "phase two" |
| Operating cost | Cost per task in production including model tokens, infrastructure, human review and vendor fees | Pilot pricing with no production unit cost |
| Governance and auditability | Approval gates, logging, data residency, credential scope, compliance ownership | Logs not exportable; compliance "handled by the platform" |
| Business ROI | A baseline metric agreed before the build and measured after | ROI expressed as hours "freed up" with no financial owner |
Questions to ask before the diagnostic
1. Who will do the work, and where will they sit?
Ask for the names and backgrounds of the engineers who will be embedded, how many days per week they will be on site or in your systems, and whether any of the build is subcontracted. Ode said more than half of its launch team were former founders 2; AWS describes pods of engineers for 45-day engagements 3. A firm that sends strategists and outsources the build is a consultancy with a newer label. See Forward Deployed Engineers.
2. What does the diagnostic cost, how long is it, and what do we get?
OpenAI's Deployment Company begins engagements with a diagnostic followed by a small number of priority workflows 4. Independent firms recommend a paid diagnostic of two to four weeks ending in a ranked list of candidate workflows and a build, buy or wait recommendation 5. Get the price and deliverable in writing.
3. Which model will go to production, and why?
This is the central question for a lab-backed partner. OpenAI's venture builds on OpenAI models 4; Ode is Claude-first but will use rivals where required 2; independents and hyperscalers vary. Independent practitioners suggest asking whether the partner will publish a head-to-head model evaluation on your task in the first week; a partner that always proposes the same model without evidence is selling lock-in 5. Note that the argument comes from independents with their own commercial interest.
4. Who owns the code, the eval set, the data and any fine-tuned weights when the engagement ends?
The harness and the evals are the durable assets. Independents commonly state that the customer owns all of them; lab-backed terms are negotiated per engagement 5. Get it in the contract.
5. What is the exit plan?
AWS designs engagements to end with an exit strategy so the customer can run the system itself 3. Ask any partner the same: who operates the system after month six, what documentation and training are included, and what it costs to keep the partner on.
Questions to ask about the first workflow
6. Is the workflow bounded and checkable?
The playbook works best where tasks are "somewhat bounded and the outputs can be checked and verified" 6. If the vendor's proposed first workflow is open-ended, ask why.
7. What is the baseline, and who owns the number?
Agree the metric (straight-through rate, cost per file, handle time, days to decision) and its current value before the build starts. Assign a business owner who will sign off on the result.
8. What does the eval suite look like?
How many real examples, who labels them, how often they are refreshed, what score clears the bar for go-live, and whether you can run the suite yourself.
9. What requires human approval, and how does autonomy expand?
Consequential actions should require a person until the agent has earned autonomy within limits you set, with every action and approval logged 7. Ask to see the approval policy and a sample of the log.
10. What is the production unit cost?
Tokens, infrastructure, human review and vendor fees per task, projected at your volume. Fournier notes that newer reasoning models can cost more per task because they use more tokens on hard problems 8; your cost model should account for that.
Questions to ask about the vendor
11. Which investors and partners does the vendor have, and does our advisor hold an interest?
McKinsey, Bain & Company and Capgemini are investors in the OpenAI Deployment Company; Accenture, Deloitte and PwC are in Anthropic's partner network 49. If the firm recommending a DeployCo has an interest in it, you should know. See the consultancy paradox in Risks and Debates.
12. Can the vendor describe three engagements it stopped early?
A firm that has never stopped an engagement either has not done many or does not stop when it should 5.
13. What happens if the vendor's parent changes model pricing, deprecates a model, or has an outage?
Portability of the harness to another model, contractual notice periods, and fallback behavior.
14. What is the vendor's financial structure and incentive?
A reported 17.5% minimum return to the OpenAI Deployment Company's backers (Financial Times; unconfirmed by Reuters) would give that vendor an incentive to expand scope quickly 10. That is not disqualifying, but it is context for every upsell.
Scoring template
A simple weighted score helps compare lab-backed, independent and hyperscaler options on the same basis. Weights are illustrative; adjust for your situation.
| Dimension | Weight | Lab-backed | Independent | Hyperscaler |
|---|---|---|---|---|
| Frontier model access and speed | 20% | |||
| Model neutrality and portability | 15% | |||
| Engineering depth of the embedded team | 20% | |||
| Governance, auditability, compliance fit | 15% | |||
| IP ownership at exit | 10% | |||
| Production unit cost | 10% | |||
| Commercial terms and incentives | 10% |
What good looks like
A credible DeployCo proposal, regardless of backing, contains: a short paid diagnostic with a defined output; a first workflow that is bounded and measurable with a baseline you agreed; named engineers and their on-site cadence; an eval suite you can run; an approval policy and exportable logs; a production unit cost; IP and data terms you understand; and an exit plan. Anything described as "phase two" should be priced.
Frequently asked questions
Should we prefer a lab-backed DeployCo or an independent?
Lab-backed firms offer frontier access and speed; independents offer model neutrality and usually clearer IP ownership. If multi-vendor optionality is a hard requirement, lean independent; if you have standardized on one model family and need speed, lab-backed firms are built for that 5.
How long should the diagnostic take?
What is the single most important contract term?
Ownership of the harness, eval set, data and any weights at exit, because it determines your switching cost.
How do we measure ROI on an AI deployment?
Agree a financial baseline before the build (cost per task, cycle time, revenue per rep) and have a business owner measure it after. Treat "hours freed" as a leading indicator, not a result.
References
- DeployCos.com editorial synthesis; criteria appear across the OpenAI, Anthropic and AWS launch materials and in practitioner guidance cited below.
- TechCrunch via The Next Web, "Anthropic and Blackstone bet the next fortune in AI is implementation, not models," 07-16-2026. thenextweb.com
- Constellation Research, "AWS launches forward deployed engineering unit with $1 billion investment," 06-30-2026. constellationr.com
- OpenAI, "OpenAI launches the OpenAI Deployment Company to help businesses build around intelligence," 05-11-2026. openai.com
- QuantSpark, "Choosing a forward-deployed AI partner" (vendor publication; the "five questions" framework). quantspark.ai
- Tanay Jaipuria, "The Rise of DeployCos," 10-13-2026. tanayj.com
- DeployCo (deployco.co), company site describing approval-gated agents with exportable logs (independent, unrelated to OpenAI). deployco.co
- The Decoder, "OpenAI's deployment chief on Codex growth, falling AI prices, and the ROI question," 2026. the-decoder.com
- Anthropic, "Building a new enterprise AI services company with Blackstone, Hellman & Friedman, and Goldman Sachs," 05-04-2026. anthropic.com
- Financial Times reporting as summarized by The Next Web, 04-22-2026; The Decoder on Reuters' inability to confirm, 05-2026. thenextweb.com ; the-decoder.com