Lesson 2 — Critically Evaluating the Solution
Module 2, Unit 1 | Lesson 2 of 3
By the end of this lesson, you will be able to:
- Distinguish the four main AI solution archetypes — off-the-shelf product, pre-built API, fine-tuned foundation model, and custom build — and the realistic trade-offs of each (K7, K13)
- Apply a vendor and solution evaluation framework that is specific to AI, not borrowed wholesale from generic IT procurement (K24, S3, S5)
- Compare the total cost implications of the four AI solution archetypes and a non-AI option, including costs that do not appear in vendor pricing (S22, S15)
- Recognise when the right answer is not to build at all — and articulate that recommendation defensibly (B1, B2, B3)
Why solution evaluation is its own discipline
By the time a project reaches the how should we build this? question, the team has usually already fallen in love with one of the answers. The data scientists want to fine-tune; the procurement team wants to buy a product; the senior sponsor read an article last week and wants to use the API the article mentioned. These are not deliberations. They are preferences dressed up as analysis, and they tend to lock in a delivery path before the actual trade-offs have been examined.
Solution evaluation is the discipline of slowing down long enough to ask which of the realistic options would actually serve the problem best. It sits between problem framing (Unit 1 Lesson 1) and delivery planning (Unit 2 Lesson 3), and the portfolio asks for it by name — critically evaluate the solution is the third item on the list.
It earns a lesson of its own, rather than a sub-bullet under scope, because the options are not variations on each other. The four AI solution archetypes have radically different cost curves, time-to-value profiles, lock-in exposures, and capability implications. Choosing between them by instinct, or by what the team already knows how to build, is one of the more common ways AI initiatives end up technically successful and operationally regretted.
🔑 Key term: Solution archetype — the structural shape of how an AI capability will be delivered. The four most common are off-the-shelf product (a vendor application with AI inside), pre-built API (calling someone else's hosted model), fine-tuned foundation model (adapting a third-party model to your data), and custom build (training a model from scratch). Each archetype implies different costs, risks, and capabilities — and choosing between them is rarely the same decision as the technical implementation choices that follow.
The four archetypes — and the fifth option
They appear below in order of how much the organisation has to build for itself: licensing a finished product, then renting a model, then adapting one, then training one. Read that as a scale of effort, not a ladder of maturity.
These four archetypes are separate categories, not a journey. Choosing between them is a question of which one fits the problem you are trying to solve — not a question of which one is more advanced or more mature. Each represents a different bet about where the value sits: in the model itself, in the integration around it, or in the data it learns from. The fifth option — no AI for this problem — belongs in the comparison from the beginning, and has its own section at the end of this lesson.
Off-the-shelf product
The off-the-shelf product archetype is a vendor application that already wraps an AI capability — a customer-service platform with built-in summarisation, a recruitment system with built-in screening, a coding assistant. You are not building anything; you are licensing a product that has done the AI work for you. The strength is that you also inherit the vendor's investment in evaluation, fairness testing, security, and ongoing improvement — work that is invisible from outside but expensive to do well. The exposure is that you also inherit the vendor's roadmap. If they discontinue the feature, change the pricing, or interpret the model's behaviour differently from how you would, you have limited recourse.
Worked example. An insurer rolls out Microsoft 365 Copilot for the workforce, and licenses a vendor platform such as HireVue for first-stage screening of recruitment applications. The AI sits inside the vendor product; the insurer's effort goes into procurement, vendor management, governance sign-off, and configuring the product to its own policies. The insurer is buying capability, not building it.
Pre-built API
The pre-built API archetype calls a hosted model that someone else trained, hosts, and updates. OpenAI, Anthropic, Google, AWS, and others all offer general-purpose APIs for language, vision, and speech. The model is theirs; you supply the prompts and the integration. The strength is speed: you can prototype in hours and deploy in days. The exposure is that you do not control the model — its behaviour can change without notice, your data may or may not be used to improve it depending on contract terms, and the per-request cost can become expensive at scale. This is usually the right archetype for general-purpose tasks where the value sits in your integration, not in the model itself.
Worked example. A bank's customer-services team uses the OpenAI or Anthropic API to summarise call transcripts and draft follow-up emails. The team owns the prompt design and the integration into the ticketing system; the model itself is rented, not built. The bank's effort sits in the workflow around the model — quality checks, escalation rules, and the human review step before a draft is sent.
Fine-tuned foundation model
The fine-tuned foundation model archetype takes a general-purpose model and adapts it to your specific domain or task by training it further on your own data. The base model's general capability is preserved; the fine-tuning adds specificity. The strength is that you get behaviour shaped to your context — your terminology, your tone, your edge cases — without the cost of training from scratch. The exposure is that you now own a model artefact, with all the lifecycle obligations that brings: monitoring, retraining when the base model updates, evaluation, governance documentation. Fine-tuning is often pitched as "the best of both worlds" but is in practice a significant capability investment, not a quick adaptation.
Worked example. An insurer fine-tunes a foundation model on its own claims-handling notes so the system understands the firm's terminology, the recurring edge cases, and the tone its customers expect. The base model still belongs to its provider; the adaptation belongs to the insurer — and so does the ongoing duty to monitor, evaluate, and re-tune the model whenever the underlying base model updates.
Custom build
The custom build archetype trains a model from scratch on your own data. It is rare, and it is rarely the right answer. Custom build belongs at the end of a process of elimination, not at the start. The bar to clear is high: the data is genuinely proprietary and unavailable to existing models, the problem sits outside what foundation models already address, the organisation has the in-house ML and MLOps capability to maintain a model over years (not months), and the value of doing this in-house clearly exceeds the multi-year cost of building, evaluating, governing, and retraining it. Most problems that feel unique on first inspection turn out to be solvable with one of the three lighter archetypes — and a 'custom build' recommendation should be treated as suspicious until each of the lighter archetypes has been ruled out on the evidence.
Worked example. A research-led bank trains its own credit-risk model on decades of internal underwriting data because no external model has access to that history, and the regulatory regime requires the bank to own and audit the model end-to-end. The team accepts that this is a multi-year capability investment, not a project, and resources it accordingly.
Five questions that decide the archetype
Choosing between the options is not a checklist; it is a structured conversation about five trade-offs. Each has a short label, used for the rest of this lesson, and a question it actually asks. Your workbook asks for the same five trade-offs in its own longer wording — that wording is the middle column below, so you can map one onto the other.
| Criterion | The question it asks | What the answer points to |
|---|---|---|
| Value location | Where does the value actually sit — in your data, or in the integration, the workflow change, and the operational fit? | Value in the data makes fine-tuning or custom build worth examining. For most problems the value is in the integration, and a product or API is enough. |
| Time-to-value | Which delivery timeline matches how urgent the problem actually is? | Products and APIs deliver in weeks, fine-tuning in months, custom build far slower. The question is fit, not speed. |
| Capability built | What capability are you building inside the organisation, and is it one you want? | Vendor management, prompt design, data engineering, or applied machine learning. None is intrinsically better — but a plan that names none builds capability by accident. |
| Lock-in | What does this tie you to long-term, and is it a dependency you can live with? | Every option creates one: a vendor, a roadmap, a base model, or your own maintenance team. "How do we avoid lock-in?" is the wrong question. |
| Data and regulatory exposure | Where does the data go, who processes it, and which rules apply? | Check this at planning, not at deployment. Initiatives have been stopped at launch over data questions that should have been raised months earlier. |
Ask all five of the non-AI option as well. It is not exempt from the comparison just because there is no model in it.
Doing this with AI
Once you have a draft recommendation, paste it back to the model with this prompt: "Score all five options — the four archetypes and the no-AI alternative — against the five criteria from my evaluation, and tell me which option I have not seriously considered and why I should." That last part is what pays off — most teams have already chosen before the evaluation begins, and the model is reliably better than colleagues at calling it out.
Total cost of ownership: comparing the options
Total cost of ownership has a specific job at this stage: comparing delivery options for the same problem. It is not the project budget — Unit 3 builds that, once the solution approach and the delivery design are settled.
Vendor pricing rarely tells you which option costs less overall. The headline number — per-request API fees, annual licence, fine-tuning compute — is only one part of the comparison. Apply the same scope, the same usage assumptions, and the same time horizon to all five options, or you are not comparing them at all. The costs that actually separate the options sit in three categories.
Integration and adaptation. Hooking an AI system into existing workflows, identity systems, data pipelines, monitoring infrastructure, and audit logging is rarely trivial. For pre-built APIs and products this is often the dominant cost — frequently several times the licence fee. A budget that shows only the vendor cost is signalling that integration has not yet been thought about.
Operation and oversight. AI systems need monitoring for drift, for performance by group, and for error patterns. They need their evaluations refreshed and their outputs overseen by people. This is recurring, not one-off. For models you fine-tune or build, it also includes retraining cycles and the data engineering that keeps training data current. The common pattern is to budget for build and not for run, then discover the run cost six months in, once the system has already degraded.
Risk and governance. Documentation, fairness assessments, security reviews, regulatory engagement, and incident response capability are all real costs, and they scale with the sensitivity of the use case. A model behind an internal productivity tool carries a different governance cost from one that makes decisions about customers.
The temptation is to treat governance as overhead. The more accurate framing is that it is part of the delivery cost — and pretending otherwise produces budgets that fail at the gate they would have passed if they had been honest.
The total-cost picture frequently changes the archetype recommendation. A product can bundle operation but add licences and configuration limits. A pre-built API can trade low setup cost for usage-linked cost and vendor dependency. Fine-tuning adds data preparation, evaluation, and retraining. Custom build carries the greatest specialist capability and maintenance burden. A non-AI workflow or rules-based option may cost less to govern and operate while still solving the root problem.
The point is not to forecast an approved budget at this stage. It is to expose how each option creates a different cost shape, and to use that as evidence in the recommendation.
Did you know?
In 2023, Bloomberg announced BloombergGPT — an AI model they had built from scratch on their own financial data. At the time it was held up as the example of doing AI seriously: train your own model, on your own data, for your own industry. About eighteen months later, general-purpose models like GPT-4 and Claude, given the right prompts, were doing as well or better on the same finance tasks. The point is not that custom build never works. The point is that the choice of archetype has to take into account how fast the field is moving — what feels like a smart, defensible build today can be overtaken before the project even finishes.
When the right answer is no AI for this problem
The fifth option is no AI for this problem. You are AI practitioners — making this recommendation is not abandoning the discipline; it is refusing to apply it where it will not help.
Sometimes a process change, a policy fix, training, a clearer escalation path, or a non-AI tool produces a better outcome at lower cost than any AI solution. Forcing AI onto a problem that does not need it produces solutions in search of problems, and tends to fail loudly later.
The patterns are familiar: the benefits do not justify the run cost once total cost of ownership is honest; the data required does not exist, or cannot be obtained at acceptable cost or risk; the regulatory landscape is moving fast enough that committing now would be premature; or the capability the solution would build is not one the organisation actually wants.
A "no AI here" recommendation is credible when it does three things. It shows the AI options were genuinely examined. It names at least one non-AI alternative, with a rough estimate of its cost and benefit. And it states the conditions that would change the answer — "we revisit if data quality improves to X, if regulatory clarity emerges on Y, if volumes exceed Z." That conditional framing is what distinguishes a thought-through recommendation from a retreat.
Project Activity — Complete section 2.1: critically evaluate the solution
Open the Module 2 Project workbook and complete section 2.1 Critically evaluate the solution. Use the problem statement and root-cause analysis from Part 1 as your anchor.
- Score all four solution archetypes: off-the-shelf product, pre-built API, fine-tuned foundation model, and custom build.
- Score each one on the five criteria from this lesson — value location, time-to-value, capability built, lock-in, and data and regulatory exposure — with a short justification behind every score.
- Compare the cost shape of all four archetypes and one credible non-AI alternative, using the same scope, usage assumptions, and time horizon for each. Include vendor or tool cost, integration and adaptation, operation and oversight, and risk and governance. Use ranges where the evidence is limited. The workbook has scoring rows for the archetypes but no dedicated cost field, so record this as a short table alongside your section 2.1 response.
- Take the no AI for this problem option seriously. Name the conditions under which the right recommendation would be to pause, redirect to the non-AI alternative, or not proceed — and identify what that alternative would be, with a rough cost and benefit so the AI options have a real baseline to beat.
- Choose your recommended archetype and write the reason as a decision, not a preference. Link it back to the problem, the root cause, your earlier opportunity estimate, and the evidence you have. Do not turn this comparison into a detailed project budget; that is completed in Unit 3.
Project Checklist
- Section 2.1 evaluates all four archetypes, not only the option the team already prefers.
- Every score uses the five criteria from this lesson and carries a short justification tied to my own project context.
- I compared the cost shape of all four AI archetypes and a credible non-AI alternative on a consistent basis.
- I named a credible non-AI alternative, with a rough cost and benefit, so the AI options have a real baseline to beat.
- I have included lifecycle implications — integration, maintenance, monitoring, governance, and capability — without presenting them as an approved project budget.
- Lock-in and data and regulatory exposure are explicit for the recommended option.
- My recommended archetype follows from the problem and evidence in Part 1.
⏭️ Up next — Lesson 3: With the solution archetype chosen, Lesson 3 turns to the pitch. You will move from analytical work to communicating it — the executive one-pager, the pitch deck, and the four AI-specific failure modes that send otherwise strong projects back to the drawing board.