Lesson 3 — Cost-Aware and Real-Time Architecture

Module 5, Unit 3 | Lesson 3 of 3

By the end of this lesson, you will be able to:

• Calculate the estimated cost of an API-based AI feature at realistic usage volumes (K8, S27)
• Identify and apply at least three techniques for reducing token consumption without degrading quality (K8)
• Describe the architectural trade-offs between synchronous and asynchronous processing for AI workloads (S27)

Why cost is an architecture decision

Many AI projects that work beautifully at prototype scale fail in production — not because the model does not perform well, but because nobody modelled the costs properly.

A call that costs £0.002 feels trivial. Ten thousand calls a day is £20/day, or £600/month. At 100 documents per user with 1,000 users, the economics change entirely. The time to do this calculation is before you commit to an architecture — not after your first invoice.

Why this matters beyond your main project:

Every AI feature you build in your career will eventually face a cost/value conversation with a business stakeholder. The developer who can walk into that conversation with a cost model — actual numbers, not hand-waving — is the developer who gets the feature approved and the infrastructure budget to run it.

Building a cost model

Token pricing is published by all major providers. A basic cost model has three inputs:

Output tokens cost more than input tokens because generating text is computationally more intensive than processing it. Always check current pricing before using these numbers in a real cost model — prices change: platform.openai.com/docs/pricing

You can also monitor your cumulative token usage and spending directly in your OpenAI account at platform.openai.com/usage — useful for catching unexpected spikes before they become costly.

estimated_calls_per_day = 500
avg_prompt_tokens = 800        # system prompt + user input + context
avg_completion_tokens = 400    # typical response length

# GPT-4o-mini pricing (example — always check current pricing)
input_price_per_1k = 0.00015   # USD per 1K tokens
output_price_per_1k = 0.0006   # USD per 1K tokens

cost_per_call = (
    (avg_prompt_tokens / 1000 * input_price_per_1k) +
    (avg_completion_tokens / 1000 * output_price_per_1k)
)

daily_cost = cost_per_call * estimated_calls_per_day
monthly_cost = daily_cost * 30

print(f"Cost per call: ${cost_per_call:.5f}")
print(f"Daily cost: ${daily_cost:.2f}")
print(f"Monthly cost: ${monthly_cost:.2f}")

Run this with your own numbers. Then stress-test it: what if usage is 5x your estimate? At what volume does your current architecture become economically unviable?

🔑 Key term — Prompt caching: A provider feature that stores frequently repeated prompt content (system prompts, large context documents) between calls, avoiding re-processing the same tokens. Supported by Anthropic and OpenAI — typically reduces costs significantly on high-volume workloads where the system prompt is large and static.

Token reduction techniques

These are not premature optimisations — they are architectural habits that should be built in from the start:

1. System prompt discipline — Every token in your system prompt is charged on every call. A 2,000-token system prompt at 10,000 calls/month is 20M tokens before the user writes a word. Review your system prompt regularly and remove anything that is not earning its keep.

2. Prompt caching — If you have a large, static system prompt or a large context document that does not change between calls, use prompt caching. It can reduce input costs by 50–90% on those tokens.

3. Output length control — Set max_tokens appropriately. If you need a classification label, you do not need 1,000 tokens of response. If you need a structured JSON object, tell the model its output should be concise and structured.

4. Model tiering — Not every task needs your best model. A classification or extraction task that works reliably on GPT-4o-mini does not need GPT-4o. Use the most capable model that reliably solves the task — not the most capable model available.

5. Caching results — For queries where the same input will produce the same output (e.g. product descriptions, FAQ answers), cache the result rather than calling the API again.

Synchronous vs asynchronous processing

The architecture of how your system calls the API depends on the user experience you need to deliver:

Synchronous (real-time) — the user sends a request and waits for the response before continuing. Required for interactive chat, live question answering, and anything where the response must be immediate. The cost: latency matters, timeouts matter, and your UI must handle slow responses gracefully.

Asynchronous (batch or queue) — requests are submitted to a queue and processed in the background. Results are returned later (via webhook, polling, or notification). Better for document processing, large-batch analysis, and non-interactive workflows. The benefit: cheaper (batch APIs are discounted), more resilient, and easier to scale.

Most production systems use both: real-time for interactive features, async for background processing.

Coach Cora

Rate limits are not a minor technical inconvenience — they are an architectural constraint that determines what your system can and cannot do at scale. Build rate limit handling into your code from day one: exponential backoff, retry logic, and a clear plan for what happens when a call fails.

Activity — Cost model and architecture decision

1. Run the cost model code above with realistic numbers for your workplace project
2. Apply at least two token reduction techniques to your code from Lessons 1 and 2 and estimate the savings
3. Write a short architecture decision note (one page maximum): for your project, should the AI calls be synchronous or asynchronous? State your reasoning with reference to the trade-offs above.
4. Commit your updated code and architecture decision note to GitHub

Add to your Commit Log: your monthly cost estimate, the techniques you applied, and your sync/async decision with a one-paragraph justification.

Checklist

• I have a working cost model with realistic numbers for my project
• I have applied at least two token reduction techniques and estimated their impact
• I have made and documented a synchronous vs asynchronous architecture decision for my project
• All code and notes are committed to GitHub and linked in my Commit Log

Quick Check

KSB evidence focus

• K8 — The capabilities, risks and implications of on-premise, cloud-based and third party solutions. Costs, rate limits, and data residency are K8 considerations made concrete. Your cost model and architecture decision document are direct K8 evidence.

• S27 — Apply technical understanding to help align business needs with technical capabilities, supporting the development of solutions that are scalable, efficient, and aligned with the organisation's strategic objectives. The sync/async decision is a S27 decision: it connects user experience requirements to technical architecture, with direct implications for cost and scalability.

Up next: Unit 4 brings everything together — documenting your architecture formally, making and justifying trade-offs, and producing the technical recommendation that caps Module 5.