AiCore logo

Under the Hood: The Architecture That Drives Every Decision

Module 5, Unit 1 | Lesson 1 of 3

Welcome to Phase 2.

You have mapped processes, challenged assumptions, built financial cases, managed stakeholders and planned a project that is going to make a real difference. That took discipline, critical thinking and no small amount of persistence. Be proud of it.

Now the fun part starts.

From here, you are not a practitioner who understands AI. You are a developer who builds with it. The system you planned exists on paper. By the end of this phase, it will exist in code.

By the end of this lesson, you will be able to:

  • Explain how the transformer architecture produces outputs and why that creates hard architectural constraints (K9)
  • Distinguish transformer, diffusion and encoder-only models and identify when each is relevant to a real build decision (K9)
  • Estimate token volume at scale for your own project and identify where the architecture starts to strain (K8, S27)
  • Create a Commit Log Agent v0 to support evidence capture, reflection and GitHub-ready documentation (B6, S27)
  • Produce an AI Architecture Constraints Note that will anchor your design decisions across the rest of this module (S27, B6)

Start your Commit Log Agent

Before you run any code, you are going to design a small assistant called your Commit Log Agent.

This is not another workbook. It is a practical AI assistant that will help you record technical evidence for the rest of the programme. From Module 5 through to the end of Module 9, you will use and improve the same agent so it can help you draft short evidence entries, suggest GitHub commit messages, prepare README updates and reflect on how each skill could apply in your workplace.

Today, you will create version 0 of the agent. Version 0 is a prompt-based agent: it asks you questions and drafts a Commit Log entry for you to review. It must not publish, upload, commit or share anything automatically. You remain responsible for checking the output before it goes into GitHub, your portfolio or any apprenticeship evidence.

From this point on, your evidence sits in two places:

  • Your Module 3 workbook: this continues to track the full project lifecycle for your main workplace project as you move from planning into implementation.
  • Your Commit Log Agent entries: these record evidence of the technical tasks and activities you complete during Modules 5 to 9. They can also become part of your professional portfolio, especially when they link to GitHub commits, screenshots, Jupyter notebooks or working prototypes.

Create this file in one of these places:

  • Recommended: in your AI Projects repository on GitHub, as a Markdown file called commit-log-agent.md. If you are creating a new repository, use a clear name such as ai-projects-[your-name].
  • If your GitHub repository is not ready yet: create it in OneDrive, Google Docs or OneNote as Commit_Log_Agent_[Your_Name], then move or link it to your GitHub repository later.
  • If your evidence includes workplace-sensitive information: keep the detailed version private and only publish a cleaned version that your organisation is happy for you to share.

This file is where you keep the agent brief and any entries you approve. Later in the programme, you can turn this into a more automated agent that prepares GitHub updates. For now, copy this starter brief into the file:

# Commit Log Agent

Name:
Apprenticeship route: Level 4 AI and Automation Practitioner - AI Developer
Main workplace project:
GitHub profile or repository link:

## Agent purpose

Help me record technical evidence from Modules 5 to 9. The agent should help me turn practical work into concise Commit Log entries, GitHub-ready notes and professional portfolio evidence.

## Agent rules

- Ask before drafting if any key detail is missing.
- Keep entries concise and evidence-focused.
- Include workplace value, possible use cases and limitations.
- Remind me to remove API keys, personal data, confidential workplace information and anything I do not have permission to share.
- Suggest GitHub commit messages and README updates, but do not claim anything has been committed or published unless I confirm it.
- Do not invent screenshots, links, code or results.

## Questions the agent should ask me

1. What did you build, test, configure or decide?
2. Which module and lesson does this relate to?
3. What evidence can you link or screenshot?
4. What did you learn from the task?
5. How could this skill apply to your workplace or main project?
6. What risks, limitations or follow-up questions should you discuss with your instructor or line manager?

## Output format

### Commit Log entry

- Date:
- Module / lesson:
- What I did:
- Evidence:
- Why it matters:
- Workplace application:
- Risks or limitations:
- Follow-up question:

### GitHub-ready update

- Suggested commit message:
- Suggested README note:
- Files or evidence to check before publishing:

Keep the agent useful and honest. For every task, it should help you answer: What did I do, what evidence proves it, and how could this skill apply to my workplace? Bring strong use cases, risks or questions to your instructor and line manager. That discussion is part of becoming an AI Developer, not an optional extra.

You will use the agent for the first time at the end of this lesson. It should draft your evidence entry, but you decide what is accurate, appropriate and safe to keep.

First Build: Run a Model Call in a Jupyter Notebook

You are going to start with a small working example. The aim is not to build a full application yet; it is to make a real model call, inspect the response and connect what you see to the architecture concepts in the rest of the lesson.

For this build, you will need:

  • VS Code with the Python and Jupyter extensions installed.
  • Python available on your machine, or access to Google Colab in your browser.
  • A Gemini API key or an organisation-approved API route.
  • Your Commit Log Agent file open nearby, so you can record what you did once the build is complete.

The Jupyter notebook will show you the model response and the technical information that sits underneath it. You will see the text the model generated, the number of tokens sent into the model, the number of tokens generated back, and the time it took for the call to complete.

Those details may look small at first. They are not. Token counts tell you how much context your system is asking the model to handle. Latency tells you how long a user or workflow has to wait. Together, they begin to show whether an AI feature is practical, scalable and suitable for a real workplace process.

Use a Jupyter notebook inside VS Code for this lesson. That means you are not running this in PowerShell, Command Prompt or a normal .py file. You are running it inside Jupyter notebook cells, where each cell can show its output directly underneath the code.

  1. Open VS Code.
  2. Create a new file called l1_1_under_the_hood.ipynb.
  3. If VS Code asks you to install the Python or Jupyter extension, install it.
  4. If VS Code asks you to select a kernel, choose the Python option. The kernel is the Python environment that runs the Jupyter notebook.
  5. Create your first code cell.
  6. Run the install command below in that first cell.
%pip install -q google-genai

Create a requirements.txt file in your project root and list your dependencies there — this makes your environment reproducible:

google-genai
python-dotenv
jupyter

Install from it with pip install -r requirements.txt.

If your laptop is not ready for Jupyter notebooks yet, use Google Colab as a temporary option. Colab behaves like a Jupyter notebook in the browser, so the learning experience is very similar.

  1. Go to https://colab.research.google.com.
  2. Create a new Colab notebook.
  3. Paste the same install cell and starter code into the notebook cells.
  4. Run each cell using the play button on the left of the cell.

Do not lose time fighting your setup in the first lesson. If the Jupyter notebook will not run after a few attempts at sensible troubleshooting, use the sample output below and come back to the setup with help.

Step 1 - Get API access

Use one of these routes:

  • Organisation-approved access: If your employer has provided an approved AI API route, use that.

Gemini API free tier: if you do not have organisation-approved API access

  1. Open https://aistudio.google.com/app/apikey.
  2. Sign in with a Google account.
  3. If Google asks you to accept terms of service, read them and continue only if you are comfortable doing so.
  4. Select Create API key.
  5. If Google asks you to choose a project, use the default project it offers. If you are not sure which project to use, stop and ask your coach rather than choosing an employer project without approval.
  6. Copy the API key when it appears. Treat it like a password.
  7. Return to your Jupyter notebook.
  8. Run the starter code. When the notebook asks Paste your Gemini API key. It will not be shown:, paste the key and press Enter.

Do not enable billing. If a page asks you to add billing details, stop and ask your coach. Do not paste the key into your Commit Log, GitHub, Teams, screenshots or any shared document.

Before your first git commit: create a .gitignore file in your project root and add .env to it. If your API key ends up in a public repository — even for a moment — revoke it immediately in your API dashboard and generate a new one.

# .gitignore
.env
__pycache__/
*.pyc
.ipynb_checkpoints/

If you cannot use an API key today: Do not stop the lesson. Read the code, use the sample output below and complete the analysis. In your Commit Log, write that you used sample output because API access was not available, then rerun the Jupyter notebook later when your API access is available.

In the Jupyter notebook below, the key is entered using getpass, which hides it while you type.

import getpass
import os
import time

from google import genai

if not os.environ.get("GEMINI_API_KEY"):
    os.environ["GEMINI_API_KEY"] = getpass.getpass(
        "Paste your Gemini API key. It will not be shown: "
    )

client = genai.Client()

prompt = """
Explain what a transformer model is in two sentences.
Then add one sentence explaining why token counts and latency matter
when designing an AI system for a workplace process.
"""

start_time = time.perf_counter()

response = client.models.generate_content(
    model="gemini-2.0-flash",
    contents=prompt,
)

latency_seconds = time.perf_counter() - start_time

usage = response.usage_metadata

print("=== RESPONSE ===")
print(response.text)

print("\n=== THE NUMBERS THAT DRIVE ARCHITECTURE ===")
print(f"input_tokens:  {usage.prompt_token_count}")
print(f"output_tokens: {usage.candidates_token_count}")
print(f"total_tokens:  {usage.total_token_count}")
print(f"latency:       {latency_seconds:.2f}s")

If you can run the Jupyter notebook, run it and use your own output. If you cannot run the API call today, use this sample output so you can continue the lesson:

=== RESPONSE ===
A transformer model is a neural network architecture that processes tokens by using attention to identify relationships between them. This allows the model to use context from different parts of the input when generating a response.
Token counts and latency matter because they affect how much context the system can handle, how quickly users receive a response and whether the design will still work at larger scale.

=== THE NUMBERS THAT DRIVE ARCHITECTURE ===
input_tokens: 34
output_tokens: 61
total_tokens: 95

latency:       1.42s

Read the output. Now read the rest of this lesson - because every concept below explains something you just saw in those numbers.

🔑 Key term — Token: The basic unit an LLM processes. Not quite a word — roughly a word fragment. "Transformer" becomes two tokens: "transform" and "er". Every API call is measured in tokens, every context window is measured in tokens, and every latency profile is shaped by tokens. This number is not a detail. It is the unit of architecture.

The architecture behind the numbers you just saw

In 2017, a team at Google published a paper called Attention Is All You Need. It introduced the transformer — and it made everything you just ran possible.

Here is what made it radical. Before transformers, language models read text the way a person reads a page with a ruler covering everything below the current line — one word at a time, left to right, forgetting earlier context as they went. They could not run in parallel. They could not hold long-range relationships in memory. They could not scale.

The transformer threw that out entirely. Instead of reading sequentially, it looks at every token in the input at the same time — and for each one, it asks: which other tokens in this input actually matter for understanding this one? That process is called attention, and it is what produces the behaviour you saw in your output.

When your prompt said "transformer model," the attention mechanism connected "transformer" to every other token in the context — finding the relationships that let it generate a coherent, contextually accurate response. It does this not once but across dozens of layers, each one refining the relationships the previous one found.

Three things fall out of this that you need to care about as a developer:

Parallelism — because transformers process all tokens simultaneously rather than one at a time, they can run efficiently on GPUs. This is why frontier models exist at scale. It is also why inference is fast enough to be useful in real products.

Context windows are a hard limit, not a setting — attention has to compare every token against every other token. That operation grows quadratically: double the input, quadruple the compute. This is physics, not a product decision. When you saw the token count in your output, you were looking at your proximity to that limit.

Emergent capability — and its trade-offs — transformers trained at scale develop capabilities nobody programmed explicitly: reasoning, code generation, translation, summarisation. They also hallucinate, because they are doing sophisticated pattern completion across training data — not retrieving verified facts. The same mechanism that makes them powerful makes them unreliable without the right architecture around them. That architecture is what this module teaches you to build.

🔑 Key term — Attention mechanism: The core operation of a transformer — for each token in the input, computing a weighted relationship with every other token. It is what allows the model to understand context rather than just sequence, and it is why larger context windows require significantly more compute to run.

The architecture landscape — what else exists and when it matters

Transformers are dominant for language tasks. But they are not the only architecture you will encounter.

Encoder-only models (BERT-style) — these read text in both directions and produce rich representations of meaning, but they do not generate new text. They are the right choice when you need to classify, score, or embed text rather than generate it. If your architecture includes a step that categorises documents or retrieves semantically similar content, you are probably using an encoder model — even if the rest of your system uses a generative LLM.

Diffusion models — the architecture behind image generation (Stable Diffusion, DALL-E, Midjourney). They work by learning to reverse a process of adding noise to data. Relevant if your system handles image generation, visual outputs, or multimodal content. You will encounter this in Lesson 1 of Unit 2.

Mixture of Experts (MoE) — an architectural variant where a large model is composed of multiple specialist sub-models, with a routing mechanism that activates only the relevant ones for a given query. You do not need to implement this in Module 5, but knowing it exists helps explain why model capability, latency and resource use do not always scale in a simple straight line.

Coach Cora

Coach Cora

You do not need to implement a transformer. You need to understand it well enough to make decisions around it — context size, model selection, resource modelling, when to use generation vs retrieval. That is the developer's relationship with architecture: not building it from scratch, but knowing it well enough to build on top of it responsibly.

Token volume as an architectural constraint

Go back to the numbers your Jupyter notebook printed. Those token counts are not just technical trivia — they are the input to every scalability decision you will make.

In this programme, you are not being asked to enable billing for model calls. The point is to learn how token volume affects latency, rate limits, infrastructure planning and future operating decisions if an organisation approves a system for deployment.

Here is how to think about it:

# Token volume model - adjust for your project
prompt_tokens_per_call = 500      # system prompt + user input
completion_tokens_per_call = 200  # typical response

tokens_per_call = prompt_tokens_per_call + completion_tokens_per_call

for daily_calls in [100, 1_000, 10_000]:
    monthly_tokens = tokens_per_call * daily_calls * 30
    print(f"{daily_calls:>6} calls/day -> {monthly_tokens:>10,} tokens/month")

Run this with your own estimates. Then ask: at what volume does this architecture need a different design? What would you change first — the model, the prompt length, the retrieval strategy, the output constraints or the response pattern?

That question — not the code — is the architectural skill this lesson is building.

What breaks architecture at scale:

  • A system prompt that is 3,000 tokens gets processed on every single call. At 10,000 calls/day that is 30M input tokens per month before a user has typed a word.
  • A context window that accumulates conversation history without a pruning strategy will hit the limit and fail silently or create unnecessary resource pressure.
  • A frontier model chosen for a high-volume, low-complexity task will usually be slower and more resource-intensive than a smaller model that handles the task equally well.

Model types as architectural decisions

The code you ran used gemini-2.0-flash. That was not a neutral choice — it was an architectural decision with latency, capability and access implications. Here is the decision space:

Standard instruct-tuned models (Gemini Flash, GPT mini models, Claude Haiku) — fast, efficient, capable at many tasks. The right default for high-volume production workloads. If your use case works reliably on one of these, there is no justification for starting with a heavier model.

Frontier models (Gemini Pro, GPT frontier models, Claude Sonnet/Opus) — significantly more capable on complex reasoning, nuanced instruction following, and difficult edge cases. They usually require more careful governance and stronger justification. Use them when quality at edge cases matters more than speed and simplicity — not as a default.

Reasoning models — allocate additional compute to intermediate reasoning steps before producing a final answer. Stronger on multi-step logic, mathematical problems, and careful comparison tasks. Usually slower and more token-intensive. Almost never the right choice for a real-time, customer-facing, high-volume system. A better fit for a background batch process that needs to get a complex decision right.

Base models (uninstruct-tuned) — the raw pre-trained model before post-training. Rarely used directly in production. Relevant if you are fine-tuning for a specialist task where the instruct-tuned behaviour gets in the way.

🔑 Key term — Post-training: The stage after pre-training where a base model is shaped for real-world use — through instruction tuning and reinforcement learning from human feedback (RLHF). This is what makes a model follow instructions, refuse unsafe requests, and behave consistently. The difference between a base model and an instruct-tuned one is not capability — it is controllability.

For an up-to-date comparison of model quality against cost, see Artificial Analysis — their intelligence vs cost chart is a practical starting point for model selection decisions.

Activity — AI Architecture Constraints Note

This is the first technical note for your Module 5 build. It is not a new workflow diagram. You already created workflow diagrams in Modules 1 and 3 to show the current process, existing infrastructure, tools, legacy systems and where the proposed solution could sit.

Use those existing diagrams as your starting point. First, identify the AI component you plan to add, improve or connect into that workflow. Then look at the technical constraints that could affect how that AI component is designed.

Once you have those details, use your Commit Log Agent to help you draft a one-page AI Architecture Constraints Note covering:

How to use your Commit Log Agent for this activity

Open your commit-log-agent.md file, or wherever you created your Commit Log Agent brief. Under the agent brief, add a new heading called:

## L1.1 - AI Architecture Constraints Note

Then paste this prompt underneath it and replace the bracketed prompts with your own information:

Use the Commit Log Agent rules above. Help me draft a one-page AI Architecture Constraints Note for L1.1.

Context:
- My existing workflow diagram is from: [Module 1 / Module 3 / both]
- The current process or workflow is: [brief description]
- The AI component I plan to add or improve is: [brief description]
- The systems, tools or legacy processes it may connect to are: [brief description]
- The longest input the AI component may need to handle is: [example, such as an email, document, form, call transcript or database record]
- My rough token estimate is: [token count or "not estimated yet"]
- My expected usage scale is: [pilot / team or department / wider organisation]
- The model type I think may fit is: [standard instruct model / reasoning model / embedding model / not sure yet]
- The main uncertainty I need to test is: [brief description]

Please draft:
1. A short AI Architecture Constraints Note.
2. A concise Commit Log entry for this task.
3. A suggested GitHub commit message.
4. Two questions I should discuss with my instructor or line manager.

Before drafting, ask me up to three questions if any important detail is missing. Do not invent facts, links, screenshots, token counts or workplace details.

Your agent should help you think, but it should not replace your judgement. Treat its output as a draft. You decide what is accurate, useful and safe to keep.

1. Input analysis What is the longest input your system will need to handle in a single call? A customer email? A 10-page report? A database record? Estimate its length in words, then use tiktokenizer.vercel.app (free, no account needed) to get a token count. How does this compare to the context window of your target model?

2. Scale estimate Using the token volume script above, estimate your monthly token volume at three scales:

  • Pilot (small internal test)
  • Moderate (team or department level)
  • Full scale (organisation-wide or customer-facing)

At which point would latency, rate limits, data access, support needs or future operating budget become a conversation you need to have with a stakeholder?

3. Model selection decision Based on your use case, state which model type you are starting with and why. Address: does it need real-time response or can it run async? Does it need a reasoning model or is a standard instruct-tuned model sufficient? Does the task volume justify a smaller, faster model?

4. Open questions What do you not yet know that would change these decisions? Name at least two.

Review before you keep or publish it

Before you add the note to GitHub, your portfolio or apprenticeship evidence, check:

  • It uses your existing Module 1 or Module 3 workflow diagram as context instead of redrawing the process.
  • It focuses on the AI component of the solution, not the whole business process.
  • It separates facts from assumptions.
  • It includes input length, token volume, latency or scale considerations.
  • It explains why the constraint matters for users, the business or technical delivery.
  • It does not include API keys, personal data, customer data, confidential employer information or anything you do not have permission to share.
  • It includes at least one question to discuss with your instructor or line manager.

If the note is safe to share, commit a cleaned version to your GitHub repository. If it is not safe to share, keep it private and record why in your Commit Log Agent entry. Then ask your Commit Log Agent to draft an evidence entry with a link, if you have one, and a one-paragraph summary of the most important constraint you identified. Review the draft before you keep it.

Checklist

  • I have created my Commit Log Agent and understand how it sits alongside my Module 3 workbook
  • I have run the starter code and can read the token usage and latency from the response object
  • I can explain in plain English why context windows are a hard limit, not a preference
  • I can describe the difference between encoder-only, diffusion and transformer architectures and when each is relevant
  • I have used my Commit Log Agent to draft an AI Architecture Constraints Note for my project
  • I have reviewed the note for accuracy, safety and workplace relevance before keeping or publishing it
  • I have committed a cleaned version to GitHub, or recorded why it cannot be shared publicly yet
  • My Commit Log Agent has drafted an evidence entry and I have reviewed it before keeping it

Your existing workflow diagram shows a customer support process. The AI component you want to add will summarise incoming messages before a human reviews them. Which note shows the right kind of architectural thinking for L1.1?

KSB evidence focus

  • K9 — AI and automation concepts, models and limitations. You are now working with the actual architecture — not a description of it. The transformer attention mechanism, context window constraints, and model type trade-offs you encountered here are K9 at developer depth.

  • K8 — The capabilities, risks and implications of on-premise, cloud-based and third party solutions. Your scale estimate makes K8 concrete: data leaving your environment, token volume at scale, and the rate limit exposure of a third-party API are no longer abstract risks — they are numbers in your AI Architecture Constraints Note.

  • S27 — Apply technical understanding to help align business needs with technical capabilities, supporting the development of solutions that are scalable, efficient, and aligned with the organisation's strategic objectives. Your AI Architecture Constraints Note is the first direct application of S27 in this module. Every constraint you identified connects a technical reality to a business decision.

  • B6 — Shows curiosity and initiative, experimenting with AI and automation, while ensuring such exploration is conducted safely, ethically, and with regard for potential impacts. You used code, measurement and reflection to investigate how model behaviour affects architecture. That is the habit this standard is asking for — and your GitHub commit is the evidence.

Up next: Lesson 2 turns your model call into a small controlled agent loop, showing how instructions, tools, state and human review work together inside an agent system.

Next: L1.2 - From a Model Call to an Agent System