Lesson 1 — Document What You Actually Built
Module 5, Unit 4 | Lesson 1 of 3
By the end of this lesson, you will be able to:
- Build an architecture audit module that produces a structured description of your agent's components (S27, K8)
- Add an execution tracer to your Commit Log Agent that captures a live run as evidence (K8, S27)
- Use both outputs to populate your Architecture Design Document from real code rather than from memory (S27)
Documentation is not the thing you do after the real work
It is part of the real work. The difference between a system that survives its original developer and one that does not is usually not capability — it is whether anyone wrote down how the decisions were made.
By the end of Unit 3 you had a working agent: it makes API calls, calls tools, tracks cost, handles errors. What you do not yet have is a record of why it is built the way it is. That record is your Architecture Design Document — and in this lesson you are going to build it from the inside out, starting with the code you have already written rather than from a blank page.
You will not be drawing data flow diagrams. You have mapped processes and flows extensively in earlier modules. What you need here is different: a written record of decisions, components, and the things you do not yet know.
What the ADD contains
An Architecture Design Document does not need to be long. It needs to be honest, specific, and justified. Six sections:
1. Problem statement — What business problem does this system solve? One paragraph, written for a non-technical reader.
2. System overview — What the system does, who uses it, and what it produces. Include your system diagram from Unit 1.
3. Component breakdown — Each major component described with: what it does, why you chose this implementation, and what happens if it fails.
4. Decisions and trade-offs — The choices you made and the alternatives you rejected. Each decision: what you chose, what you did not choose, and why.
5. Open questions and risks — What you do not yet know. What could go wrong. This section is not optional.
6. Data handling — What user or organisational data enters the system, where it goes, and what the compliance implications are.
You will populate these sections today. Not from scratch — from the audit output your code generates.
The diagram below shows how your two generated outputs map onto the six ADD sections — so you know where to look for each piece of content before you open ADD.md.
Before you start — package structure
All the modules in Unit 4 live inside a Python package called commit_log_agent. Before writing any code, create this structure in the root of your repository:
commit_log_agent/
├── __init__.py ← create this first (it can be empty)
├── audit.py ← you will build this in Lesson 1
├── tracer.py ← you will build this in Lesson 1
├── validator.py ← you will build this in Lesson 2
├── retry.py ← you will build this in Lesson 2
├── fallback.py ← you will build this in Lesson 2
└── recommendation_drafter.py ← you will build this in Lesson 3
To create the package, run in your terminal (from the repo root):
mkdir commit_log_agent
touch commit_log_agent/__init__.py
You only need to do this once. Every new module you create in this unit gets added to that folder.
Build Part 1 — The architecture audit module
Add a new file to your project: commit_log_agent/audit.py. This module describes your agent's components in a structured format and prints a report you can paste directly into your ADD.
# commit_log_agent/audit.py
from datetime import datetime
# ─── Component registry ────────────────────────────────────────────────────
# Each entry is one architectural component of your agent.
# Fill in the why_this_choice field for each one — that becomes
# Section 3 (Component Breakdown) and Section 4 (Decisions) of your ADD.
COMPONENTS = [
{
"name": "LLM Client",
"what_it_does": "Sends structured prompts to the model API and returns completions.",
"implementation": "OpenAI Python SDK. API key loaded from environment variable. Model specified at call time.",
"why_this_choice": "", # TODO: fill this in — it becomes your Section 4 evidence
"failure_mode": "AuthenticationError if key is missing or expired. APIConnectionError on network failure.",
"ksb_evidence": "K8 — third-party API dependency with cost, uptime and data-handling implications.",
},
{
"name": "Tool Gateway",
"what_it_does": "Exposes an allowlisted set of callable functions to the LLM.",
"implementation": "JSON tool schemas passed in the API call. Only whitelisted function names are dispatched.",
"why_this_choice": "",
"failure_mode": "LLM requests an unlisted function → tool_not_found returned to the model, not raised.",
"ksb_evidence": "K9 — function calling architecture. S27 — controlled tool exposure as a business control.",
},
{
"name": "Cost Guard",
"what_it_does": "Estimates cost before each API call and blocks calls that exceed the monthly budget.",
"implementation": "Reads token estimate from the prompt, calculates cost at current model pricing, checks against threshold.",
"why_this_choice": "",
"failure_mode": "BudgetExceededError raised — the call is not made. Error is logged and surfaced to caller.",
"ksb_evidence": "K8 — cost as an architectural constraint. S27 — aligning capability with business constraints.",
},
{
"name": "Usage Logger",
"what_it_does": "Records prompt tokens, completion tokens, latency, cost and model for every call.",
"implementation": "Appends a JSON line to usage.log after each call. File path configurable via environment variable.",
"why_this_choice": "",
"failure_mode": "If the log directory is missing, falls back to stdout. Does not fail the API call.",
"ksb_evidence": "S27 — operational observability as a system requirement.",
},
]
def run_audit():
print("\n" + "═" * 64)
print(" COMMIT LOG AGENT — ARCHITECTURE AUDIT")
print(f" {datetime.now().strftime('%Y-%m-%d %H:%M')}")
print("═" * 64 + "\n")
print("SECTION 3 — COMPONENT BREAKDOWN\n")
incomplete = []
for i, comp in enumerate(COMPONENTS, 1):
has_rationale = bool(comp["why_this_choice"].strip())
marker = "✓" if has_rationale else "□ add decision rationale"
if not has_rationale:
incomplete.append(comp["name"])
print(f"{i}. {comp['name']} [{marker}]")
print(f" What it does: {comp['what_it_does']}")
print(f" How: {comp['implementation']}")
if has_rationale:
print(f" Why this choice: {comp['why_this_choice']}")
print(f" If it fails: {comp['failure_mode']}")
print(f" KSB evidence: {comp['ksb_evidence']}")
print()
print("─" * 64)
if incomplete:
print("\nDECISIONS NEEDING RATIONALE (for ADD Section 4)\n")
for name in incomplete:
print(f" • Why did you choose this implementation for: {name}?")
print()
print("Next step: copy this output into ADD.md — Sections 3 and 4.")
print("Fill in the why_this_choice fields above to complete Section 4.\n")
print("─" * 64 + "\n")
if __name__ == "__main__":
run_audit()
Run this: python -m commit_log_agent.audit
Read every line of the output. The □ markers show exactly where your ADD has gaps. Fill in each why_this_choice field in the code — not in the document directly — so your documentation stays connected to the thing it describes. Then run it again and copy the output into ADD.md.
Curious Cat
In 2017, the Knight Capital Group lost $440 million in 45 minutes because a deployment activated old, undocumented trading code that nobody knew was still in the system. The code had been dormant for years. There was no record of what it did, why it existed, or that it was still present. The company filed for bankruptcy shortly afterwards. The failure was not a coding error. It was a documentation failure — a gap in the record of what the system actually contained and why.Build Part 2 — The execution tracer
A data flow diagram describes how your system should work. An execution trace captures how it actually worked, on a specific run, with real inputs. That distinction matters when you are producing evidence for an apprenticeship portfolio.
Add a second file: commit_log_agent/tracer.py. This module records every significant event in a single agent run and exports it as JSON.
# commit_log_agent/tracer.py
import json
import uuid
from dataclasses import dataclass, field
from datetime import datetime
from typing import Optional
@dataclass
class TraceEvent:
step: int
timestamp: str
event_type: str # "call_start" | "tool_request" | "tool_result" | "call_complete" | "error" | "cost_check"
detail: str
tokens_used: Optional[int] = None
cost_usd: Optional[float] = None
@dataclass
class ExecutionTrace:
session_id: str = field(default_factory=lambda: str(uuid.uuid4())[:8])
started_at: str = field(default_factory=lambda: datetime.now().isoformat())
events: list = field(default_factory=list)
_step: int = field(default=0, repr=False)
def record(
self,
event_type: str,
detail: str,
tokens_used: Optional[int] = None,
cost_usd: Optional[float] = None,
):
self._step += 1
self.events.append(
TraceEvent(
step=self._step,
timestamp=datetime.now().isoformat(),
event_type=event_type,
detail=detail,
tokens_used=tokens_used,
cost_usd=cost_usd,
)
)
def export(self, path: str = "trace.json"):
payload = {
"session_id": self.session_id,
"started_at": self.started_at,
"events": [vars(e) for e in self.events],
}
with open(path, "w") as f:
json.dump(payload, f, indent=2)
print(f"Trace exported → {path}")
def print_summary(self):
print(f"\n=== EXECUTION TRACE [{self.session_id}] ===")
for e in self.events:
cost_str = f" ${e.cost_usd:.5f}" if e.cost_usd is not None else ""
tok_str = f" {e.tokens_used} tokens" if e.tokens_used is not None else ""
print(f" {e.step:02d} [{e.event_type:<16}] {e.detail}{tok_str}{cost_str}")
print()
Now wire the tracer into your main agent loop. Wherever your agent makes an API call, requests a tool, receives a tool result, or hits an error, add a trace.record(...) call. Add from commit_log_agent.tracer import ExecutionTrace and trace = ExecutionTrace() at the top of whichever file contains your main agent loop — the same file where you call client.chat.completions.create(...) or client.models.generate_content(...). For example:
from commit_log_agent.tracer import ExecutionTrace
trace = ExecutionTrace()
# Before the API call:
trace.record("call_start", f"model={model}, prompt_tokens≈{estimated_tokens}")
# After the response:
trace.record(
"call_complete",
f"finish_reason={response.choices[0].finish_reason}",
tokens_used=response.usage.total_tokens,
cost_usd=calculated_cost,
)
# If the model requests a tool:
trace.record("tool_request", f"tool={tool_call.function.name}, args={tool_call.function.arguments}")
# After running the tool:
trace.record("tool_result", f"result_length={len(str(result))} chars")
# At the end of the run:
trace.print_summary()
trace.export("traces/run_latest.json")
Run your agent with a realistic input. Export the trace. Open trace.json and read it. That JSON file is your data flow evidence — richer than any diagram because it is tied to a real execution.
Add a traces/ folder to your repository and commit at least one exported trace. In your ADD's Section 2 (System Overview), write two sentences describing the execution sequence using your trace as the source of truth, not your memory of how you intended it to work.
Coach Cora
Section 5 of your ADD — open questions and risks — is where experienced developers demonstrate judgement. Anyone can describe the system as it is. It takes discipline to write honestly: "I do not yet know how this will perform under concurrent requests" or "I have not verified that our data agreement permits this use case." Your coach would rather read a candid open question than a confident claim you cannot yet substantiate. If your Section 5 is empty, you have not looked hard enough.
Build — Pull it together into ADD.md
Open or create ADD.md in the root of your repository. Populate it section by section using the outputs you have just generated:
Section 1 — Problem Statement. One paragraph for a non-technical reader. What business problem does your system solve? What would happen if it did not exist?
Section 2 — System Overview. Two to three sentences describing what the agent does, who uses it, and what it produces. Reference your execution trace for the sequence — not your intuition.
Section 3 — Component Breakdown. Copy the audit output. Fill in every why_this_choice field. No placeholder text.
Section 4 — Decisions and Trade-offs. For each component where you had an alternative, write: what you chose, what you did not choose, and why. If you never considered an alternative, that is itself worth noting — and worth asking yourself why.
Section 5 — Open Questions and Risks. At least two things you genuinely do not yet know. Be specific. Name the risk, not a category.
Section 6 — Data Handling. What data enters your system? Where does it go? Does any user or organisational data leave your environment? What are the compliance implications?
Commit ADD.md and your traces/ folder to GitHub.
Add to your Commit Log: a link to your committed ADD, which section was hardest to complete honestly, and what the audit module revealed that you had not previously thought to document.
Checklist
-
audit.pyruns without errors and allwhy_this_choicefields are filled in -
tracer.pyis wired into my agent and at least one exported trace is committed totraces/ -
ADD.mdhas all six sections completed — no placeholders - ADD.md and traces are committed to GitHub and linked in my Commit Log
KSB evidence focus
-
S27 — Apply technical understanding to help align business needs with technical capabilities, supporting the development of solutions that are scalable, efficient, and aligned with the organisation's strategic objectives. Your completed ADD — built from audit output and execution trace, not from wishful thinking — is the primary S27 evidence document for Module 5. The quality of your decision rationale is the evidence.
-
K8 — The capabilities, risks and implications of on-premise, cloud-based and third party solutions. Your data handling section makes K8 considerations explicit: where does your system's data go, what are the third-party implications, and what risks have you identified?
Up next: Lesson 2 takes your working agent and makes it robust. You will add input validation, retry logic with exponential backoff, and a fallback provider — and each defensive pattern you write will map directly to a row in your risk register.