How AI systems are built, one layer at a time

An LLM is a next-token predictor. Text is cut into tokens (word pieces; roughly 4 characters or 0.75 English words each). The model repeatedly produces a probability distribution over the next token, picks one, appends it and repeats until it emits a stop token. Your prompt plus everything it has generated so far must fit in the context window, the model's working memory for a single call.

Sampling controls shape the output. Temperature rescales the probabilities: near 0 the model almost always takes the most likely token (repeatable, good for extraction), higher values spread the choice (varied, good for brainstorming). Top-p restricts sampling to the smallest set of tokens whose probabilities add up to p. Max output tokens caps length and cost.

Even this bare setup is useful for research and drafting: summarising a paper, comparing options, outlining a document, rewriting for another audience. But the model only knows its training data, cannot look anything up unless a product wraps a search tool around it, and has a training cutoff.

A system prompt sets role, rules, tone and constraints for every call. Few-shot examples show the model two or three input/output pairs so it imitates the pattern. Delimiters (XML tags, headings) separate instructions from untrusted input. A prompt builder fills a template with the user's task so behaviour is consistent and version-controlled.

A JSON Schema is the output contract: field names, types, enums, required keys. With structured-output features, constrained decoding masks any token that would break the schema at each step, so the output is valid by construction. Without it, you parse, validate and retry. Either way a program, not a human, consumes the result.

The function-calling format is the same idea: you describe a function with a name, description and parameter schema, and the model returns a JSON object of arguments instead of prose. Your code runs the function. That is the bridge to tool use in stage 4.

Ingestion (offline). Load documents, parse them, split into chunks (typically a few hundred tokens, ideally cut along headings or sections, with slight overlap), turn each chunk into an embedding vector with an embedding model, and store vector, text and metadata (source, date, access rights) in a vector database next to a BM25 keyword index.

Query (online). Embed the question, run vector search (finds paraphrases) and BM25 keyword search (finds exact terms, part numbers, IDs) in parallel, merge the lists with reciprocal rank fusion, then let a reranker (a cross-encoder that reads question and chunk together) re-score the top candidates. Enforce access rights at query time: filter the search by the caller's permissions (group or ACL metadata) so restricted chunks are never retrieved, rather than hoping the model withholds them; permissions copied into the index at ingestion go stale when someone's access changes, so re-sync them or check at the source. The best few chunks go into the prompt with the instruction to answer only from them and to cite chunk IDs.

The result is a grounded answer with citations: a reader can click through to the source, and the model is steered away from inventing facts. Hybrid search matters because vectors alone miss exact identifiers and BM25 alone misses meaning.

The model never executes anything. You send it a list of tool definitions (name, description, JSON Schema for arguments). When it decides a tool is needed it returns a structured tool call instead of prose. Your executor validates the arguments, runs the function, and returns the result as a new message; the model then continues, and may call another tool or write the final answer. Retrieval from stage 3 can simply be one of the tools.

Typical tools: business APIs (look up an order, create a ticket), database queries, web search, and a code sandbox for calculations and data analysis, run in an isolated environment with no access to your network or secrets.

MCP (Model Context Protocol) is an open standard, using JSON-RPC messages, for how an AI application discovers and calls capabilities exposed by a server: tools, resources and prompt templates. Servers are reached over stdio (a local process) or Streamable HTTP (remote). You write the integration once and any MCP-capable client can reuse it, instead of rebuilding a connector per application.

The ReAct pattern alternates reasoning and acting: the model writes a short thought, chooses an action (a tool call), the controller runs it and appends the observation, and the cycle repeats. The loop controller is ordinary code. It assembles the prompt from the goal, history and memory, calls the model, executes the chosen action, updates state and decides whether to continue.

Short-term memory is what is in the context window: recent messages, a scratchpad of intermediate results, a running summary of earlier steps (older content is trimmed or summarised so it fits). Long-term memory lives outside the model in a vector store, key-value store or database; the agent reads it through retrieval and writes to it deliberately. Checkpointing state after each step lets you pause, resume and replay a run.

Stop conditions are mandatory: the model emits a final answer, a maximum step count is hit, a token or cost budget is exhausted, a timeout expires, a repeated-action loop is detected, or a human interrupts.

An orchestrator (supervisor) breaks the goal into subtasks and delegates them: a researcher agent that uses RAG, an analyst agent that uses the code sandbox, an operator agent that calls business APIs. A handoff transfers control, and the relevant context, from one agent to another. Smaller toolsets and shorter contexts make each worker more reliable and cheaper.

Agents coordinate through shared state (a blackboard): a structured store everyone reads from and writes to, instead of passing long transcripts that lose detail. The orchestrator merges results, resolves conflicts and decides when the goal is met.

Human-in-the-loop approval is a gate in the workflow: the system pauses before irreversible or high-impact actions (payments, deletions, external emails), when confidence is low, or when policy requires sign-off. Because state is checkpointed, the human can approve minutes or days later. Use multiple agents only when a single agent demonstrably hits a limit; each extra agent adds cost, latency and failure modes.

A golden dataset is a versioned set of representative inputs with expected answers or grading criteria, drawn from real logs, subject-matter experts and known edge cases. Graders include programmatic checks (valid schema, correct tool called), LLM-as-judge scoring against a rubric, and human review used to calibrate the judge.

RAG metrics separate retrieval from generation: faithfulness (are the answer's claims supported by the retrieved context), answer relevancy, context precision (is the relevant material retrieved and ranked high) and context recall (was everything needed retrieved). Agent evals score task success, tool-call correctness and the trajectory: how many steps, which tools, how much cost.

Offline, the suite runs in CI on every change to a prompt, model or retriever, and blocks the merge on regression. Online, you sample production traffic, run shadow or canary releases, A/B test variants and collect user feedback.

Input guardrails detect and mask PII, flag prompt-injection and jailbreak attempts, enforce topic scope and authenticate the caller. Output guardrails validate the schema, check toxicity and policy, check groundedness against the retrieved sources, redact secrets, and either refuse or regenerate.

Action guardrails govern tools: allow-list the tools each agent and user may use, issue least-privilege scoped credentials, default to read-only, validate arguments, require human approval for high-impact calls, restrict network egress, and apply rate limits and cost budgets. Every decision goes to an append-only audit log.

Use defence in depth, because no single filter is reliable. In particular, prompt injection cannot be solved by wording the system prompt carefully. Treat every retrieved document and tool result as untrusted data, and limit what the system can do even if the model is fooled.

Tracing. A request is a trace; each model call, retrieval, tool call, guardrail check and agent step is a span with inputs, outputs, latency, token counts, status and a parent-child link. OpenTelemetry-style spans make a five-agent run readable, and OpenTelemetry publishes semantic conventions for generative-AI calls.

Versioning. Prompts, model versions, retriever settings and tool schemas are versioned artifacts with release and rollback, and each trace records which versions served it. Cost is tokens in and out multiplied by price per model, attributed per feature, team and user, with budgets and alerts. Latency is tracked as time to first token and total time, p50 and p95, broken down per step.

The feedback loop closes the circle: thumbs-down, user edits and escalations are triaged, failing cases are added to the golden dataset (stage 7), the fix is evaluated, then deployed, and the new version is watched in production.

Read the diagram top to bottom as the request path, and the strips on the sides as the controls that apply across it. Identity and the gateway authenticate and rate-limit every caller. The orchestration layer runs agents, memory, workflows and approval gates. The LLM gateway hides model vendors behind one interface: it routes by task, cost, latency and data-residency, caches semantically similar requests, falls back when a provider fails and enforces token budgets.

The knowledge layer keeps enterprise data searchable with access rights intact (vector index, keyword index, knowledge graph for relationships). The tool layer exposes systems of record such as ERP, CRM, PLM and ticketing through MCP servers behind a policy gate. Guardrails, evals, observability and governance are cross-cutting rather than bolted on at the end.

The numbered badges mark six data paths, listed below the diagram. The point of the platform is separation of concerns: a use-case team writes prompts, tools and evals; the platform team owns the gateway, identity, guardrails, observability and cost; the governance body owns policy and risk.