Prompting and structured output
The lowest rung of the leverage ladder — and the one that solves more problems than its reputation suggests.
Prompting has a reputation problem: it sounds like folklore, so engineers skip it and go straight to retrieval or fine-tuning. That is backwards and expensive. The prompt is the cheapest thing to change, takes effect instantly, and — treated as a specification rather than an incantation — solves more production problems than either of the heavier tools. This chapter is prompting as engineering.
The system prompt is a spec
Write it like one: the role and scope (“you are a support assistant for X; you do not discuss Y”), the constraints (length, tone, format), the refusal behaviour (what to do when the answer is not in the provided material — say so, never improvise), and the edge cases that burned you last week. Concrete beats abstract everywhere: “answer in 2–4 sentences citing the source id” outperforms “be concise and accurate”. And because it is a spec, it is code: versioned, reviewed, diffed, and gated by the evals chapter before it ships. Teams that treat prompts as chat messages get chat-message reliability.
Examples beat instructions
Few-shot prompting — two to five worked examples of input → ideal output — is the single highest-leverage technique in the book. Models imitate patterns better than they follow descriptions; three good examples routinely replace three paragraphs of rules, and one example of the hard case handled well (the ambiguous request, the missing data) outteaches any amount of prose about edge cases. Curate them like test fixtures, because that is what they are. Chain-of-thought — asking the model to reason before answering — helps on genuinely multi-step problems (arithmetic, logic, planning) and merely spends tokens elsewhere; and note the newest reasoning models do this internally, turning it into a knob you buy rather than prompt.
Structured output — where prompts meet parsers
Production callers rarely want prose; they want JSON their code can act on. The ladder, in order of preference: constrained decoding where your provider offers it (the sampler literally cannot emit tokens that violate your schema — the strongest guarantee available); function/tool-call formats, which are schema-shaped by construction; and schema-in-prompt with examples as the fallback. Then parse defensively anyway: validate against the schema, and on failure retry once with the validator’s error appended — “your output failed: missing field ‘reason’” fixes most failures in one round trip. Design the schema for the model too: descriptive field names, an unknown enum member so uncertainty has somewhere legal to go, no clever nesting.
Where it bites
- Prompt changes shipped without evals — every edit is a global behaviour change to an unversioned production system.
- Variable content (the user’s document) placed before the stable system prompt — prefix caching destroyed, money burned (previous chapter).
- Instructions and untrusted data concatenated with no boundary — the security chapter is already wincing.
- Parsing model output with a regex and hope — the schema-validate-retry loop is twenty lines and pays for itself the first afternoon.