System Prompt Architecture: Role, Rules, Tools, and Escape Hatches

Why the chat-prompt habit fails here

Prompting a chat model and prompting an agent are different jobs, and most system prompts fail because they are written for the first while running the second. In chat, the prompt shapes one response and a human reads it. In an agent, the prompt has to hold for twelve turns, across tool results you did not anticipate, while the model decides which of thirty functions to call and when to stop.

That changes what belongs in it. "You are a helpful assistant" costs tokens on every turn and buys nothing. What earns its place is anything that constrains a decision the model will actually face: which tool when, what to do when a tool fails, when to stop, and when to ask rather than guess.

The structure below is four blocks in a fixed order. The order matters because models weight early instructions more heavily, and because the last block is the one that stops runaway behaviour.

Block one: role, scoped to a job

The role block establishes what this agent is for and — more usefully — what it is not for. Scope is the part people skip, and it is the part that prevents an agent designed for order lookups from confidently freelancing on refund policy.

Keep it to two or three sentences. A long persona is a common failure: it eats context on every turn and models follow concrete constraints far more reliably than they follow character descriptions.

You are the order support agent for an e-commerce platform. You answer
questions about order status, shipping, and returns using the tools provided.

You do not process refunds, change prices, or make exceptions to policy. If a
customer needs any of those, say so plainly and hand off to a human.

Block two: rules that resolve real conflicts

A rule earns its place only if you can name the situation where the model would otherwise get it wrong. "Be accurate" is not a rule — no model is deciding between accurate and inaccurate. "Never state a delivery date the carrier has not confirmed" is a rule, because the model absolutely will infer one from an order date if you let it.

The most valuable rules are the ones that fire when evidence is missing or contradictory. Those are precisely the moments where an unguided model produces a confident, plausible, wrong answer.

Order rules by how costly the mistake is. Anything that touches money, identity or an irreversible action goes first.

Rules:
1. Only state facts returned by a tool this turn. Never infer a delivery date.
2. If a tool returns no results, say you could not find it. Do not guess an
   explanation for why.
3. If two tools disagree, surface the disagreement rather than picking one.
4. Never reveal internal IDs, SQL, or tool names to the customer.
5. One clarifying question maximum. If still ambiguous, hand off.

Block three: tool guidance the schema cannot carry

Tool descriptions handle per-tool selection — that is their job, and the earlier post in this series on tool schemas covers how to write them. What the system prompt adds is cross-tool sequencing: the ordering constraints and preconditions that no single schema can express.

Keep this block short. If it is growing past a handful of lines, that is a signal the tools themselves are badly drawn — you are papering over an interface problem with prose, and prose is the weaker fix.

Tool use:
- Always resolve the customer with find_customer before any order lookup.
  Order tools take a customer_id, never an email.
- get_order_status is cheap; call it freely. run_carrier_trace costs a credit
  and takes ~10s; call it only when the customer explicitly asks where a
  parcel is right now.
- Never call more than one write tool per turn.

Block four: the escape hatch

This is the block almost nobody writes, and it is the one that separates an agent that degrades gracefully from one that spirals. It answers a single question: what should you do when you cannot do the thing?

Without it, a stuck model does what models do — it keeps trying. It rephrases the failing query, calls a tool it already knows does not help, invents a partial answer to fill the silence. Every one of those is worse than saying "I could not do this, here is why".

Name the specific dead ends and the specific exit. Vague permission to give up does not work; models need to know what giving up looks like.

When you cannot proceed:
- Tool failed twice with the same error: stop. Tell the customer what you were
  trying to do and that it is not working. Do not try a third variation.
- Request is outside your scope (refunds, pricing, account deletion): say so in
  one sentence and call handoff_to_human with a summary.
- You are unsure whether an action is allowed: do not do it. Ask.

Ending a turn with "I was not able to X because Y" is a correct outcome. A
plausible guess is not.

What does not belong in there

Anything the schema already enforces. Repeating parameter formats in prose adds tokens and creates a second source of truth that drifts from the first.

Few-shot examples of entire conversations. They are enormous, they anchor the model on their specific shape, and in an agent loop they get re-sent every turn. If you need to demonstrate a format, demonstrate the one field that is going wrong, not the whole exchange.

Politeness scaffolding. "Please try your best" and "this is very important" do not survive contact with a tool loop and inflate every request you will ever make.

Treat it as versioned code, because it is

A one-word change to a system prompt is a production behaviour change with no type system to catch it. Give the prompt a version string, log that version with every trace, and run your eval set before and after any edit. When behaviour shifts a week later, the first question is which prompt version was live, and you want that answerable from a log rather than from memory.

The failure this prevents is specific and common: someone adds a rule to fix one bad case, three other cases quietly regress, and nobody connects the two because the change was invisible. Prompts drift the same way code does. Version them the same way.

PROMPT_VERSION = "order-agent/v7"

SYSTEM_PROMPT = f"""{ROLE}

{RULES}

{TOOL_GUIDANCE}

{ESCAPE_HATCH}"""

# Every span carries the version, so a behaviour change is traceable
# to the edit that caused it.
span.set_attribute("prompt.version", PROMPT_VERSION)