Structured Output: JSON Mode, Tool Calling, or Grammar Constraints?

The problem, stated precisely

Something downstream needs typed data. A database write, a UI component, another service. The model produces text. Between those two facts sits a decision with more depth than it first appears.

The naive approach — ask for JSON in the prompt and parse what comes back — works most of the time, which is the worst possible failure profile. At ninety-five percent success you will not catch it in testing and you will absolutely see it in production, usually as a markdown code fence wrapped around otherwise-valid JSON, or a trailing comma, or a cheerful 'Here's the JSON you asked for:' prefix.

Three mechanisms give you a real guarantee instead. They are not interchangeable.

Mechanism one: JSON mode

JSON mode constrains decoding so the output is syntactically valid JSON. Nothing more. You are guaranteed it parses; you are not guaranteed it has the fields you wanted, or that the types are right, or that a required key is present.

That makes it a good fit when the shape is simple and you validate afterwards anyway, and a poor fit when the schema is nested or the consumer is strict. In practice JSON mode has been largely superseded by schema-constrained variants for new work, but it is still what you get by default in a lot of code and it is worth knowing what it does and does not promise.

One trap: with plain JSON mode you must still describe the desired keys in the prompt. The mode constrains syntax, not vocabulary. Teams that switch it on and delete the schema from the prompt get valid JSON with invented field names.

Mechanism two: schema-constrained output

This is the strict version. You supply a JSON Schema and the provider constrains generation so that the output conforms — every required field present, every type correct, no additional properties. The parse is guaranteed and so is the shape.

The catch is that providers support a subset of JSON Schema, and the unsupported keywords are usually the ones you reach for when you get precise. Things like minimum/maximum on numbers, minLength, and complex conditional composition are commonly ignored even when accepted. The safe pattern is to treat the provider's constraint as structural and enforce semantics yourself with a real validator on the way out.

Use this when a machine consumes the output and the schema is fixed: extraction, classification, form filling, any populate-this-record task.

SCHEMA = {
  "type": "object",
  "properties": {
    "sentiment": {"type": "string", "enum": ["positive", "neutral", "negative"]},
    "themes":    {"type": "array", "items": {"type": "string"}},
    "urgency":   {"type": "integer"},          # provider guarantees integer
    "quote":     {"type": "string"},
  },
  "required": ["sentiment", "themes", "urgency", "quote"],
  "additionalProperties": False,
}

raw = call_model_with_schema(review_text, SCHEMA)   # shape guaranteed

# Semantics are NOT guaranteed \u2014 validate the range yourself.
if not 1 <= raw["urgency"] <= 5:
    raw["urgency"] = max(1, min(5, raw["urgency"]))

Mechanism three: a tool the model calls to answer

The third option is to define a tool whose only purpose is to receive the answer — submit_analysis, record_extraction — and instruct the model to finish by calling it. The tool's input schema is your output schema, and you take the arguments as the result.

This sounds like a hack and is in fact the cleanest option inside an agent, for two reasons. First, you are already in a tool-calling loop, so the machinery exists and the schema enforcement is the same. Second, and more usefully, it composes with real tools: the model can search, query, and read on turns one through four, then call submit_analysis on turn five. A response-format constraint applies to the final message only and sits awkwardly beside a tool loop; a terminal tool is part of it.

It also gives you a clean stop condition, which is the thing the first post in this series flagged as a design decision. The loop ends when submit_analysis is called, not when the model happens to emit text.

SUBMIT = {
    "name": "submit_analysis",
    "description": "Record the final analysis. Call this exactly once, when "
                   "you have gathered enough evidence. Calling it ends the task.",
    "input_schema": SCHEMA,
}

def run(text):
    messages = [{"role": "user", "content": text}]
    for _ in range(MAX_TURNS):
        reply = model(messages, tools=[*RESEARCH_TOOLS, SUBMIT])
        messages.append(assistant(reply))
        for call in tool_calls(reply):
            if call.name == "submit_analysis":
                return call.input          # typed result, loop over
            messages.append(run_tool(call))
    raise RuntimeError("no analysis submitted")

The cost nobody mentions: constraint can degrade content

Here is the part that matters and rarely appears in the documentation. Forcing a model to emit a rigid structure can reduce the quality of what is inside the structure. The format comes out perfect; the reasoning inside it gets thinner.

The mechanism is straightforward once you see it. Unconstrained, a model will often work towards an answer — weigh a couple of options, notice a contradiction, then commit. A schema that demands a bare verdict as the first key gives it nowhere to do that work, so it commits first and the reasoning never happens.

The fix is to put the reasoning in the schema. Add a field for it, and order the schema so that field comes before the conclusion, because generation is sequential and a model cannot condition on something it has not written yet. This costs tokens and it is usually worth it on any judgement task. On pure extraction, where there is nothing to reason about, skip it.

If you have an eval set — and this is one of the clearest arguments for having one — measure it rather than guessing. Run the same task constrained and unconstrained, score the content, and see what the constraint cost you. On classification the answer is often nothing. On anything analytical it is frequently substantial, and recoverable with a reasoning field.

{
  "type": "object",
  "properties": {
    "evidence":  { "type": "string",
                   "description": "Specific quotes or facts that informed the call." },
    "reasoning": { "type": "string",
                   "description": "Weigh the evidence. Note anything contradictory." },
    "verdict":   { "type": "string", "enum": ["approve", "reject", "escalate"] },
    "confidence":{ "type": "string", "enum": ["low", "medium", "high"] }
  },
  "required": ["evidence", "reasoning", "verdict", "confidence"]
}
// Order matters: the model writes evidence and reasoning BEFORE verdict,
// so the verdict is conditioned on them. Reverse it and you get a guess
// followed by a justification for that guess.

Choosing, in one pass

If the output is prose for a human, do not constrain it at all. This sounds obvious and is violated constantly — a schema wrapped around a single message field buys nothing and costs flexibility.

If you need typed data at the end of a single call, with no tools involved, use schema-constrained output. It is the lowest-ceremony option that gives a real guarantee.

If you are inside an agent loop that also calls real tools, use a terminal tool. It composes with what you already have and gives you a clean stop condition.

Whichever you pick, validate on the way out. Provider constraints cover structure, not meaning — an integer where you wanted 1 to 5 can still arrive as 47, and an enum can still be the wrong member. And whenever the task involves judgement, put a reasoning field in the schema before the conclusion. That single ordering decision recovers most of what strict formatting takes away.