Structured Outputs Cheatsheet: JSON Schema, Pydantic, Constrained Decoding

Most LLM features end in code, not in a chat window. An extractor fills a database row, a router picks a queue, an agent calls a function. Every one of those needs output a program can parse, and "please answer in JSON" is a request, not a guarantee.

Two years ago the standard answer was a prompt, a regex to strip the markdown fences, json.loads in a try block, and a retry. Today every major API and every serious open-source inference server can constrain generation to a schema: the model is not allowed to produce a token that would break it. That changes what you need to test, but it does not remove the need to validate.

This post is a cheatsheet for that stack: the vocabulary, the JSON Schema subset that actually works, Pydantic patterns, how constrained decoding works, what research says about quality, and the failure modes that survive a "guaranteed" schema.

1. The vocabulary

The progression matters: prompted JSON gives you probably parseable, JSON mode gives you parseable, structured outputs give you parseable and correctly shaped. None of them give you correct.

2. Where each option stands (May 2026)

If you are upgrading old code: the guided_json field in vLLM and the output_format field in Claude's beta are both gone. Check before you copy a snippet from an old issue thread.

3. The JSON Schema subset that actually works

Every provider supports a subset of JSON Schema, and the subsets differ. Design for the intersection unless you are locked to one stack.

OpenAI also documents hard size limits: up to 5,000 object properties, 10 levels of nesting, and 1,000 enum values per schema. You will hit nesting limits long before property limits.

Rules of thumb:

• Make every field required, and use null for "not present". It is portable, and it forces the model to make an explicit choice instead of silently skipping a field.
• Use enums for anything categorical. A free-text status field will eventually contain "Pending (awaiting reply)".
• Add description to fields. Descriptions go to the model with the schema. They are the field-level prompt.
• Keep constraints you cannot enforce in validation. If a provider ignores maximum, the schema will still accept 9,999 stars on a 1-5 rating.
• Put reasoning before answers. Keys are generated in order. A reasoning field before label gives the model room to think; a label before reasoning produces a justification for a decision already made.

4. Pydantic as the source of truth

Write the schema once as a Pydantic model and derive everything else from it: the JSON Schema you send, the parsing, the validation and your type hints. Pydantic v2 (2.13 is current at the time of writing) has the three methods you need.

# /// script
# dependencies = ["pydantic>=2"]
# ///
import json
from enum import Enum

from pydantic import BaseModel, ConfigDict, Field, ValidationError, field_validator


class Priority(str, Enum):
    low = "low"
    medium = "medium"
    high = "high"


class Ticket(BaseModel):
    model_config = ConfigDict(extra="forbid")  # emits additionalProperties: false

    reasoning: str = Field(description="One or two sentences on how priority was chosen")
    title: str = Field(description="One-line summary of the issue")
    priority: Priority
    customer_email: str | None = Field(description="Email if present in the text, else null")
    tags: list[str] = Field(description="Zero to five lowercase tags")

    @field_validator("tags")
    @classmethod
    def at_most_five(cls, v: list[str]) -> list[str]:
        if len(v) > 5:
            raise ValueError("at most 5 tags")
        return [t.lower() for t in v]


schema = Ticket.model_json_schema()        # send this to the model
print(json.dumps(schema["required"]))      # every field is required, email is nullable

raw = '{"reasoning": "Payment issue.", "title": "Refund not received", "priority": "urgent", "customer_email": null, "tags": ["billing"]}'
try:
    Ticket.model_validate_json(raw)        # parse and validate in one step
except ValidationError as e:
    print(e.errors()[0]["loc"], e.errors()[0]["type"])   # ('priority',) enum

Provider SDKs accept the model directly. With OpenAI's Python SDK:

from openai import OpenAI

client = OpenAI()
response = client.responses.parse(
    model="gpt-4o-2024-08-06",
    input=[
        {"role": "system", "content": "Extract a support ticket from the email."},
        {"role": "user", "content": email_text},
    ],
    text_format=Ticket,
)
ticket = response.output_parsed  # a Ticket instance, or None on refusal

With Claude, send the schema in output_config:

import anthropic

client = anthropic.Anthropic()
message = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": f"Extract a support ticket:\n\n{email_text}"}],
    output_config={"format": {"type": "json_schema", "schema": Ticket.model_json_schema()}},
)
ticket = Ticket.model_validate_json(message.content[0].text)

Either way, validate with your own model at the end. The provider guarantees the shape it understood, and your validators check the rules it did not.

5. How constrained decoding works

A language model produces a probability for every token in its vocabulary at each step. Constrained decoding inserts one step before sampling:

1. Compile the schema or regex into an automaton: a finite-state machine for regular languages, a pushdown automaton for context-free grammars such as nested JSON.
2. At each step, compute which vocabulary tokens are valid continuations from the current state.
3. Set the probability of all other tokens to zero (mask the logits), then sample as usual.
4. Advance the automaton with the chosen token.

The hard part is speed. Vocabularies have 100,000 or more tokens, and a token can span several grammar symbols. Each library attacks that differently:

Practical consequences:

• The first request with a new schema can be slower. Grammars are compiled and cached. Claude's GA notes mention improved grammar compilation latency; reuse schemas instead of generating a new one per request.
• Constraints change the distribution. Masking forces the model down paths it may have assigned low probability. A schema that fights the model's natural phrasing (odd key names, unexpected order) costs quality.
• Constraints do not add knowledge. A required invoice_number will be filled whether or not the document contains one. Give the model a legal way to say "not found": a nullable field or an explicit "unknown" enum value.

6. Does constraining hurt quality?

The research is mixed, and the details matter.

The reconciling view: constraints hurt when they leave the model no room to reason or when the schema fights the task. They help or are neutral when the schema includes a reasoning field before the answer and the prompt matches the format. Test on your own task: run the same eval with and without the schema, and compare accuracy, not only parse rate.

7. Failure modes that survive a schema

A provider-independent wrapper for the whole loop is short. It also works with prompted JSON on models without native support:

from typing import Callable, TypeVar

from pydantic import BaseModel, ValidationError

T = TypeVar("T", bound=BaseModel)


def generate_structured(call: Callable[[str], tuple[str, str]], model: type[T], prompt: str, retries: int = 2) -> T:
    """call(prompt) returns (text, stop_reason). Retries once per validation error, with feedback."""
    attempt_prompt = prompt
    for attempt in range(retries + 1):
        text, stop = call(attempt_prompt)
        if stop in ("length", "max_tokens"):
            raise RuntimeError("Output truncated: raise the token limit or shrink the schema")
        try:
            return model.model_validate_json(text)
        except ValidationError as e:
            if attempt == retries:
                raise
            attempt_prompt = f"{prompt}\n\nYour previous answer was invalid:\n{e}\nReturn corrected JSON only."
    raise AssertionError("unreachable")

With native structured outputs the retry branch should almost never fire on shape errors, but it still catches your own validator rules. Count how often it fires: a rising retry rate is an early warning that a prompt or model change broke something.

8. Testing structured output

Treat the schema as an API contract and the extraction as a model behavior. Test both.

The point is to separate the two failure classes. A validity failure is a plumbing problem: schema, provider, truncation. An accuracy failure is a model or prompt problem. Structured outputs largely solved the first; the second is where your eval effort belongs.

Why this matters

• Shape is solved; content is not. Native structured outputs remove parse errors, so the remaining errors are about correctness, which is where validation and evals must focus.
• The portable subset is small. Required-plus-nullable fields, enums, additionalProperties: false and shallow nesting work almost everywhere; range and length constraints belong in validation.
• One Pydantic model, many uses. Schema, parser, validator and type hints from one definition prevents drift between prompt and code.
• Schema design is prompt design. Field order, names, descriptions and an explicit "not found" option change answers, not just formatting.
• Measure the trade-off on your task. Research on format restrictions disagrees; a with-and-without eval on your data settles it for your case.

Final Thoughts

Structured outputs moved JSON from "hope and retry" to a guarantee enforced at the token level. That is a real improvement, and it makes LLMs much easier to wire into ordinary software.

But a guarantee about shape tempts teams to stop checking content. Keep the Pydantic validators, keep the null options, keep an eval with labeled examples. The schema makes the output parseable; only your tests make it trustworthy.

Sources

• OpenAI, Introducing Structured Outputs in the API (August 2024): launch, strict: true, 100% vs. under 40% schema-following eval.
• OpenAI, Structured Outputs guide: supported schema subset, required and nullable fields, additionalProperties: false, size limits, JSON mode vs. structured outputs, refusals.
• OpenAI, openai-python changelog: parse helpers, moved out of beta in v1.92.0 (June 2025).
• Anthropic, Claude API release notes (November 2025, January 2026): structured outputs beta and general availability, move to output_config.format.
• Anthropic, Structured outputs: JSON outputs, strict tool use, unsupported schema features.
• Google, Improving Structured Outputs in the Gemini API (November 2025): JSON Schema keyword support, Pydantic and Zod, key ordering.
• Google, Structured output, Gemini API docs: unenforced keywords, schema size limits.
• vLLM, Structured Outputs and release notes for v0.11.0 (October 2025) and v0.12.0 (December 2025): parameters, backends, removal of guided_*.
• llama.cpp, GBNF grammars README: grammar syntax, JSON Schema conversion, unsupported keywords.
• Ollama, Structured outputs (December 2024): format parameter with Pydantic schemas.
• Pydantic, pydantic on PyPI (2.13.4, May 2026) and JSON Schema docs.
• Willard and Louf, Efficient Guided Generation for Large Language Models (July 2023): FSM-based guided generation (Outlines).
• Dong et al., XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models (November 2024): token splitting, up to 100x speedup.
• guidance-ai, llguidance README: integrations and OpenAI adoption note (May 2025).
• Tam et al., Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models (August 2024).
• Will Kurt (.txt), Say What You Mean: A Response to 'Let Me Speak Freely' (November 2024).
• Geng et al., Generating Structured Outputs from Language Models: Benchmark and Studies (JSONSchemaBench) (January 2025).