Rails LLM Structured Output: Reliable JSON from Claude and OpenAI with Schema Validation

A logistics client called me on a Tuesday because their AI-powered shipment classifier was silently dropping 3% of orders into an “unclassified” queue that nobody was watching. It had been running clean for six weeks. The change that broke it was not a code change — Anthropic had released a new Claude model, the classifier had swapped to it, and the model was now wrapping its JSON response in ` json ... ` markdown fences roughly one time in thirty. Their parser did JSON.parse(response.text) and raised, the exception was rescued, the shipment got tagged unclassified, and their ops team quietly re-classified them by hand every morning without telling engineering. Rails LLM structured output — done properly, with schemas and validation, not string parsing — would have caught this on the first request.

After nineteen years of Rails and three of shipping LLM features into production, I have deleted every regex I ever wrote to extract JSON from a model response. The industry finally converged on schema-constrained outputs in late 2024, both Claude and OpenAI expose it through their APIs, and there is no longer any reason to gsub triple backticks or count opening braces. This post is how I wire Rails LLM structured output into a Rails app for both providers, how I validate what comes back, and how I handle the failure modes that still occur even with schema constraints.

Why Rails LLM Structured Output Beats Prompt Engineering

The pre-schema approach was to write a prompt like “return only JSON, no commentary, no markdown, no code fences” and then bolt a regex on top to extract the first {...} block in the response. This works most of the time. Most of the time is not good enough for production. Models drift, prompts get edited, a new fine-tune leaks a training artifact, and suddenly 3% of your requests come back as Here is the JSON you requested: json {...}`` and your parser explodes.

Rails LLM structured output using provider-native schema constraints is different. The model is not asked to produce JSON — it is grammatically restricted at the token-sampling step so it can only emit tokens that keep the output valid against your schema. Claude does this via tool use (you define a tool with an input schema and force the model to call it). OpenAI does this via response_format: { type: "json_schema", ... }. In both cases the output is guaranteed to parse as JSON and guaranteed to conform to your schema keys, types, and required fields. Refusals and safety responses still exist, but you know at parse time whether you got a valid payload or a refusal — no ambiguity.

The other reason to care: Rails apps that call LLMs generally feed the response into some downstream typed system — an ActiveRecord model, a Sidekiq job argument, a Turbo Stream partial. When the shape is guaranteed, you can drop the defensive response.dig(:items, 0, :name) rescue nil scaffolding that accumulates around every LLM call site.

The Schema Is the API Contract

Before touching either provider SDK, define your schema in one place. I use plain Ruby hashes because both provider APIs want JSON Schema and I do not want a translation layer between my schema definition and the wire format.

# app/llm/schemas/shipment_classification.rb
module LLM
  module Schemas
    SHIPMENT_CLASSIFICATION = {
      type: "object",
      properties: {
        category: {
          type: "string",
          enum: %w[electronics apparel food hazardous other],
          description: "Primary shipment category based on the item description."
        },
        confidence: {
          type: "number",
          minimum: 0,
          maximum: 1,
          description: "Model confidence between 0.0 and 1.0."
        },
        requires_review: {
          type: "boolean",
          description: "True if the item is ambiguous or borderline hazardous."
        },
        reasoning: {
          type: "string",
          description: "One-sentence justification, plain English, no markdown."
        }
      },
      required: %w[category confidence requires_review reasoning],
      additionalProperties: false
    }.freeze
  end
end

Two things matter here. required must list every key you plan to read, or the model may omit them and your .fetch(:category) will raise. additionalProperties: false prevents the model from inventing extra fields that then get logged and confuse debugging. Both providers respect these constraints strictly.

The description fields are load-bearing — they are the actual prompt the model uses to decide what to emit for each field. Treat them like docstrings for future-you.

Claude Tool Use for Structured Output

Claude does not have a json_mode flag. The idiomatic pattern is to define a single tool whose input_schema is your schema, then set tool_choice to force the model to call it. The response comes back as a tool_use block whose input is a validated instance of your schema.

# app/llm/claude_client.rb
require "anthropic"

class LLM::ClaudeClient
  MODEL = "claude-sonnet-5"

  def initialize
    @client = Anthropic::Client.new(api_key: ENV.fetch("ANTHROPIC_API_KEY"))
  end

  def structured(system:, user:, schema:, tool_name:)
    response = @client.messages.create(
      model: MODEL,
      max_tokens: 1024,
      system: system,
      messages: [{ role: "user", content: user }],
      tools: [{
        name: tool_name,
        description: "Return the classification result.",
        input_schema: schema
      }],
      tool_choice: { type: "tool", name: tool_name }
    )

    tool_use = response.content.find { |b| b.type == "tool_use" }
    raise LLM::EmptyResponse, "no tool_use block returned" unless tool_use

    tool_use.input.deep_symbolize_keys
  end
end

tool_choice: { type: "tool", name: tool_name } is the part people miss. Without it, Claude may respond in plain text if it thinks the tool is not the best answer, which reintroduces the parsing problem you were trying to avoid. Forcing the tool guarantees the response is a structured tool_use block. The input on that block is a Ruby hash whose shape matches your schema.

One quirk: Claude occasionally returns nil or empty for optional-looking numeric fields even when they are required. I have not been able to reproduce it reliably. My defense is downstream validation, which I get to in a moment.

OpenAI json_schema Response Format

OpenAI added response_format: { type: "json_schema" } in late 2024 and it is now the correct way to do Rails LLM structured output with GPT models. The syntax is verbose but explicit — you pass the schema, a name, and strict: true.

# app/llm/openai_client.rb
require "openai"

class LLM::OpenAIClient
  MODEL = "gpt-4o-2024-11-20"

  def initialize
    @client = OpenAI::Client.new(access_token: ENV.fetch("OPENAI_API_KEY"))
  end

  def structured(system:, user:, schema:, schema_name:)
    response = @client.chat(
      parameters: {
        model: MODEL,
        messages: [
          { role: "system", content: system },
          { role: "user",   content: user }
        ],
        response_format: {
          type: "json_schema",
          json_schema: {
            name: schema_name,
            strict: true,
            schema: schema
          }
        }
      }
    )

    message = response.dig("choices", 0, "message")
    raise LLM::Refusal, message["refusal"] if message["refusal"]

    JSON.parse(message["content"], symbolize_names: true)
  end
end

Two OpenAI-specific things. First, strict: true is required for the schema constraint to be enforced — without it the model treats the schema as a suggestion. Second, when the model refuses (safety filter, policy violation), the response comes back with content: null and a refusal field containing the refusal reason as plain text. Check for it explicitly. If you skip that check and do JSON.parse(nil) you will get a TypeError and lose the refusal reason, which is exactly the information you needed to log.

A Provider-Agnostic Adapter

Real apps end up using both providers — one for the primary path, one for fallback, sometimes one per feature depending on cost and latency. Wrap them behind a single interface so callers do not care which one served the request.

# app/llm/structured_output.rb
class LLM::StructuredOutput
  Result = Struct.new(:data, :provider, :model, :usage, keyword_init: true)

  def self.call(prompt:, schema:, schema_name:, provider: :claude)
    client = case provider
             when :claude then LLM::ClaudeClient.new
             when :openai then LLM::OpenAIClient.new
             else raise ArgumentError, "unknown provider #{provider}"
             end

    raw = client.structured(
      system: prompt[:system],
      user: prompt[:user],
      schema: schema,
      tool_name: schema_name,
      schema_name: schema_name
    )

    Result.new(data: raw, provider: provider, model: client.class::MODEL)
  end
end

client.structured takes tool_name and schema_name because Claude wants the former and OpenAI the latter; each client ignores the argument it does not need. This keeps the caller ignorant of provider quirks.

Validating What Came Back

Schema-constrained outputs still fail in ways schemas cannot catch. A confidence of 0.5 may satisfy minimum: 0, maximum: 1 but be nonsense for your business logic if you require confidence above 0.7 to auto-approve. A category of hazardous may pass the enum check but conflict with the item description in a way you can only detect by cross-referencing your own database. Treat the schema as syntactic validation and add semantic validation on top.

I use dry-validation for the semantic pass because its contracts compose well with ActiveRecord and produce structured error objects I can log.

# app/llm/contracts/shipment_classification.rb
require "dry/validation"

module LLM
  module Contracts
    class ShipmentClassification < Dry::Validation::Contract
      schema do
        required(:category).filled(:string, included_in?: %w[electronics apparel food hazardous other])
        required(:confidence).filled(:float, gteq?: 0.0, lteq?: 1.0)
        required(:requires_review).filled(:bool)
        required(:reasoning).filled(:string, min_size?: 5, max_size?: 500)
      end

      rule(:reasoning) do
        key.failure("looks like markdown") if value.include?("```") || value.include?("**")
      end
    end
  end
end

Now the call site pattern becomes: call the LLM, validate with the contract, decide what to do on failure.

class ShipmentClassifier
  def classify(item)
    result = LLM::StructuredOutput.call(
      prompt: build_prompt(item),
      schema: LLM::Schemas::SHIPMENT_CLASSIFICATION,
      schema_name: "shipment_classification",
      provider: :claude
    )

    validation = LLM::Contracts::ShipmentClassification.new.call(result.data)

    if validation.success?
      Classification.create!(item: item, **result.data, model: result.model)
    else
      Rails.logger.warn(
        event: "llm_semantic_validation_failed",
        errors: validation.errors.to_h,
        raw: result.data,
        model: result.model
      )
      raise LLM::ValidationError, validation.errors.to_h
    end
  end
end

The Rails.logger.warn line is not decorative. Semantic failures against otherwise schema-valid output are the interesting bugs — a leaked training artifact, prompt drift, a new model behaving differently. You want them in your logs where you can query them later, not swallowed.

Retrying, But Only for the Right Failures

Structured output can still fail three ways: transport error (network, 5xx), refusal, and semantic validation. Retry the first, do not retry the second, retry the third once with a nudge in the prompt.

class LLM::WithRetry
  MAX_ATTEMPTS = 3

  def self.call(&block)
    attempts = 0
    begin
      attempts += 1
      block.call(attempt: attempts)
    rescue Faraday::TimeoutError, Faraday::ConnectionFailed, Anthropic::APIError => e
      retry if attempts < MAX_ATTEMPTS
      raise
    rescue LLM::Refusal
      raise
    rescue LLM::ValidationError => e
      raise if attempts >= 2
      retry
    end
  end
end

Refusals are not retryable — the model already decided your request was outside policy, so retrying with the same input will refuse again and waste tokens. Semantic validation failures are worth one retry because the model may sample a different (valid) response, but not more than one or you are just paying for degraded quality.

For the retry on validation, pass the previous response back in the next call so the model can see what it got wrong:

def build_prompt(item, previous_error: nil)
  user = "Classify this shipment item: #{item.description}"

  if previous_error
    user += "\n\nYour previous response had this validation error: " \
            "#{previous_error}. Please fix and re-emit."
  end

  { system: SYSTEM_PROMPT, user: user }
end

What This Replaces

Before Rails LLM structured output landed as a first-class feature, every serious Rails app had a file that looked like this:

def extract_json(text)
  cleaned = text.gsub(/^```(?:json)?\s*/, "").gsub(/```$/, "")
  match = cleaned.match(/\{.*\}/m)
  return nil unless match
  JSON.parse(match[0])
rescue JSON::ParserError
  nil
end

Every one of those files was a source of silent bugs. The regex misses nested braces, the rescue swallows the real error, the return value is nil on failure so the caller cannot distinguish “no response” from “invalid response”. Delete these files. Use response_format on OpenAI and forced tool use on Claude. This is not a micro-optimization — it is closing an entire class of production incidents.

There is a related pattern for extracting structured output from open-source models that do not support schemas natively (Llama, Mistral). The library ecosystem has converged on grammar-constrained sampling (llama.cpp GBNF grammars, outlines in Python, guidance). If you are running your own inference and need Rails LLM structured output, the Ruby side is the same — pass a JSON Schema over HTTP — but the server has to support it. Most Rails teams I work with are not running their own inference in 2026; they are calling Claude, OpenAI, or Gemini directly, and all three support schemas.

Costs and Latency

Schema-constrained output costs the same as unconstrained output on both providers — you pay for input and output tokens, and the schema takes some input tokens. In practice the schema description also reduces output tokens because the model does not need to explain what it is returning; it just emits the fields. On my logistics client’s classifier, moving from prompt-engineered JSON to schema-constrained tool use cut output tokens by about 40% because the model stopped adding “Here is the classification you requested:” preambles.

Latency is essentially identical. Grammar-constrained sampling is a small overhead on the model side, but network round-trip dominates so you will not measure it.

FAQ

What is Rails LLM structured output and why does it matter?

Rails LLM structured output is the pattern of using provider-native schema constraints (Claude tool use, OpenAI json_schema response format) to force LLM responses into a validated JSON shape instead of parsing free-text with regexes. It eliminates the entire class of “the model wrapped its output in markdown” bugs and lets you treat LLM responses like typed API responses.

Does Claude support JSON mode like OpenAI?

Not as a top-level flag. Claude achieves the same result through forced tool use — define a tool whose input_schema is your schema, set tool_choice: { type: "tool", name: "..." }, and the response comes back as a tool_use block whose input is a validated instance of your schema. Functionally equivalent to OpenAI’s response_format: { type: "json_schema" }.

How do I handle refusals with structured output?

OpenAI returns a refusal field on the message object when the model declines to respond; check for it before calling JSON.parse on content. Claude does not have an explicit refusal field for tool use — if the model refuses, it will return a text block instead of a tool_use block, so check for the presence of the expected tool_use before extracting. In both cases, log the refusal and do not retry with the same input.

Can I use dry-validation with LLM structured output?

Yes, and I recommend it. Schema constraints give you syntactic validation (right types, right keys), but semantic validation (values that make sense for your business) still requires a separate pass. Dry-validation contracts compose well with ActiveRecord, produce structured error objects, and let you enforce rules the JSON schema cannot express — like “the reasoning field must not contain markdown”.

Need help wiring reliable AI features into a Rails app without the JSON-parsing horror show? TTB Software specializes in Rails, LLM integration, and shipping production-grade features that do not break at 3am. We’ve been doing this for nineteen years.