Rails LLM Observability: Tracing Prompts, Latency, and Token Usage in Production with Langfuse

Three weeks after a client shipped their AI-powered document summarizer, I got a Slack message on a Tuesday afternoon: “Why is the summarize button taking forever?” I opened the Rails request log. Every LLM call was returning in 4.2 seconds. Two weeks earlier, the median had been 800 milliseconds. The feature had been degrading for days before anyone noticed, because nobody was watching the right numbers.

The request log gave me HTTP response times. The OpenTelemetry setup I’d wired in the previous quarter gave me database query timings. Neither told me what was happening inside the LLM call: which model was being used, how many tokens it was sending, whether the prompt had changed recently, or whether the slowdown was coming from token generation or network round-trip. I was flying blind inside the AI layer.

Rails LLM observability is the instrumentation that fixes this. It is not the same as HTTP request tracing or database query profiling — it is a layer above those, specific to how LLMs behave in production.

Why Generic Observability Misses the AI Layer

OpenTelemetry wired into Rails gives you spans for every HTTP request, database query, and background job. I covered the full setup in the OpenTelemetry Rails post. It is excellent infrastructure and I run it on every client app. But it cannot tell you:

• Which prompt template generated the slow request
• Whether your new Claude Sonnet 5 migration actually reduced latency
• How many input tokens the average summarization request sends
• Whether yesterday’s prompt edit caused the error rate to spike
• What your daily token cost is by operation, not just by billing month

These are the questions that matter when an AI feature misbehaves in production. HTTP tracing sees a 200-OK with a 4.2-second body. Rails LLM observability sees “the summarize operation on claude-sonnet-5 with prompt version v14 processed 8,200 input tokens in 3.8 seconds, cost $0.041, and succeeded — but the same operation on requests with documents over 5,000 tokens degraded by 2x starting Thursday morning.”

That is actionable. The HTTP trace is not.

Instrumenting LLM Calls with ActiveSupport::Notifications

The Rails instrumentation layer is ActiveSupport::Notifications. It is already in your app — it is what powers ActiveRecord::LogSubscriber and the request timing you see in your development logs. Add a wrapper around your LLM client that publishes an event for every call.

Start by defining the event schema in one place:

# app/llm/instrumentation.rb
module LLM
  module Instrumentation
    EVENT_NAME = "llm.call".freeze

    def self.instrument(operation:, model:, prompt_version: nil, tenant_id: nil, &block)
      payload = {
        operation:      operation,
        model:          model,
        prompt_version: prompt_version,
        tenant_id:      tenant_id,
        started_at:     Process.clock_gettime(Process::CLOCK_MONOTONIC)
      }

      ActiveSupport::Notifications.instrument(EVENT_NAME, payload) do
        result = block.call
        payload[:input_tokens]  = result[:usage]&.dig(:input_tokens)
        payload[:output_tokens] = result[:usage]&.dig(:output_tokens)
        result
      end
    rescue => e
      payload[:error]     = e.class.name
      payload[:error_msg] = e.message.first(200)
      raise
    ensure
      elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - payload[:started_at]
      payload[:duration_ms] = (elapsed * 1000).round
    end
  end
end

Wrap your existing Claude client to emit this event on every call:

# app/llm/claude_client.rb
require "anthropic"

class LLM::ClaudeClient
  MODEL = "claude-sonnet-5"

  def initialize
    @client = Anthropic::Client.new(api_key: ENV.fetch("ANTHROPIC_API_KEY"))
  end

  def complete(system:, user:, operation:, prompt_version: nil, tenant_id: nil, max_tokens: 1024)
    LLM::Instrumentation.instrument(
      operation:      operation,
      model:          MODEL,
      prompt_version: prompt_version,
      tenant_id:      tenant_id
    ) do
      response = @client.messages.create(
        model:      MODEL,
        max_tokens: max_tokens,
        system:     system,
        messages:   [{ role: "user", content: user }]
      )

      {
        text:  response.content.first.text,
        usage: {
          input_tokens:  response.usage.input_tokens,
          output_tokens: response.usage.output_tokens
        }
      }
    end
  end
end

The prompt_version parameter is a string like "v14" or an 8-character content hash. It is optional — you can add it to call sites incrementally. The tenant_id parameter lets you filter traces per customer, which matters the moment you have more than one paying account and someone files a support ticket.

At the call site:

class DocumentSummarizer
  PROMPT_VERSION = "v14"
  SYSTEM_PROMPT  = <<~PROMPT.freeze
    You are a precise document summarizer. Return a factual summary
    in 2-4 sentences covering the main points. No bullet points.
    No preamble. Just the summary.
  PROMPT

  def summarize(document, tenant_id:)
    client = LLM::ClaudeClient.new
    result = client.complete(
      system:         SYSTEM_PROMPT,
      user:           "Summarize this document:\n\n#{document.body}",
      operation:      "document.summarize",
      prompt_version: PROMPT_VERSION,
      tenant_id:      tenant_id,
      max_tokens:     256
    )
    document.update!(summary: result[:text])
  end
end

Subscribing and Shipping Traces to Langfuse

Langfuse is an open source Rails LLM observability platform with a self-hostable Docker server and a cloud-hosted version. Its data model speaks LLM natively: traces, generations, prompt versions, scores. Your ActiveSupport::Notifications subscriber publishes every call to Langfuse in the background without blocking the request path.

Add the gem:

# Gemfile
gem "langfuse"

Then build the subscriber:

# app/llm/langfuse_subscriber.rb
class LLM::LangfuseSubscriber
  INPUT_TOKEN_COSTS = {
    "claude-sonnet-5"   => 0.000003,
    "claude-haiku-4-5"  => 0.00000025
  }.freeze

  OUTPUT_TOKEN_COSTS = {
    "claude-sonnet-5"   => 0.000015,
    "claude-haiku-4-5"  => 0.00000125
  }.freeze

  def self.attach
    ActiveSupport::Notifications.subscribe(LLM::Instrumentation::EVENT_NAME) do |*args|
      event = ActiveSupport::Notifications::Event.new(*args)
      new(event).call
    end
  end

  def initialize(event)
    @event   = event
    @payload = event.payload
    @client  = Langfuse.new(
      public_key: ENV.fetch("LANGFUSE_PUBLIC_KEY"),
      secret_key: ENV.fetch("LANGFUSE_SECRET_KEY"),
      host:       ENV.fetch("LANGFUSE_HOST", "https://cloud.langfuse.com")
    )
  end

  def call
    trace = @client.trace(
      name:     @payload[:operation],
      metadata: {
        prompt_version: @payload[:prompt_version],
        tenant_id:      @payload[:tenant_id],
        rails_env:      Rails.env
      }
    )

    trace.generation(
      name:           @payload[:operation],
      model:          @payload[:model],
      start_time:     @event.time,
      end_time:       @event.end,
      usage:          usage_hash,
      level:          @payload[:error] ? "ERROR" : "DEFAULT",
      status_message: @payload[:error_msg]
    )

    @client.flush
  rescue => e
    Rails.logger.warn("LLM::LangfuseSubscriber failed: #{e.class}: #{e.message}")
  end

  private

  def usage_hash
    input_tokens  = @payload[:input_tokens].to_i
    output_tokens = @payload[:output_tokens].to_i
    model         = @payload[:model]

    {
      input:       input_tokens,
      output:      output_tokens,
      total:       input_tokens + output_tokens,
      input_cost:  (input_tokens  * INPUT_TOKEN_COSTS.fetch(model,  0)).round(6),
      output_cost: (output_tokens * OUTPUT_TOKEN_COSTS.fetch(model, 0)).round(6),
      total_cost:  ((input_tokens  * INPUT_TOKEN_COSTS.fetch(model,  0)) +
                    (output_tokens * OUTPUT_TOKEN_COSTS.fetch(model, 0))).round(6),
      unit:        "TOKENS"
    }
  end
end

Attach the subscriber in an initializer:

# config/initializers/llm_observability.rb
Rails.application.config.after_initialize do
  LLM::LangfuseSubscriber.attach
end

The rescue inside call is not optional. Observability infrastructure that takes down production when Langfuse is unreachable is worse than no observability at all.

If You Prefer to Stay in Postgres

Langfuse is the right tool for most teams shipping AI features in 2026. But if you cannot introduce another hosted service, a PostgreSQL-backed trace table takes thirty minutes to set up and integrates naturally with your existing Rails monitoring and reporting.

# db/migrate/20260804000000_create_llm_traces.rb
class CreateLlmTraces < ActiveRecord::Migration[8.0]
  def change
    create_table :llm_traces do |t|
      t.string  :operation,      null: false
      t.string  :model,          null: false
      t.string  :prompt_version
      t.bigint  :tenant_id
      t.integer :input_tokens
      t.integer :output_tokens
      t.integer :duration_ms
      t.decimal :cost,           precision: 10, scale: 6
      t.string  :error_class
      t.text    :error_message
      t.jsonb   :metadata,       default: {}
      t.timestamps
    end

    add_index :llm_traces, :operation
    add_index :llm_traces, :tenant_id
    add_index :llm_traces, :created_at
    add_index :llm_traces, [:operation, :created_at]
    add_index :llm_traces, [:tenant_id, :created_at]
  end
end

Once data is flowing into llm_traces, Postgres can answer the same questions Langfuse answers — just without the pre-built UI:

# p95 latency by operation over the last 24 hours
LlmTrace
  .where(created_at: 24.hours.ago..)
  .group(:operation)
  .select(
    "operation",
    "percentile_cont(0.95) WITHIN GROUP (ORDER BY duration_ms) AS p95_ms",
    "AVG(duration_ms) AS avg_ms",
    "COUNT(*) AS total"
  )

# daily cost by model over the past 7 days
LlmTrace
  .where(created_at: 7.days.ago..)
  .group(:model, "DATE(created_at)")
  .sum(:cost)

# error rate per operation
LlmTrace
  .where(created_at: 24.hours.ago..)
  .group(:operation)
  .select(
    "operation",
    "COUNT(*) AS total",
    "SUM(CASE WHEN error_class IS NOT NULL THEN 1 ELSE 0 END) AS errors",
    "ROUND(100.0 * SUM(CASE WHEN error_class IS NOT NULL THEN 1 ELSE 0 END) / COUNT(*), 2) AS error_pct"
  )

I covered the pg_stat_statements approach to querying slow SQL in the pg_stat_statements post. The same mindset applies here: percentile distributions over averages, always segment by operation and model before drawing conclusions about the system as a whole.

Automatic Prompt Version Tracking

Every time you change a prompt, you are running an implicit A/B test on production traffic. Without version tags on your traces, you cannot know whether the change helped or hurt. A manual string constant works but drifts — someone changes the prompt and forgets to bump PROMPT_VERSION.

Hash the prompt content automatically instead:

# app/llm/prompt_registry.rb
module LLM
  class PromptRegistry
    def self.version_for(prompt_string)
      Digest::SHA1.hexdigest(prompt_string)[0, 8]
    end

    def self.register(name, content)
      @prompts       ||= {}
      @prompts[name]   = { content: content, version: version_for(content) }
    end

    def self.get(name)
      @prompts&.fetch(name) { raise KeyError, "Unknown prompt: #{name}" }
    end
  end
end

Register prompts at load time:

# config/initializers/llm_prompts.rb
LLM::PromptRegistry.register(
  :document_summarizer,
  <<~PROMPT
    You are a precise document summarizer. Return a factual summary
    in 2-4 sentences covering the main points. No bullet points.
    No preamble. Just the summary.
  PROMPT
)

Use them in the call site:

class DocumentSummarizer
  def summarize(document, tenant_id:)
    prompt   = LLM::PromptRegistry.get(:document_summarizer)
    client   = LLM::ClaudeClient.new

    result = client.complete(
      system:         prompt[:content],
      user:           "Summarize:\n\n#{document.body}",
      operation:      "document.summarize",
      prompt_version: prompt[:version],
      tenant_id:      tenant_id
    )

    document.update!(summary: result[:text])
  end
end

When you edit the prompt in the initializer, the SHA1 hash changes, the new version tag appears automatically in Langfuse, and you can compare p95 latency and error rates between a3f2b1c0 and 7e9d4a21 without touching the call site. I covered the deeper topic of automated prompt regression testing in CI in the LLM evals post.

Alerting on What Matters

Rails LLM observability is only useful if it triggers action. Four alerts cover most production AI incidents:

Token cost spike. Compare the last hour’s spend against the 7-day rolling hourly average for the same operation. More than 2x is worth waking someone up, because it usually means a prompt bug is sending runaway context windows.

# app/jobs/llm_cost_alert_job.rb
class LlmCostAlertJob < ApplicationJob
  queue_as :monitoring

  def perform
    LlmTrace.distinct.pluck(:operation).each do |op|
      recent_cost = LlmTrace
        .where(operation: op, created_at: 1.hour.ago..)
        .sum(:cost).to_f

      baseline = LlmTrace
        .where(operation: op, created_at: 7.days.ago..1.day.ago)
        .sum(:cost).to_f / 168.0  # hourly average over 6 days

      next if baseline < 0.01

      if recent_cost > baseline * 2.0
        Sentry.capture_message(
          "LLM cost spike: #{op} — #{recent_cost.round(4)} vs " \
          "#{baseline.round(4)} per hour (#{(recent_cost / baseline).round(1)}x)"
        )
      end
    end
  end
end

Latency degradation. A p95 above your SLO should fire within minutes. Schedule this every five minutes via Solid Queue’s recurring task:

# config/recurring.yml
llm_cost_alert:
  class: LlmCostAlertJob
  schedule: "0 * * * *"   # every hour

llm_latency_check:
  class: LlmLatencyAlertJob
  schedule: "*/5 * * * *"

# app/jobs/llm_latency_alert_job.rb
class LlmLatencyAlertJob < ApplicationJob
  SLO_MS = { "document.summarize" => 3000, "chat.respond" => 5000 }.freeze

  def perform
    SLO_MS.each do |operation, threshold_ms|
      recent = LlmTrace.where(operation: operation, created_at: 15.minutes.ago..)
      next if recent.count < 10

      p95 = recent.pluck(:duration_ms).sort.then { |ms| ms[(ms.size * 0.95).ceil - 1] }

      if p95.to_i > threshold_ms
        Sentry.capture_message(
          "LLM latency SLO breach: #{operation} p95=#{p95}ms (SLO #{threshold_ms}ms)"
        )
      end
    end
  end
end

Error rate. More than 5% of traces with error_class set is unusual. Anything above 10% is a production incident.

Silent operations. An operation that produces zero traces in a window where it normally sees hundreds means a code path is broken or unreachable. This is the hardest failure to notice without explicit monitoring, because every individual request either never arrives at the LLM layer or fails silently before it.

Streaming Responses

Streaming responses — used when you pipe LLM output to the browser token by token via Turbo Streams — do not have a complete token count at the moment the stream opens. You accumulate usage from the final event. Update the client wrapper:

def stream(system:, user:, operation:, prompt_version: nil, tenant_id: nil, &block)
  accumulated_output = ""
  usage              = {}

  LLM::Instrumentation.instrument(
    operation:      operation,
    model:          MODEL,
    prompt_version: prompt_version,
    tenant_id:      tenant_id
  ) do
    @client.messages.stream_message(
      model:      MODEL,
      max_tokens: 1024,
      system:     system,
      messages:   [{ role: "user", content: user }]
    ) do |event|
      case event.type
      when "content_block_delta"
        chunk = event.delta.text
        accumulated_output += chunk
        block.call(chunk) if block
      when "message_delta"
        usage = {
          input_tokens:  event.usage.input_tokens,
          output_tokens: event.usage.output_tokens
        }
      end
    end

    { text: accumulated_output, usage: usage }
  end
end

The usage event arrives at the end of every Claude stream response. Accumulating it inside the instrumentation block ensures token counts appear in your Langfuse traces even for streaming calls. I covered the full SSE streaming setup for Rails in the streaming Claude responses post.

Healthcheck Endpoint

Expose LLM health to your load balancer and uptime monitor via a dedicated endpoint:

# app/checks/llm_health_check.rb
class LlmHealthCheck
  def self.call
    recent = LlmTrace.where(created_at: 10.minutes.ago..)
    total  = recent.count
    return { status: :ok, message: "no recent traffic" } if total < 5

    error_count = recent.where.not(error_class: nil).count
    error_rate  = error_count.to_f / total
    p95         = recent.pluck(:duration_ms).sort
                        .then { |ms| ms[(ms.size * 0.95).ceil - 1].to_i }

    {
      status:     (error_rate > 0.1 || p95 > 5000) ? :degraded : :ok,
      error_rate: error_rate.round(3),
      p95_ms:     p95,
      total:      total
    }
  end
end

Wire it into your routes:

# config/routes.rb
get "/up/llm", to: lambda { |_env|
  health = LlmHealthCheck.call
  status = health[:status] == :ok ? 200 : 503
  [status, { "Content-Type" => "application/json" }, [health.to_json]]
}

Pingdom or BetterUptime on /up/llm gives you an AI-specific alert without touching your primary /up health check.

Per-Tenant Debugging

On multi-tenant apps, the tenant_id on every trace means you can answer “did this specific customer’s AI features degrade?” without sifting through global telemetry. In Langfuse, filter by the metadata tenant_id field. Against your llm_traces table:

# Average latency for tenant 847 vs the global median over the past 7 days
tenant_p50 = LlmTrace
  .where(tenant_id: 847, created_at: 7.days.ago..)
  .pluck(:duration_ms)
  .sort
  .then { |ms| ms[ms.size / 2] }

global_p50 = LlmTrace
  .where(created_at: 7.days.ago..)
  .pluck(:duration_ms)
  .sort
  .then { |ms| ms[ms.size / 2] }

When a customer reports “the AI has been slow for us today,” you run this query, confirm that their p50 is 3x the global median, and now you have a fact instead of a hypothesis. From there, you filter their traces by prompt version and model to isolate where the regression started. Without Rails LLM observability, that investigation starts with “let me look at all traces and see if I can find a pattern,” which costs an hour of wall time before you even have a theory.

The per-tenant cost aggregation from these traces feeds naturally into the billing and quota system I built in the LLM cost tracking post.

The Thing That Always Trips People Up

The most common mistake I see when teams add Rails LLM observability is treating it as a day-one concern rather than a fire-after-the-fact concern. They instrument LLM calls after the first production incident, when the trail is already cold.

Wire the instrumentation before you ship the first AI feature. Add the llm_traces table and the subscriber in the same pull request that adds the first LLM call. The cost is two files and one migration. The payoff is that when your CEO asks on a Tuesday afternoon why the AI feature is slow, you have seven days of trace data behind you instead of zero.

The 4.2-second summarizer regression I opened with took me forty minutes to diagnose with proper traces. The root cause was a third-party OCR service prepending two pages of metadata to extracted text before passing it to the summarizer — the input token count had quietly tripled. I found it in four queries. Without the traces, I would have started by reading the Anthropic status page, then blaming the model, then eventually staring at the OCR output and noticing the metadata by accident. Two days of debugging versus forty minutes. That is what Rails LLM observability is for.

FAQ

What is Rails LLM observability and why is it different from standard Rails monitoring?

Rails LLM observability is instrumentation specifically for your LLM call layer — which operations ran, which models processed them, how many tokens they consumed, how long each call took, and what failed. Standard Rails monitoring (OpenTelemetry, Datadog APM) sees LLM calls as opaque HTTP requests: a 200-OK in 3 seconds. LLM observability sees “the document.summarize operation on claude-sonnet-5 with prompt v14 processed 9,200 input tokens in 3.2 seconds at a cost of $0.046.” The actionable difference is significant when debugging a latency regression or cost spike.

Why Langfuse and not Datadog or Honeycomb for LLM tracing?

Langfuse speaks LLM natively — its data model has first-class concepts for traces, generations, prompt versions, and evaluation scores. Datadog and Honeycomb are excellent generic observability platforms but you would have to build LLM-specific dashboard primitives yourself. Langfuse is also open source and self-hostable, which matters for teams with data residency requirements. The ActiveSupport::Notifications instrumentation layer in this post is backend-agnostic — swapping LLM::LangfuseSubscriber for a Honeycomb or Datadog subscriber takes an afternoon.

How do I track prompt version changes automatically?

Hash the prompt content with Digest::SHA1.hexdigest(prompt_string)[0, 8]. When the prompt changes, the hash changes, the new version tag appears in traces automatically, and you can compare metrics across versions without updating a string constant. The tradeoff is that version strings are not human-readable (a3f2b1c0 vs v14), but they cannot go stale — the hash is always derived from the actual content in use.

Should I instrument LLM calls in development and staging too?

Yes, every environment. Development traces catch latency regressions before they reach production, and you can use cost data from development to estimate production spend before shipping a feature. Point LANGFUSE_HOST at a local Langfuse instance, or write to a separate llm_traces_development table. Keep development and production data in separate projects in Langfuse to avoid mixing noise into your production dashboards.

Shipping AI features into production without observability is the same as deploying Rails without logs. After nineteen years of production Rails and three years of integrating LLMs into production applications, Rails LLM observability is the first thing I wire up before any AI feature goes live. TTB Software helps teams build observable, production-grade AI features in Rails — the kind that do not degrade silently for two weeks before a customer notices. If your AI layer is a black box right now, we can fix that.