Rails Vernier Profiler: Ruby 3.3 Sampling Profiler for Production Flamegraphs and Slow Requests

A fintech client rang me on a Wednesday because their /dashboard endpoint had drifted from a healthy 180ms P95 to a groggy 1.4s over six months and nobody could tell them why. They had New Relic, they had rack-mini-profiler in staging, they had a Datadog APM trace that shrugged and said “Ruby”. What they did not have was any picture of what the Ruby VM was actually doing inside those 1.4 seconds. I asked them to add one gem — vernier — capture a single production request with Vernier.trace, drop the JSON into the Firefox Profiler UI, and within four minutes we were staring at a flamegraph that showed 62% of wall time inside a nested each block calling Object#respond_to? on 40,000 hashes. Two-line fix. Rails Vernier profiler turned an invisible slowdown into an obvious bug.

After nineteen years of Rails I have used every profiler that made it through a hype cycle: ruby-prof, perftools.rb, stackprof, rblineprof, memory_profiler, rack-mini-profiler. Since Ruby 3.3 landed there is a new default and I have already migrated all my client apps to it. Rails Vernier profiler is a sampling profiler written for the post-3.3 Ruby VM, it understands YJIT, it captures GVL and GC events natively, and it produces flamegraphs you actually read instead of squint at. This post is how I install it, how I capture production requests without blowing up the process, and how I read the output.

What Rails Vernier Profiler Actually Is

Vernier is a Ruby sampling profiler by John Hawthorn (of Shopify) that ships as a gem and requires Ruby 3.2.1+ for basic use, with the good stuff — real stack sampling via the new rb_profile_frames hooks — landing in Ruby 3.3. Unlike stackprof, which was built for the pre-YJIT Ruby VM and gets confused by JIT-compiled frames, Rails Vernier profiler knows about YJIT, ZJIT, native extensions, thread scheduling, and GVL contention. Unlike ruby-prof, it does not instrument every method call, so its overhead is low enough to run in production on real traffic.

Practically, Vernier does three things stackprof cannot:

It captures multiple threads. A Puma worker handling one request while another thread runs an ActionCable connection is a common shape, and stackprof only sees the calling thread. Vernier records every Ruby thread and every native thread, so you can see GC threads, Puma reactor threads, and background threads on the same timeline.

It captures GVL events. When a thread is waiting to acquire the Global VM Lock — usually because a native extension is blocking, or because another thread is holding the lock during a long block — Vernier records the wait as a distinct event. If your slow request is actually spending 300ms blocked on the GVL because a background job is hogging it, Vernier shows this directly. Stackprof cannot.

It captures GC and allocation events. Every major GC, every minor GC, every compaction pause shows up as a marker on the timeline, so you can see whether your slow request coincided with a 400ms GC compaction or whether the app was actually doing work. This matters more in production than in development because production heap sizes are different.

The output format is compatible with the Firefox Profiler, which is the same UI that Mozilla’s C++ engineers stare at when they profile Firefox itself. It is genuinely good — zoomable, filterable, thread-aware, with a flamegraph view and a call-tree view and a marker chart. You upload the JSON at profiler.firefox.com and it runs entirely in your browser; nothing is uploaded to a server.

Installing Rails Vernier Profiler

Add the gem to your Gemfile. I keep it in the default group because I sometimes want it in production, but if you are cautious you can gate it behind a group and require it manually when you need it.

# Gemfile
gem "vernier", "~> 1.5"

Then bundle install. That is the entire setup. Vernier is a pure gem — no Rack middleware to mount, no initializer to write, no configuration file. You call it from Ruby when you want a trace.

Verify the install:

bin/rails runner 'Vernier.trace(out: "/tmp/warmup.vernier.json") { 100.times { "x" * 1000 } }'

You should get a JSON file in /tmp. Drag it onto profiler.firefox.com and you should see a flamegraph. If the JSON is under a few kilobytes and the flamegraph is empty, your Ruby is probably below 3.3 and Vernier is not sampling frames — upgrade Ruby before going further. Everything about Rails Vernier profiler assumes Ruby 3.3+.

Profiling a Rails Controller Action

The simplest useful thing is to wrap one controller action for a single request. I do this by adding an around-action that only activates for a specific query parameter, so you can trigger a trace on demand without redeploying.

class ApplicationController < ActionController::Base
  around_action :profile_with_vernier, if: -> { params[:vernier].present? && trusted_admin? }

  private

  def profile_with_vernier
    require "vernier"
    path = Rails.root.join("tmp", "vernier-#{Time.now.to_i}-#{SecureRandom.hex(4)}.json")
    Vernier.trace(out: path.to_s) do
      yield
    end
    response.headers["X-Vernier-Path"] = path.to_s
  end

  def trusted_admin?
    Current.user&.admin? && request.remote_ip.in?(TRUSTED_ADMIN_IPS)
  end
end

Hit any page with ?vernier=1 while logged in as an admin, and the response headers tell you where the trace JSON landed on the server. scp it locally, drag into the Firefox Profiler, done. Two things matter about this pattern. First, the trusted_admin? guard — never let a random client trigger a Vernier trace, because profiling a request is not free and someone could DoS you by requesting ?vernier=1 in a loop. Second, the trace path is in tmp/ so it does not pollute your log volume; add a cron or a scheduled job to prune old traces.

If you prefer not to litter the codebase with profiling code, Vernier ships a Rack middleware you can mount from an initializer that lives only in development or behind a feature flag:

# config/initializers/vernier.rb
if ENV["VERNIER_MIDDLEWARE"] == "1"
  require "vernier/middleware"
  Rails.application.config.middleware.use Vernier::Middleware
end

Boot the app with VERNIER_MIDDLEWARE=1 bin/rails s, hit any request with ?vernier=true, and the middleware writes a trace and returns the profiler URL as a response header. I prefer the around-action approach for production because it composes with authorization; middleware is fine for staging.

Profiling a Background Job

Half of the slow-request work in a mature Rails app is not in controllers, it is in Sidekiq or Solid Queue jobs that timeout at 15 minutes and nobody knows why. Vernier is perfect for this because you already have the job class, you can wrap perform conditionally, and you have log context to correlate the trace back to a Sidekiq job ID.

class ReportGenerationJob < ApplicationJob
  queue_as :reports

  around_perform do |job, block|
    if ENV["VERNIER_JOBS"] == "1" || job.arguments.first == "profile-me"
      require "vernier"
      path = Rails.root.join("tmp", "vernier-job-#{job.job_id}.json")
      Vernier.trace(out: path.to_s, &block)
      Rails.logger.info(vernier_trace: path.to_s, job_id: job.job_id)
    else
      block.call
    end
  end

  def perform(scope)
    Report.slow_query(scope).each { |row| row.crunch! }
  end
end

Enqueue with a synthetic argument, wait for the job to finish, grep the log for vernier_trace, scp the JSON off the worker, upload to the profiler. On my client’s system this workflow found a job that was spending 47% of its time inside ActiveRecord::Base#reload because someone had put record.reload inside a .each in a helper — a mistake stackprof would show, but Vernier’s marker chart also showed the accompanying 12 minor GCs, which is what convinced them to fix it right away.

Continuous Profiling in Production

For production I use two patterns depending on the risk profile. Both are safe when configured properly and both give you the ability to answer “what did the app spend time on last Tuesday at 3pm” without deploying.

The first pattern is triggered on-demand profiling — the around-action above, gated by admin auth. This is what I use on smaller apps where I do not need continuous coverage.

The second pattern is timed background sampling — a thread that starts and stops Vernier every N minutes for a short window. This is what I use on larger apps where problems reproduce intermittently.

# config/initializers/vernier_background_sampler.rb
if Rails.env.production? && ENV["VERNIER_SAMPLER"] == "1"
  Thread.new do
    Thread.current.name = "vernier-sampler"
    loop do
      begin
        require "vernier"
        window = 15
        path = Rails.root.join("tmp", "vernier-bg-#{Time.now.to_i}.json")
        collector = Vernier::Collector.new(:wall, out: path.to_s)
        collector.start
        sleep window
        collector.stop
        Rails.logger.info(vernier_bg_trace: path.to_s, window_sec: window)
        upload_to_s3(path)
        File.delete(path)
      rescue => e
        Rails.logger.error(vernier_sampler_error: e.message)
      ensure
        sleep 5.minutes.to_i
      end
    end
  end
end

Every five minutes it samples for fifteen seconds, uploads the trace to S3, deletes the local file. The overhead of a wall-clock sample is around 1-2% and only during the sampling window, so you are paying roughly 0.1% overall. When a customer complains that something was slow at 14:37 UTC, you have a trace within a few minutes of that timestamp already sitting in S3.

Two production caveats. First, run Vernier on one worker per host at a time; a whole fleet profiling simultaneously will burn CPU. I usually gate the sampler on Process.pid.even? or on a specific hostname pattern so only a subset of workers profile. Second, the wall mode captures wall-clock time (best for finding blocked IO); retained mode captures allocations retained across GC (best for memory growth). Wall is the right default; switch to retained when you suspect a leak.

Reading a Vernier Flamegraph

The Firefox Profiler UI has three panels I use in order. The timeline at the top shows CPU usage, GVL activity, and GC markers across the trace. The flamegraph in the middle shows a stack sample distribution: wider = more samples spent there. The call tree on the right lets you sort by total time or self time.

The heuristic I teach clients is a two-pass read.

First pass — top of the flamegraph. Look for the widest boxes near the top of the stack. Those are the leaf functions where samples land most often. If you see ActiveRecord::Base#reload occupying 40% of the width, that is where your time is going. If you see #[] from a hash occupying 25%, someone is looking up hash keys a lot — probably an N+1 in disguise, or a badly memoized helper.

Second pass — GVL and GC markers. In the timeline, look for red or orange bars indicating GVL waits and GC pauses. If the GC bar is 30% of the timeline, your problem is allocation pressure, not slow code. Fix it by finding the widest allocation boxes in the flamegraph (Vernier tags allocation events on the retained mode) and reducing them — often a .to_a inside a loop, a .dup that should be a reference, or a String#+ that should be String#<<.

The mistake beginners make with flamegraphs is reading the bottom (the widest boxes at the base, usually Rack::Server#call or ActionDispatch::Executor#call) as the “slow” thing. It is not — the base of the flamegraph is just the entry point, and every request goes through it. The interesting information is at the top, where the samples actually land.

Vernier vs Stackprof vs Rack Mini Profiler

These three tools sound similar and they are not. I use all three for different jobs, but the overlap is smaller than newcomers think.

Rack mini profiler is a Rack middleware that shows you a per-request breakdown of SQL queries, view rendering time, and total request time as a toolbar in the browser. It is a workflow tool for finding “which endpoint is slow” and “how many SQL queries did this page run”. It does not sample Ruby stack frames — it hooks into Rails’ notifications. Best for development and staging. I covered it in detail in Rack Mini Profiler.

Stackprof is a sampling profiler by Aman Gupta that predates YJIT. It is stable, small, and still fine for pre-3.3 Ruby. Its output format is proprietary and its viewer is stackprof --text, which is fine for command-line reading but less useful than a flamegraph. It does not see GVL, does not see multiple threads well, and does not understand YJIT frames.

Rails Vernier profiler is the successor: multi-threaded, YJIT-aware, GVL-aware, GC-aware, with a good UI. Use it for anything on Ruby 3.3+. Use stackprof only if you are stuck on older Ruby.

The rough decision tree: “I want to know why this page is slow right now” → rack-mini-profiler. “I want to know what my process is spending time on across an entire request or job” → Vernier. “I want continuous production profiling with a good UI” → Vernier with a background sampler.

Common Traps I Have Hit

Three mistakes I have made or watched clients make with Rails Vernier profiler in the last year.

Sampling a single request that is not slow. A one-second Vernier trace on a fast request shows very few samples and looks empty. Trigger the trace on a request you already know is slow, or use the background sampler to capture longer windows.

Forgetting Ruby 3.3+. On Ruby 3.2 you get partial data — stack frames are captured but GVL and native thread information is not, and the flamegraph looks anaemic. If your production is still on 3.2, upgrade first; Vernier is one of the best reasons to move.

Profiling with tests. Vernier works inside RSpec, but the boot phase dominates the flamegraph if you profile a fast test. Wrap only the code under test, not the whole run. For a full-suite profile use Vernier.trace(mode: :wall, out: "/tmp/spec.json") { RSpec::Core::Runner.run(...) } and look at what happens after the boot.

FAQ

Does Vernier work with YJIT and Ruby 3.3?

Yes — Vernier was designed for the Ruby 3.3 profiler API and understands YJIT-compiled frames. That is one of its main reasons to exist. Older profilers like stackprof see JIT frames as unknown or missing. Rails Vernier profiler shows them correctly labelled, which matters if you have YJIT enabled in production.

Is it safe to run Vernier in production?

Yes, with two caveats. Sampling overhead is 1-3% during the trace window, which is fine for a few-second capture but adds up if you run it continuously on every worker. Gate the sampler to one worker per host, use short windows (15-30 seconds), and keep the sample interval at the default (1ms wall). Do not enable Vernier globally on every request — profile specific slow requests or take timed background samples.

How do I profile memory allocations with Vernier?

Use Vernier.trace(mode: :retained, out: "path.json") { ... } to sample allocations that survive GC. This is what you want for detecting leaks and hot allocation sites. mode: :allocations samples every allocation regardless of retention and is more expensive; use it in development when you need every last allocation.

Where do I view Vernier output?

The generated JSON is compatible with the Firefox Profiler at profiler.firefox.com. Everything runs client-side in your browser; nothing is uploaded to Mozilla. For CI or headless environments, there is also vernier view which writes an HTML file you can serve locally.

Struggling to explain where your Rails process is spending its time? TTB Software does hands-on Rails performance work — Vernier, YJIT, Postgres, background jobs. We have been doing this for nineteen years.