Rails StackProf: Flamegraph CPU Profiling for Ruby Production Applications

The /reports endpoint took eleven seconds. Sentry knew this, in the way Sentry knows things — a broad paintbrush, no detail. ActiveRecord: 9.2 seconds. That was the entire entry. No method names, no line numbers, just a single swipe of orange labelled “database.”

rack-mini-profiler confirmed the query count: 47 queries in that one request. Something was iterating and hitting the database inside the loop. But where? The action touched forty models across a deeply nested call graph. I had no idea which of those 47 queries was the problem — or whether the problem was the queries at all, or something in the Ruby code between them.

This is the situation rack-mini-profiler was not designed for. It is brilliant at showing you which layer of Rails is slow — ActiveRecord versus views versus middleware. It is not brilliant at showing you which Ruby code inside that layer is responsible. For that, you need a sampling profiler that watches the Ruby call stack itself.

StackProf is that tool.

What Rails StackProf Is

StackProf is a sampling profiler for Ruby, written by Koichi Sasada — ko1, the Ruby core developer who also built Fiber and contributed significantly to the YARV VM. It works by interrupting the Ruby VM at regular intervals and recording the current call stack. After a profiling run, you have a statistical picture: frames that appear in 40% of samples were active for 40% of the time. The resolution depends on the sample rate, but even at conservative intervals you get enough signal to identify bottlenecks within seconds.

The sampling approach is different from instrumentation. rack-mini-profiler wraps specific hooks — query execution, controller dispatch, view rendering — and measures elapsed time around those hooks. StackProf sees everything: your code, Rails internals, gems, C extensions where they surface. If 30% of your wall time is spent deserializing a model attribute you did not know was serialized on every read, StackProf shows it. rack-mini-profiler attributes that time to “ActiveRecord” and moves on.

Overhead is low. At the default 1ms sample interval, expect 1–3% CPU overhead. At 10ms, under 0.5%. This is safe to run in production.

Installing StackProf

Add it to your Gemfile without a group restriction, because you will use it in production:

gem "stackprof"

Run bundle install. No configuration file required.

Profiling in the Rails Console

The fastest way to profile a specific operation is in the Rails console, against real data:

data = StackProf.run(mode: :wall, raw: true) do
  Report.generate_monthly(User.first, Date.today)
end

StackProf::Report.new(data).print_text(limit: 20)

raw: true stores the full per-frame data needed for flamegraph output. Without it you can only read the text summary — no flamegraph.

print_text outputs something like this:

 total    self
  47.2%  47.2%  ActiveRecord::ConnectionAdapters::PostgreSQL::OID::Integer#deserialize
  12.8%   0.0%  ActiveRecord::Result#initialize_copy
  11.4%   0.0%  Array#each
   8.9%   8.9%  PG::TypeMap#fit_to_copy_get
   4.1%   4.1%  Report#format_row

The self column shows frames where the sample actually fired — where the process was at that instant. The total column includes time in all frames that method called. A method with high self time is where your CPU stops. A method with high total but low self is an aggregator — it calls expensive code downstream.

To save the data for flamegraph generation:

data = StackProf.run(mode: :wall, raw: true) do
  Report.generate_monthly(User.first, Date.today)
end

File.binwrite(Rails.root.join("tmp/report_profile.dump"), Marshal.dump(data))

Then from your terminal:

# Text summary
bundle exec stackprof tmp/report_profile.dump --text --limit 20

# Interactive HTML flamegraph
bundle exec stackprof tmp/report_profile.dump --d3-flamegraph > tmp/flamegraph.html
open tmp/flamegraph.html

Three Profiling Modes

Choosing the right mode changes what you see.

:wall mode

Samples on a real-time clock regardless of what the thread is doing. Captures time blocked on IO, waiting for Postgres, sleeping in a mutex. This is the mode I reach for first on a slow Rails request, because most Rails latency is not CPU — it is waiting for the database, waiting for external HTTP calls, waiting for Redis.

StackProf.run(mode: :wall, interval: 1000, raw: true) { ... }

interval is in microseconds. 1000 is 1ms, meaning 1000 samples per second. For production, 10000 (100 samples/second) is conservative and nearly free. The reports endpoint I profiled: :wall mode showed hundreds of samples in PG::Connection#exec_params and, crucially, exactly which Ruby frames were calling it in a loop.

:cpu mode

Fires samples only when the Ruby process is scheduled on CPU. IO wait, sleep, and time blocked on external calls are invisible. Use this when you suspect a CPU-bound bottleneck: string processing, JSON encoding, complex calculations, heavy regular expressions. If your request is slow but :cpu shows almost no samples, your bottleneck is IO, not computation.

StackProf.run(mode: :cpu, interval: 1000, raw: true) { ... }

:object mode

Counts object allocations instead of time. Samples fire on every Nth allocation rather than on a timer. Use this when GC tuning tells you your major GC frequency is high and you want to find which code is responsible.

StackProf.run(mode: :object, interval: 100, raw: true) { ... }

interval: 100 fires on every 100th allocation. The flamegraph then shows allocation count per frame rather than time. Hash#merge at 30% self means your code creates a new hash on every merge call — switch to merge! and watch GC pressure drop.

Reading a Flamegraph

Open the generated HTML or upload the dump to Speedscope (browser-based, nothing uploaded). Three rules:

Width equals time. A wide frame spent more time there than a narrow frame. Ignore depth when scanning for problems. Look for wide frames near the top of the visible stack — those are your bottlenecks.

Stack grows upward. The bottom frame started everything: Puma::Server#handle_request or your job’s perform method. Frames above it are what it called. The topmost frame in any column is where the sample fired.

Position is not time order. Flamegraphs sort frames alphabetically by default. Two side-by-side frames did not necessarily run consecutively.

The most useful pattern to scan for: a wide frame near the top with one or two caller frames directly below it that are almost equally wide. That means a single code path dominates the time, and following the chain downward shows you where in your codebase it originates.

In the reports endpoint case, the flamegraph showed ActiveRecord::Associations::CollectionProxy#find at 41% total wall time, called from Reportable#format_row, called from Array#each, called from Report#build_rows. The N+1 was there in the flamegraph, obvious and measured. The fix — moving the association load outside the loop — brought the endpoint from eleven seconds to 900ms.

Production Rack Middleware

The most useful thing you can do with StackProf is run it as production middleware that profiles a small percentage of real traffic and writes dump files to disk. This catches slow paths you cannot reproduce locally — heavy data, unusual parameter combinations, concurrent load effects.

# lib/stackprof_middleware.rb

class StackProfMiddleware
  SAMPLE_RATE = ENV.fetch("STACKPROF_SAMPLE_RATE", "0.005").to_f
  INTERVAL_US = ENV.fetch("STACKPROF_INTERVAL_US", "5000").to_i

  def initialize(app, mode: :wall)
    @app  = app
    @mode = mode
  end

  def call(env)
    return @app.call(env) unless rand < SAMPLE_RATE

    status = headers = body = nil
    data = StackProf.run(mode: @mode, interval: INTERVAL_US, raw: true) do
      status, headers, body = @app.call(env)
    end

    save(data, env)
    [status, headers, body]
  end

  private

  def save(data, env)
    return if data[:samples] < 3

    route = env
      .fetch("PATH_INFO", "unknown")
      .gsub(/[^a-z0-9\-_\/]/, "_")
      .delete_prefix("/")
      .tr("/", "_")[0, 40]

    ts   = Time.now.strftime("%Y%m%d_%H%M%S")
    name = "#{ts}-#{route}-#{SecureRandom.hex(4)}.dump"
    path = Rails.root.join("tmp", "stackprof", name)

    FileUtils.mkdir_p(path.dirname)
    File.binwrite(path, Marshal.dump(data))
  rescue => e
    Rails.logger.warn("StackProfMiddleware: #{e.message}")
  end
end

Register it in config/application.rb:

require_relative "../lib/stackprof_middleware"
config.middleware.use StackProfMiddleware

With STACKPROF_SAMPLE_RATE=0.005 (the default), one in 200 requests gets profiled. On a 10 req/s app that is one profile every 20 seconds — manageable. On a 1000 req/s app, five profiles per second — too many dumps. Tune the rate down to 0.0005 for high-traffic apps.

The environment variable lets you crank up sampling during an incident without a redeploy: heroku config:set STACKPROF_SAMPLE_RATE=0.1 or the equivalent for your platform.

On a Kamal deployment (see the Rails 8 Kamal guide), pull dumps from the running container:

kamal app exec --interactive -- ls tmp/stackprof
kamal app exec -- cat tmp/stackprof/20261020_143022-reports-a3f1.dump > local.dump
bundle exec stackprof local.dump --d3-flamegraph > flamegraph.html

Analyzing with Speedscope

The built-in --d3-flamegraph HTML viewer works but Speedscope has better ergonomics for long sessions. Convert a dump to Speedscope format:

bundle exec stackprof --speedscope tmp/local.dump > tmp/profile.json

Open speedscope.app and load the JSON. Everything stays local — nothing is uploaded. Three views:

Left Heavy merges identical call stacks and sorts by total time. This is the flamegraph view. Use it first.

Time Order preserves chronological order, left to right. Useful when a bottleneck appears episodically — a slow operation that fires three times during a request shows up as three separate flame columns.

Sandwich gives you the text summary as a sortable table with self and total columns. I filter to my app namespace (app/) to focus on code I control. This view alone is often enough to identify the problem frame.

Profiling Background Jobs

Background jobs are frequently the worst-performing code in a Rails app. They run asynchronously, slow jobs fail silently, and performance only surfaces when queues back up. Adding StackProf to a background job is the same pattern:

class MonthlyReportJob < ApplicationJob
  PROFILE_RATE = ENV.fetch("JOB_STACKPROF_RATE", "0.02").to_f

  def perform(account_id, period)
    account = Account.find(account_id)

    if rand < PROFILE_RATE
      profiled_run(account, period)
    else
      generate(account, period)
    end
  end

  private

  def profiled_run(account, period)
    data = StackProf.run(mode: :wall, interval: 1000, raw: true) do
      generate(account, period)
    end

    name = "job-#{self.class.name.underscore}-#{SecureRandom.hex(6)}.dump"
    path = Rails.root.join("tmp", "stackprof", name)
    FileUtils.mkdir_p(path.dirname)
    File.binwrite(path, Marshal.dump(data))
  end

  def generate(account, period)
    Report.generate(account, period)
  end
end

To profile a specific slow job, temporarily set JOB_STACKPROF_RATE=1.0 in your production environment. Every run of that job writes a dump. Pull a dozen dumps, run them through the flamegraph viewer, and the bottleneck will be obvious.

Object Allocation Profiling

When your GC major collection count is high but you cannot find the source with time-based profiling, switch to :object mode:

data = StackProf.run(mode: :object, interval: 10, raw: true) do
  Report.generate(account, period)
end
StackProf::Report.new(data).print_text(limit: 15)

interval: 10 fires on every 10th allocation. A typical output:

 total    self
  41.2%  41.2%  Hash#merge
  28.7%   0.0%  ActiveRecord::Core#init_with_attributes
  14.1%   0.0%  Array#map
   9.3%   9.3%  String#gsub

Hash#merge at 41% self time: somewhere in the hot loop, hash.merge(other_hash) creates a new hash on every call. Replace with hash.merge!(other_hash) or build the hash once outside the loop. ActiveRecord::Core#init_with_attributes at 28% total means the code is instantiating many ActiveRecord objects — consider select to reduce attribute count or pluck when you only need raw values and do not need the model object at all.

Common Flamegraph Patterns

After reading hundreds of Rails flamegraphs, these patterns appear regularly:

Wide read_attribute with many identical callers — a serialized attribute read on every model instantiation. JSON or Marshal deserialization for each attribute access adds up fast. Memoize the deserialized value with an instance variable.

Wide PG::Connection#exec_params called from Array#each — a classic N+1, visible in the flamegraph as a wide blue-green band with the loop frame directly below. See the N+1 post for remediation, but StackProf tells you the exact Ruby method that triggered it.

Thin frames for String#gsub or Regexp#match spread across the flamegraph — regex-heavy text processing scattered through multiple call sites. Compiled constants (PATTERN = /foo/ at the class level, not inside methods) help. Sometimes the bottleneck is not the regex but the String allocation around it.

Wide Marshal.load or JSON.parse — large cache values being deserialized on every cache hit. Consider caching smaller objects, using oj in place of the stdlib JSON, or caching rendered HTML fragments instead of raw data structures.

GC frames sprinkled through the flamegraph — allocation pressure. Switch to :object mode to find the source.

FAQ

How does Rails StackProf compare to rack-mini-profiler?

They answer different questions. rack-mini-profiler answers “which layer of Rails is slow?” — it wraps query execution, view rendering, and controller dispatch, and shows you the breakdown. StackProf answers “which Ruby code inside that layer is responsible?” Use them together: rack-mini-profiler to identify the slow section, StackProf to pin down the specific method. For any Rails request where rack-mini-profiler points to “ActiveRecord” but cannot tell you which code is driving the queries, switch to StackProf in :wall mode.

Is StackProf safe to run in production?

Yes, with a sensible sample rate. Overhead at the default 1ms interval is 1–3%. At 10ms it is under 0.5%. The middleware approach described here profiles only a fraction of requests, keeping aggregate overhead negligible. I run StackProf middleware continuously on several production Rails apps at SAMPLE_RATE=0.005.

What is the difference between :cpu and :wall mode in StackProf?

:cpu mode fires only when the thread is scheduled on CPU. Time blocked waiting for Postgres, Redis, or external HTTP is invisible. :wall mode fires on a real-time clock regardless of what the thread is doing — it captures all latency sources. For a typical Rails web request where most time is IO wait, :wall shows the full picture. For pure computation — encoding, hashing, building data structures — :cpu is more precise because it ignores IO noise.

Why are some Ruby methods missing from my flamegraph?

Two reasons. First, C extensions: StackProf records Ruby frames. Array#sort appears as Array#sort, not the C comparison function executing inside it. Second, YJIT inlining: Ruby 3.3+ and YJIT inline short methods, which can cause them to appear merged with their caller’s frame or disappear entirely. If a critical method is invisible and you are running YJIT, disable it for the profiling session with RUBY_YJIT_ENABLE=0 and compare the flamegraphs.

Chasing slow Rails endpoints with no clear picture of where the bottleneck is? TTB Software profiles and optimizes production Rails applications. We have been doing this for nineteen years.