Rails Idempotency Keys: Safe API Retries, Stripe-Style, for Payments and Webhooks

A client shipped a mobile app last spring that charged customers twice on flaky airport Wi-Fi. Not sometimes — reliably, whenever the network hiccuped between the app tapping “Pay” and the Rails API’s 201 Created making it back to the phone. The app retried, the API happily created a second Charge row, Stripe cheerfully authorized a second capture, and support fielded twenty duplicate-refund tickets a week. The fix was a single HTTP header, a small Postgres table, and about eighty lines of Ruby. The name for that header, and the pattern behind it, is Rails idempotency keys.

After nineteen years of shipping Rails APIs I have watched teams reinvent this wheel three or four times each, usually badly, usually after their first double-charge incident. This post is the exact Rails idempotency keys implementation I install: how to design the header contract, how to store keys in Postgres without race conditions, how to replay the same response byte-for-byte on retry, how long to keep keys around, and the specific bugs that will bite you if you get it wrong.

What Rails Idempotency Keys Actually Guarantee

An idempotency key is a client-generated identifier that a caller sends with a non-idempotent request — usually POST — so the server can recognize a retry of the same operation and return the original response instead of doing the work twice. Stripe popularized the pattern with the Idempotency-Key header, and it has become the de facto standard for any API where duplicate execution costs money, sends a physical package, or triggers a side effect the client cannot easily undo.

The contract, precisely stated, is this: for a given (api_key, request_path, idempotency_key) triple within some retention window, the server MUST return exactly the same HTTP status code, headers, and body as it did the first time — regardless of whether the second request’s payload is identical, different, in flight concurrently, or arrives days later. That is what makes Rails idempotency keys genuinely safe to retry: the caller can hammer the endpoint until it gets a 2xx response and never worry about the server doing the underlying work more than once.

Two things this does NOT give you and I have to remind teams of constantly. First, it does not protect against different clients issuing genuinely different requests that happen to be logically duplicate (“two users hit Buy from the same shared login within the same second”) — that is a business-uniqueness problem, not an idempotency problem, and you solve it with a unique index. Second, it does not deduplicate across payload changes if you decide to reject mismatched payloads; the caller who retries a POST /charges with a different amount under the same key needs a clear error, not silent last-write-wins. I will show both.

The HTTP Contract: Header, Scope, and Response Shape

Pick the header name Stripe picked and do not get creative. Every serious API SDK — theirs, Square’s, PayPal’s, GitHub’s — expects Idempotency-Key, and every client library retry helper writes it automatically. Use anything else and you force your customers to hand-roll retry logic.

# The client sends:
#   POST /api/v1/charges
#   Idempotency-Key: 8b3f4c19-1e6e-4c2c-9c1b-4a5c9c9b3f1e
#   Content-Type: application/json
#
#   { "amount": 4999, "currency": "eur", "customer_id": "cus_123" }
#
# On the FIRST request we run the charge and store the response.
# On any RETRY with the same key we replay the stored response.

Scope keys per API credential, not globally. If tenant A generates key abc and tenant B independently generates key abc, the second request must not see the first’s cached response — that is a data-leak bug waiting to happen. The primary key of your idempotency table is always the composite (api_key_id, key), never just key.

Constrain the key format on ingress: 8–255 characters, printable ASCII, and reject anything that does not match. UUIDv4 is the sensible default and what every SDK generates. A missing Idempotency-Key on a non-idempotent endpoint is a per-API-choice; my default is to require it on any endpoint that spends money, sends an email, or creates an external resource, and to reject with 400 Bad Request if absent. Optional-keys endpoints are a source of “why did this replay?” confusion — pick a lane.

The Postgres Table That Backs It

The storage model is where teams get creative and wrong. I have seen Redis-only implementations that lost keys during a failover and re-executed a batch of payments, and I have seen keys stored on the request model itself with no UNIQUE index and a helpful find_or_create_by race. Do it in Postgres with a real unique constraint, and use a separate table so you can prune independently.

# db/migrate/20260820120000_create_idempotency_keys.rb
class CreateIdempotencyKeys < ActiveRecord::Migration[8.0]
  def change
    create_table :idempotency_keys do |t|
      t.references :api_credential, null: false, foreign_key: true
      t.string     :key,             null: false, limit: 255
      t.string     :request_path,    null: false, limit: 500
      t.string     :request_method,  null: false, limit: 10
      t.string     :request_fingerprint, null: false, limit: 64
      t.integer    :response_status
      t.jsonb      :response_headers
      t.text       :response_body
      t.string     :status, null: false, default: "in_progress", limit: 20
      t.datetime   :locked_at
      t.datetime   :expires_at, null: false
      t.timestamps
    end

    add_index :idempotency_keys,
              [:api_credential_id, :key],
              unique: true,
              name: "idx_idem_keys_scoped_unique"
    add_index :idempotency_keys, :expires_at
  end
end

request_fingerprint is a SHA-256 of the canonical request body (or the sorted query string for GETs you decide to protect). It lets you detect the “same key, different payload” case explicitly and return a 409 Conflict instead of quietly replaying a response for a different request. status distinguishes in_progress from completed — a caller who retries while the first request is still executing must not race and run the work twice. Related: my post on Postgres advisory locks covers the same class of race condition from the cron side.

Middleware That Wraps the Whole Response

The cleanest place to enforce Rails idempotency keys is a Rack middleware that sits after authentication and before your controllers. That way every POST endpoint inherits the behavior for free and you cannot forget to opt one in.

# app/middleware/idempotency_middleware.rb
class IdempotencyMiddleware
  IDEMPOTENT_METHODS = %w[POST PATCH].freeze
  PROTECTED_PATHS    = %r{\A/api/v1/(charges|refunds|subscriptions|transfers)}

  def initialize(app)
    @app = app
  end

  def call(env)
    req = Rack::Request.new(env)
    return @app.call(env) unless applicable?(req)

    key = req.get_header("HTTP_IDEMPOTENCY_KEY")
    return json_error(400, "Idempotency-Key header required") if key.blank?
    return json_error(400, "Invalid Idempotency-Key format") unless valid_key?(key)

    credential = env["api.credential"] # set by auth middleware upstream
    return json_error(401, "Unauthorized") unless credential

    IdempotencyHandler.new(
      app: @app, env: env, req: req, key: key, credential: credential
    ).call
  end

  private

  def applicable?(req)
    IDEMPOTENT_METHODS.include?(req.request_method) && PROTECTED_PATHS.match?(req.path)
  end

  def valid_key?(key)
    key.is_a?(String) && key.bytesize.between?(8, 255) && key.match?(/\A[[:print:]]+\z/)
  end

  def json_error(status, message)
    [status, { "Content-Type" => "application/json" }, [{ error: message }.to_json]]
  end
end

Wire it in config/application.rb after your API authentication middleware. If you have not built one yet, my Rails API authentication post covers the JWT and API-key patterns I use for the upstream credential resolution this middleware depends on.

The Handler: Acquire, Execute, Cache, Replay

The interesting logic lives in IdempotencyHandler. It does four things in a specific order: try to insert an in_progress row (winning the race), execute the wrapped app, persist the response, and on any retry replay the stored response. The trick is that inserting the row happens inside a transaction with ON CONFLICT semantics so exactly one request wins and every other retry sees the winning row.

# app/services/idempotency_handler.rb
class IdempotencyHandler
  LOCK_TIMEOUT = 30.seconds
  RETENTION    = 24.hours

  def initialize(app:, env:, req:, key:, credential:)
    @app, @env, @req, @key, @credential = app, env, req, key, credential
  end

  def call
    fingerprint = compute_fingerprint

    record = acquire_or_find(fingerprint)

    if record.status == "completed"
      return replay(record) if record.request_fingerprint == fingerprint
      return conflict_response
    end

    if record.status == "in_progress" && !stale?(record)
      return in_progress_response
    end

    execute_and_store(record, fingerprint)
  rescue ActiveRecord::RecordNotUnique
    retry
  end

  private

  def acquire_or_find(fingerprint)
    IdempotencyKey.transaction do
      IdempotencyKey.create!(
        api_credential_id: @credential.id,
        key: @key,
        request_path: @req.path,
        request_method: @req.request_method,
        request_fingerprint: fingerprint,
        status: "in_progress",
        locked_at: Time.current,
        expires_at: RETENTION.from_now
      )
    end
  rescue ActiveRecord::RecordNotUnique
    IdempotencyKey.find_by!(api_credential_id: @credential.id, key: @key)
  end

  def execute_and_store(record, fingerprint)
    status, headers, body = @app.call(@env)
    body_str = body.respond_to?(:each) ? body.each.to_a.join : body.to_s

    record.update!(
      response_status: status,
      response_headers: cacheable_headers(headers),
      response_body: body_str,
      request_fingerprint: fingerprint,
      status: "completed",
      locked_at: nil
    )

    [status, headers, [body_str]]
  end

  def replay(record)
    [record.response_status, record.response_headers.merge("Idempotent-Replay" => "true"), [record.response_body]]
  end

  def conflict_response
    body = { error: "Idempotency-Key reused with a different request body" }.to_json
    [409, { "Content-Type" => "application/json" }, [body]]
  end

  def in_progress_response
    body = { error: "A request with this Idempotency-Key is still in progress" }.to_json
    [409, { "Content-Type" => "application/json", "Retry-After" => "2" }, [body]]
  end

  def stale?(record)
    record.locked_at.present? && record.locked_at < LOCK_TIMEOUT.ago
  end

  def compute_fingerprint
    body = @req.body.read.to_s
    @req.body.rewind
    Digest::SHA256.hexdigest("#{@req.request_method}:#{@req.path}:#{body}")
  end

  def cacheable_headers(headers)
    headers.reject { |k, _| %w[Set-Cookie Transfer-Encoding].include?(k) }
  end
end

Five details in there earn their place. The ActiveRecord::RecordNotUnique rescue plus retry-then-find pattern is how you handle two concurrent first requests without a row-level lock. The Idempotent-Replay header on the replay makes it debuggable in production — you can grep for it in your logs. Stripping Set-Cookie prevents leaking the first caller’s session cookie to a retrier who is technically the same API credential but a different browser. The stale? check unwedges dead in_progress rows if the original request crashed the worker before it could finish. And the request-body rewind matters because Rack streams and once-read bodies are empty on the second read — I have debugged this exact bug at 2 AM more than I care to remember.

The Race Condition Everybody Forgets

The bug pattern I see most often is this: developer writes find_or_create_by(key: key) in a controller, tests it locally, and ships. Then production hits a concurrent double-tap — mobile app retries at 2000ms while the first request lands at 2001ms — and both requests run the underlying work because find_or_create_by is a SELECT followed by an INSERT with no unique index enforcement between them.

The middleware pattern above dodges this because the create! with the unique index fails atomically for the loser, and the loser’s rescue clause finds the winner’s row. But if you are tempted to shortcut it, read the actual generated SQL and convince yourself the race is closed. INSERT ... ON CONFLICT DO NOTHING RETURNING * with a follow-up SELECT for the null-return case is another safe pattern; find_or_create_by without the transaction dance is not.

For the specifically dangerous case of long-running work — a Stripe charge that takes 800ms to authorize — the in_progress state buys you correctness. A retry within the window returns 409 with Retry-After: 2, the client waits and retries, and by then the original response is cached and gets replayed. This is what Stripe’s own API does: their docs are explicit that a retry while the first is executing returns a specific error, and their SDKs handle it by backing off and retrying with the same key.

Payload-Change Detection and Why It Matters

If a client sends the same Idempotency-Key with a different body, you have a choice: replay the original response (Stripe’s behavior for their Idempotency-Key) or reject with 409 Conflict. My default is reject, because silent replay masks client bugs that would otherwise show up in QA — imagine a developer thinking they are creating a $100 refund and always getting a $50 one back because they forgot to rotate the key from a previous test.

The request_fingerprint column above is what makes the check cheap. When the middleware finds an existing completed row, it compares the current request’s fingerprint against the stored one and returns 409 on mismatch with a body explaining exactly why. This is exceptionally useful during integration development — every developer will hit it at least once, and the error message tells them “you reused a key” rather than “why did my update not take?”

Fingerprint over the canonical body, not the raw one. {"a":1,"b":2} and {"b":2,"a":1} should hash the same, so either sort keys before hashing or agree with your callers to use a canonical JSON form. For most SDK-driven APIs the difference does not come up, but it will bite you the first time a customer writes their own client in Go.

Retention, Pruning, and Cost

Idempotency keys are not free storage — every POST to a protected endpoint costs one row plus the size of its response body. My default retention is 24 hours, which comfortably covers every real retry scenario (mobile client backgrounded, network partition, deploy-driven timeout) without keeping years of dead data.

Prune with a scheduled job:

# app/jobs/prune_idempotency_keys_job.rb
class PruneIdempotencyKeysJob < ApplicationJob
  queue_as :low_priority

  def perform
    IdempotencyKey.where("expires_at < ?", Time.current).in_batches(of: 5_000).delete_all
  end
end

Schedule it hourly. If you are on Rails 8’s Solid Queue, my post on Solid Queue recurring jobs has the cron syntax; if you are on Sidekiq, use sidekiq-cron. Ballpark: a service doing 1 million protected POSTs per day at ~2 KB average response body accumulates ~2 GB per day of idempotency rows before pruning, ~2 GB steady-state at 24-hour retention. That is a rounding error on any real Postgres, but worth knowing before you turn it on.

Compression is optional and rarely worth it — response bodies for financial API calls are typically small JSON objects, and the JSONB overhead of storing response_headers structurally is already efficient. If your responses are large (paginated lists, embedded documents), store a URL to blob storage rather than inlining megabytes into Postgres.

Testing Idempotency Keys Without Flakes

Two categories of tests are worth writing and I see teams skip both. First, the happy-path replay: same key, same body, verify the same response and the Idempotent-Replay: true header. Second, the race: fire two concurrent requests with the same key and prove exactly one executed.

# spec/requests/idempotency_spec.rb
require "rails_helper"

RSpec.describe "Idempotency keys", type: :request do
  let(:credential) { create(:api_credential) }
  let(:headers) do
    {
      "Authorization"    => "Bearer #{credential.token}",
      "Idempotency-Key"  => SecureRandom.uuid,
      "Content-Type"     => "application/json"
    }
  end
  let(:payload) { { amount: 4999, currency: "eur", customer_id: "cus_test" }.to_json }

  it "replays the first response on retry" do
    post "/api/v1/charges", params: payload, headers: headers
    first_status, first_body = response.status, response.body
    expect(first_status).to eq(201)

    post "/api/v1/charges", params: payload, headers: headers
    expect(response.status).to eq(first_status)
    expect(response.body).to eq(first_body)
    expect(response.headers["Idempotent-Replay"]).to eq("true")
    expect(Charge.count).to eq(1)
  end

  it "rejects the same key with a different payload" do
    post "/api/v1/charges", params: payload, headers: headers
    post "/api/v1/charges", params: { amount: 9999, currency: "eur", customer_id: "cus_test" }.to_json, headers: headers
    expect(response.status).to eq(409)
    expect(Charge.count).to eq(1)
  end

  it "runs the work exactly once under concurrent requests" do
    threads = 5.times.map do
      Thread.new { post "/api/v1/charges", params: payload, headers: headers }
    end
    threads.each(&:join)
    expect(Charge.count).to eq(1)
  end
end

The concurrent test is the one that catches implementation bugs — if you drop the unique index, or replace create! with find_or_create_by, this test flakes on the first CI run and stays flaky. For deeper end-to-end verification see Rails system tests with Capybara, though for pure API idempotency the request-spec level is exactly the right altitude.

Observability: Log the Replays

Every replay is either a benign client retry or a bug in your client SDK, and you cannot tell which without logging. Emit a structured log line on every replay and every 409, and put both on a Grafana panel next to your API error rates.

# In IdempotencyHandler#replay
Rails.logger.info(
  event: "idempotency.replay",
  key: @key,
  credential_id: @credential.id,
  path: @req.path,
  original_status: record.response_status,
  age_seconds: (Time.current - record.created_at).to_i
)

The age_seconds field is the interesting one. Retries within 5 seconds are network hiccups; retries at 30+ seconds are usually a client SDK with an over-aggressive retry loop or a badly-configured circuit breaker. Both are worth knowing about — one wants a client fix, one wants an alert.

Common Mistakes I Still See in Production

A short list of the failure modes I have debugged more than once:

• No unique index on (api_credential_id, key). find_or_create_by looks correct and is not; concurrent requests both execute the underlying work. Always let Postgres enforce uniqueness.
• Caching the wrong headers. Set-Cookie, Transfer-Encoding, and any Authorization: Bearer echoes must be stripped from the stored response or you leak state between callers.
• No in_progress handling. A retry that arrives while the first is still running races and executes the work twice. Store status and return 409 Retry-After while the first is in flight.
• Storing keys in Redis without persistence. A Redis failover between the write and the retry loses the key and re-executes. Postgres is the right home for anything that guards money movement.
• Global key scope instead of per-credential. Client A’s key collides with client B’s key and one gets the other’s response. Composite index, always.
• Not rewinding the request body. Rack input streams are single-read; fingerprint computation empties the body and your controller sees an empty payload. req.body.rewind after any read.

Frequently Asked Questions

How is a Rails idempotency key different from a database unique constraint?

They solve different layers. A unique constraint (say, on order.external_reference) prevents duplicate rows for a business identifier the client already generated for its own reasons. Rails idempotency keys protect the HTTP boundary against transport-level retries — the same request being sent twice by a network layer that does not know the first one succeeded. You often want both: the idempotency key stops the second POST from executing at all, and the unique index is the belt-and-braces backup if you ever accidentally disable the middleware.

Should I put Rails idempotency keys on GET endpoints?

No. GET is already idempotent by definition, and adding a key would only add storage cost with no correctness benefit. Reserve Rails idempotency keys for POST, PATCH, and occasionally DELETE on endpoints that trigger side effects. If a GET is genuinely expensive (LLM calls, report generation), reach for HTTP caching with ETag — see Rails HTTP caching — not for idempotency keys.

Can I use Redis instead of Postgres for Rails idempotency keys?

You can, but I would not for anything guarding money movement. Redis lacks Postgres’ durability guarantees during failover; a well-timed primary swap can lose keys that were written and acknowledged, causing re-execution of a payment. Redis is fine for shorter-lived deduplication of low-stakes endpoints (analytics ingestion, telemetry) where the cost of an occasional duplicate is a rounding error.

What retention window should I use for Rails idempotency keys?

24 hours is my default and covers essentially every realistic retry scenario without ballooning storage. Stripe uses 24 hours. Shorter (say, 15 minutes) fails to protect against mobile-app-backgrounded-for-a-while retries; longer (7 days) balloons storage without protecting anything real — nobody retries a POST seven days later thinking it might have failed. Start at 24h, tune with data.

Need help designing an API that customers can actually retry safely, or auditing an existing one for silent double-execution bugs? TTB Software has been shipping production Rails systems for nineteen years, and idempotency keys are on the checklist for every payment or webhook-generating endpoint we ship.