Rails Transactional Outbox Pattern: Reliable Event Publishing Without Dual-Write Failures

Three weeks after going live with a new fulfilment integration, a client called me on a Friday afternoon. Their warehouse had seventeen unshipped orders. The orders were marked status: "fulfilled" in the Rails database. The warehouse management system had no record of any of them.

The difference? A Rails process had died during a rolling deploy window — Kamal was swapping containers, and seven requests were mid-flight. Each one had committed the database transaction successfully, updated the order to fulfilled, then raised a Net::ReadTimeout on the HTTP POST to the warehouse API. No event reached the warehouse. No retry was scheduled. The database and the external system had silently diverged, and neither side had any idea.

This is the dual-write problem. It is not a bug in your code. It is a structural guarantee violation that is invisible until it causes damage, and it happens to everyone who writes to a database and calls an external service in the same request.

The Rails transactional outbox pattern eliminates it.

The Dual-Write Problem, Exactly

Every time you do this, you have a dual-write:

def fulfill_order(order)
  order.update!(status: :fulfilled, fulfilled_at: Time.current)   # Write 1: database
  WarehouseClient.post("/shipments", order.to_shipment_payload)   # Write 2: external system
end

There are four possible outcomes. Three of them are fine. One is catastrophic:

1. Both succeed — fine.
2. DB write fails, HTTP call never happens — fine, consistent state, retry the whole operation.
3. DB write succeeds, HTTP call fails with a clean error — theoretically handleable with a rescue and retry, but you are writing retry logic inside the request path, which delays the response and is difficult to get right under concurrency.
4. DB write succeeds, HTTP call times out or the process dies — you will never know the event was lost, because the exception happened after the commit and the transaction is already gone.

Scenario four is what caused seventeen unshipped orders on that Friday afternoon. The process died. No exception was raised from the caller’s perspective. No dead letter queue. No alert. Just silence.

The instinct is to reverse the order — call the external service first, then commit the database. That does not help. You now get the opposite silent failure: the webhook fires but the database update never commits. You need a fundamentally different approach.

The Transactional Outbox Pattern

The pattern is simple. Instead of calling the external system directly, you write the intention to call it into a database table — the outbox — inside the same transaction as your business data change. A separate background process reads from the outbox table and handles actual delivery. Because the outbox record and the business record are written atomically in one transaction, they either both succeed or both fail. The background processor retries delivery until it succeeds.

The result is at-least-once delivery: an event may be delivered more than once if the relay crashes between marking it delivered and persisting that status. Your consumers should be idempotent. In practice, idempotency is easier to implement on the consumer side than it sounds — usually a unique index on an event ID is all it takes.

Setting Up the Outbox Table

Start with the migration:

# db/migrate/20260728000000_create_outbox_events.rb
class CreateOutboxEvents < ActiveRecord::Migration[8.0]
  def change
    create_table :outbox_events do |t|
      t.string  :aggregate_type,  null: false
      t.bigint  :aggregate_id,    null: false
      t.string  :event_type,      null: false
      t.jsonb   :payload,         null: false, default: {}
      t.string  :idempotency_key, null: false
      t.datetime :published_at
      t.integer  :attempts,       null: false, default: 0
      t.datetime :last_attempted_at
      t.text     :last_error
      t.timestamps
    end

    add_index :outbox_events, :idempotency_key, unique: true
    add_index :outbox_events, :created_at,
              where:  "published_at IS NULL",
              name:   "idx_outbox_events_pending"
  end
end

aggregate_type and aggregate_id identify which record owns this event — Order and 42, for example. event_type is a namespaced string like order.fulfilled. payload is the data the consumer needs. idempotency_key is a unique string per logical event that prevents double-processing in the consumer if the relay delivers it twice. The partial index on created_at WHERE published_at IS NULL keeps the polling query fast even with millions of already-delivered events sitting in the table.

The model:

# app/models/outbox_event.rb
class OutboxEvent < ApplicationRecord
  MAXIMUM_ATTEMPTS = 10

  scope :pending, -> {
    where(published_at: nil)
      .where("attempts < ?", MAXIMUM_ATTEMPTS)
      .order(:created_at)
  }
end

Publishing to the Outbox Atomically

The entire value of this pattern comes from writing to the outbox table inside the same transaction block as your business logic. Rails wraps each save! call in its own implicit transaction, so you need an explicit block to bundle the two writes together:

class Order < ApplicationRecord
  def fulfill!
    transaction do
      update!(status: :fulfilled, fulfilled_at: Time.current)

      OutboxEvent.create!(
        aggregate_type:  "Order",
        aggregate_id:    id,
        event_type:      "order.fulfilled",
        payload:         {
          order_id:    id,
          customer_id: customer_id,
          line_items:  line_items.map(&:to_event_payload),
          fulfilled_at: fulfilled_at.iso8601
        },
        idempotency_key: "order.fulfilled.#{id}"
      )
    end
  end
end

If OutboxEvent.create! raises — a uniqueness violation because the event was already queued, a database constraint — the whole transaction rolls back. If the process dies after update! but before OutboxEvent.create!, Postgres rolls back automatically. Either way: the order is either fulfilled with an outbox record, or neither happened. No silent divergence.

For reuse across multiple models, extract a concern:

# app/models/concerns/outbox_publisher.rb
module OutboxPublisher
  extend ActiveSupport::Concern

  def publish_outbox_event(event_type, payload = {}, idempotency_key: nil)
    OutboxEvent.create!(
      aggregate_type:  self.class.name,
      aggregate_id:    id,
      event_type:      event_type,
      payload:         payload,
      idempotency_key: idempotency_key || "#{self.class.name.underscore}.#{event_type}.#{id}"
    )
  end
end

Include it and call it inside your transactions:

class Subscription < ApplicationRecord
  include OutboxPublisher

  def upgrade!(new_plan)
    transaction do
      old_plan = plan
      update!(plan: new_plan, upgraded_at: Time.current)
      publish_outbox_event("subscription.upgraded", {
        subscription_id: id,
        old_plan:        old_plan,
        new_plan:        new_plan,
        upgraded_at:     upgraded_at.iso8601
      })
    end
  end
end

One design decision worth making explicit: the idempotency_key in the default concern is subscription.upgraded.123, which is unique per subscription. If a subscription can legitimately upgrade multiple times, include a version or timestamp in the key to avoid the uniqueness constraint blocking the second event: "subscription.upgraded.#{id}.#{upgraded_at.to_i}".

The Relay Job

The relay job polls the outbox table, delivers each pending event to its destination, and marks it delivered. It runs every few seconds via Solid Queue’s recurring job scheduler — I covered the full configuration in the Solid Queue recurring jobs post.

# app/jobs/outbox_relay_job.rb
class OutboxRelayJob < ApplicationJob
  queue_as :outbox

  def perform
    OutboxEvent.pending.limit(100).lock("FOR UPDATE SKIP LOCKED").each do |event|
      relay(event)
    end
  end

  private

  def relay(event)
    event.update_columns(
      attempts:          event.attempts + 1,
      last_attempted_at: Time.current
    )

    EventRouter.dispatch(event)

    event.update_column(:published_at, Time.current)
  rescue => e
    event.update_column(:last_error, "#{e.class}: #{e.message}"[0, 2000])
    raise
  end
end

FOR UPDATE SKIP LOCKED is the key to safe parallel processing. It locks the rows being selected and skips any already locked by another transaction. If you run two relay workers simultaneously — for throughput — each picks up a non-overlapping batch without blocking or racing. Without SKIP LOCKED, workers queue up behind each other’s locks, serialising what should be parallel delivery. I covered Postgres locking semantics in detail in the advisory locks post.

Schedule the relay to run frequently via config/recurring.yml:

# config/recurring.yml
outbox_relay:
  class: OutboxRelayJob
  queue: outbox
  schedule: "*/5 * * * * *"   # every 5 seconds (cron with seconds support)

Or inline in config/application.rb if you prefer:

config.solid_queue.recurring_tasks = {
  outbox_relay: {
    class:    "OutboxRelayJob",
    schedule: "*/5 * * * * *"
  }
}

Routing Events to Their Destinations

The relay job delegates to an EventRouter. The router maps event type patterns to delivery adapters, keeping the relay job small and each publisher independently testable:

# app/services/event_router.rb
class EventRouter
  ROUTES = {
    /^order\./        => OrderWebhookPublisher,
    /^subscription\./ => BillingSystemPublisher,
    /^customer\./     => CrmSyncPublisher
  }.freeze

  def self.dispatch(event)
    publisher_class = ROUTES.find { |pattern, _| pattern.match?(event.event_type) }&.last
    raise UnroutableEventError, "No publisher for: #{event.event_type}" unless publisher_class

    publisher_class.new(event).call
  end
end

An HTTP webhook publisher:

# app/publishers/order_webhook_publisher.rb
class OrderWebhookPublisher
  def initialize(event)
    @event = event
  end

  def call
    subscribers_for(@event.event_type).each do |subscriber|
      payload_json = @event.payload.to_json
      response = Faraday.post(
        subscriber.endpoint_url,
        payload_json,
        "Content-Type"      => "application/json",
        "X-Event-Type"      => @event.event_type,
        "X-Idempotency-Key" => @event.idempotency_key,
        "X-Signature"       => sign(payload_json, subscriber.secret)
      )
      raise WebhookDeliveryError, "HTTP #{response.status}" unless response.success?
    end
  end

  private

  def subscribers_for(event_type)
    WebhookSubscriber.active.for_event_type(event_type)
  end

  def sign(payload_json, secret)
    OpenSSL::HMAC.hexdigest("SHA256", secret, payload_json)
  end
end

For internal consumers — Solid Queue jobs, Sidekiq workers — skip HTTP entirely and enqueue directly:

# app/publishers/billing_system_publisher.rb
class BillingSystemPublisher
  def initialize(event)
    @event = event
  end

  def call
    SyncSubscriptionToBillingJob.perform_later(
      subscription_id: @event.aggregate_id,
      event_type:      @event.event_type,
      payload:         @event.payload,
      idempotency_key: @event.idempotency_key
    )
  end
end

This is an important point that often surprises people: the Rails transactional outbox pattern works equally well whether you are calling external HTTP webhooks or dispatching to internal background jobs. The relay is just a dispatcher. The pattern solves the same dual-write problem in both cases.

Handling Failures and Dead Letters

The relay job raises on failure. The ActiveJob framework marks the job failed and the scheduler re-enqueues it. On the next relay run, the outbox record still has published_at: nil with attempts: N and gets picked up again — up to MAXIMUM_ATTEMPTS.

After ten attempts, the pending scope excludes the record. At this point you need visibility. Build a simple health check:

# app/checks/outbox_health_check.rb
class OutboxHealthCheck
  def self.call
    dead_count = OutboxEvent
      .where(published_at: nil)
      .where("attempts >= ?", OutboxEvent::MAXIMUM_ATTEMPTS)
      .count

    {
      status: dead_count.zero? ? :ok : :degraded,
      dead_letter_count: dead_count
    }
  end
end

Wire it into your monitoring. On client engagements I usually drop this into a Sentry cron check or expose it via /up as a named check. For individual stuck records, add a reset method:

class OutboxEvent < ApplicationRecord
  def retry!
    update!(attempts: 0, last_error: nil, last_attempted_at: nil)
  end
end

Call OutboxEvent.find(id).retry! from a Rails console to push a dead letter event back into the retry queue. For batch recovery after fixing a broken consumer:

OutboxEvent
  .where(published_at: nil)
  .where("attempts >= ?", OutboxEvent::MAXIMUM_ATTEMPTS)
  .update_all(attempts: 0, last_error: nil)

Pruning Delivered Events

The outbox table accumulates delivered events indefinitely without cleanup. Add a prune job:

# app/jobs/outbox_prune_job.rb
class OutboxPruneJob < ApplicationJob
  queue_as :maintenance

  def perform
    OutboxEvent
      .where.not(published_at: nil)
      .where("published_at < ?", 30.days.ago)
      .delete_all
  end
end

Use delete_all rather than destroy_all. You do not need callbacks on deletion, and delete_all issues one SQL DELETE instead of N individual round-trips. Schedule it daily alongside the relay. Thirty days gives you a comfortable audit window without unbounded table growth.

What to Test

The outbox pattern splits neatly into two independent concerns, which makes it straightforward to test:

# spec/models/order_spec.rb
RSpec.describe Order do
  describe "#fulfill!" do
    it "creates an outbox event in the same transaction" do
      order = create(:order, :ready_to_fulfill)

      expect { order.fulfill! }.to change(OutboxEvent, :count).by(1)

      event = OutboxEvent.last
      expect(event.event_type).to eq("order.fulfilled")
      expect(event.aggregate_id).to eq(order.id)
      expect(event.payload["order_id"]).to eq(order.id)
    end

    it "rolls back the outbox event if the order update fails" do
      order = create(:order, :ready_to_fulfill)
      allow(order).to receive(:update!).and_raise(ActiveRecord::RecordInvalid.new(order))

      expect { order.fulfill! }.to raise_error(ActiveRecord::RecordInvalid)
      expect(OutboxEvent.where(event_type: "order.fulfilled").count).to eq(0)
    end
  end
end

Test the relay job and each publisher independently with a pre-seeded outbox record. Keep the two test concerns separate: one spec verifies the outbox record is created correctly; another verifies the publisher delivers it correctly. The relay job is responsible for calling publishers — test that routing separately too.

When Is the Outbox Pattern Overkill?

The pattern adds a polling loop, a database table, and a relay layer. For some use cases that is more than you need.

If your event volume is very low — a few events per day — and the consequence of a lost event is genuinely low (a non-critical analytics ping, for example), a well-implemented rescue block with a background retry may be enough. The outbox pattern shines where event loss has business consequences and where events need to survive process restarts.

If you are already running a proper message broker like Kafka or RabbitMQ with transactional semantics, the broker itself may provide the durability guarantees you need. In practice, reliably coordinating a database transaction and a broker write is exactly the dual-write problem this pattern solves — often the outbox is the right answer even when a broker is in the picture.

For deeper retry patterns on the consumer side — exponential backoff, circuit breakers, discard policies — those pair naturally with the outbox. I covered them in the Rails Active Job retries post.

Production Hardening Checklist

Every time I set up the Rails transactional outbox pattern on a client codebase, I verify these before calling it done:

• Unique index on idempotency_key — prevents duplicate outbox records and gives the constraint violation that triggers a rollback if you accidentally queue the same event twice.
• Partial index on pending events — WHERE published_at IS NULL keeps the polling query O(pending) not O(all). Without it, the query slows as delivered events accumulate.
• FOR UPDATE SKIP LOCKED — safe parallel relay workers without application-level coordination.
• MAXIMUM_ATTEMPTS cap — prevents infinite retry storms against a permanently broken consumer.
• Dead letter monitoring — a health check or Sentry alert fires when events exhaust their retries.
• retry! mechanism — you need a safe way to reset dead letter events after fixing a broken consumer.
• Payload size guard — validate during development that payload.to_json.bytesize stays under a sane limit (256KB is generous). Oversized payloads indicate design issues.
• Prune job — prevents unbounded table growth.
• Idempotent consumers — document and enforce that all event consumers must handle duplicate delivery.

FAQ

What is the Rails transactional outbox pattern?

The Rails transactional outbox pattern solves the dual-write problem: the risk that a database write and an external API call succeed or fail independently, leaving your system in an inconsistent state. You write the event to an outbox_events table inside the same database transaction as the business data change. Because both writes are atomic in Postgres, they either both succeed or both fail. A separate background job reads the outbox and handles delivery with retries. The result is at-least-once event delivery with no risk of silent data loss from process restarts or network failures.

How is the outbox pattern different from just enqueueing a background job?

Enqueueing a background job is itself a write to an external system — Redis for Sidekiq, or the jobs table for Solid Queue. If your Rails process dies after committing the database transaction but before successfully enqueuing the job, the event is lost. The transactional outbox avoids this by writing the delivery intent to the same Postgres database, inside the same transaction. Postgres guarantees that if the transaction commits, the outbox record exists. The relay job then reads from Postgres, which never loses a committed record. Even if you use Solid Queue — which stores jobs in Postgres — the solid_queue_jobs table is a different database connection and not covered by your application transaction.

How does FOR UPDATE SKIP LOCKED prevent relay workers from conflicting?

SELECT ... FOR UPDATE SKIP LOCKED locks the selected rows and, critically, skips any rows that are already locked by another transaction. When two relay workers run simultaneously, each picks up a non-overlapping batch — they do not block each other. Without SKIP LOCKED, workers queue behind each other’s locks, serialising what should be parallel delivery. The pattern also prevents double-delivery: the lock on a row held by worker A prevents worker B from selecting the same event until A either commits or rolls back. I cover Postgres locking semantics in detail in the advisory locks post.

Can I use the outbox pattern with Sidekiq instead of Solid Queue?

Yes. The relay job is a standard ActiveJob and the scheduler is decoupled from the outbox table itself. Replace OutboxRelayJob with a Sidekiq worker and schedule it with a sidekiq-cron or whenever/cron entry. The OutboxEvent model and the FOR UPDATE SKIP LOCKED query work identically regardless of job backend. If you are migrating from Sidekiq to Solid Queue, your outbox implementation does not need to change — only the scheduler configuration does.

Silent data loss between your database and external systems is one of the hardest production bugs to diagnose, because you do not know it happened until a customer notices. After nineteen years of Rails production support, the transactional outbox pattern is standard infrastructure on every client engagement where webhooks, billing integrations, or warehouse systems are involved. TTB Software helps teams build reliable event-driven Rails applications that survive deploys, network failures, and process restarts. If your integrations drop events under load or during rolling deploys, we can help.