Solid Queue Recurring Jobs: Replace Whenever and Sidekiq-Cron in Rails 8

Three years ago I was the fourth engineer to touch a Capistrano deploy script that had accumulated cron job management like a coral reef accumulates polyps. Twelve jobs scattered across four locations: a whenever schedule that wrote to the app server crontab, three Heroku Scheduler entries nobody had documented, a sidekiq-cron initializer for the Redis-backed ones, and two rake tasks wired directly into the server crontab via a comment that read “do not touch, added by Dave, 2019.” Dave had left the company. Nobody knew what those two tasks did. Everyone was afraid to remove them.

Running all of this in production meant the app server had cron state, Redis had cron state, Heroku’s scheduler had cron state, and the Rails codebase had none of these things visible to it. When a job failed at 3am, the only evidence was a missing row in the database and an email in a shared inbox nobody monitored.

With Rails 8 and Solid Queue recurring jobs, that architecture is finally unnecessary. Everything lives in the database. The schedule is a YAML file in your repository. The dispatcher process handles timing. Failures surface in the same place as every other failed job. I have migrated four production codebases off sidekiq-cron and whenever in the last year and not once wished for the old approach.

Why the Old Approaches Break Down

whenever writes a crontab on the app server. This is a problem the moment you have more than one server — each server runs its own copy of the schedule, and you need external coordination to prevent duplicate execution. It is also invisible to Ruby: whenever you look at the running jobs, the crontab is outside the application’s concern. Exception trackers know nothing about it. Deploys can fail to update the crontab if Capistrano is misconfigured. And the job process is spawned fresh for each execution, meaning no connection pool warmup, no shared state, and a cold Ruby boot on every run.

sidekiq-cron is better — it runs inside the Sidekiq process and uses Lua scripts in Redis to achieve a distributed lock on scheduled execution. But it requires Redis, it requires an additional gem on top of Sidekiq, and the schedule configuration is typically defined in a Ruby initializer that runs at boot time. Test environments frequently boot with the cron schedule disabled via environment conditionals that someone eventually breaks. When you move to Solid Queue, you lose sidekiq-cron entirely and need a replacement anyway.

Solid Queue’s recurring tasks solve all of this. The schedule is a YAML file tracked in source control. The dispatcher process reads it at startup, stores the schedule in your Postgres database, and fires executions at the right times. One server, ten servers — the database provides the distributed lock. Failures surface as failed SolidQueue::Job records, identical to any other failed background job.

Configuring Solid Queue Recurring Jobs

The recurring schedule lives in config/recurring.yml by default. A minimal production schedule:

# config/recurring.yml
production:
  cleanup_stale_sessions:
    class: CleanupStaleSessionsJob
    schedule: "0 2 * * *"
    queue: maintenance

  send_weekly_digest:
    class: SendWeeklyDigestJob
    schedule: "0 9 * * 1"
    args: [{ force: false }]
    queue: mailers
    priority: 5

  refresh_exchange_rates:
    class: RefreshExchangeRatesJob
    schedule: "*/15 * * * *"
    queue: default

Each entry has a name used as an identifier in the database, a class pointing to an ActiveJob subclass, and a schedule in standard five-field cron format. Optional fields are args (passed directly to perform), queue, and priority.

The cron format is parsed by the fugit gem, which Solid Queue depends on. Fugit understands standard cron syntax plus macros like @hourly, @daily, and @weekly.

Tell Solid Queue where the recurring file lives in config/solid_queue.yml:

# config/solid_queue.yml
default: &default
  dispatchers:
    - polling_interval: 1
      batch_size: 500
      recurring_tasks_file: config/recurring.yml

  workers:
    - queues: "*"
      threads: 3
      polling_interval: 0.1

production:
  <<: *default
  workers:
    - queues: [critical, default, mailers, maintenance]
      threads: 5
      polling_interval: 0.1

The dispatchers section is what enables recurring tasks. Without at least one dispatcher configured, Solid Queue will process jobs fine but never schedule recurring executions. One dispatcher handles the clock. Multiple dispatchers are safe — the database record prevents double-scheduling.

Running the Solid Queue Dispatcher

In development, the simplest setup is the combined supervisor:

bin/rails solid_queue:start

This reads config/solid_queue.yml and spawns the dispatcher and worker processes. In production via Kamal, add the worker as a separate service so you can scale and restart it independently of the web process:

# config/deploy.yml
servers:
  web:
    hosts:
      - 192.168.1.1
  worker:
    hosts:
      - 192.168.1.2
    cmd: bin/rails solid_queue:start
    env:
      RAILS_ENV: production

If you are on a single-server setup, you can run Solid Queue as a Puma plugin, which starts the supervisor inside the Puma process:

# config/puma.rb
plugin :solid_queue

The plugin approach is convenient but blurs process boundaries. I prefer the separate worker service for anything beyond a low-traffic single-server app — it lets you scale job processing independently and restart workers without touching Puma. The Kamal deployment guide covers the multi-service setup in detail.

Job Classes and Arguments

Recurring jobs are plain ActiveJob classes. Nothing special is required:

# app/jobs/cleanup_stale_sessions_job.rb
class CleanupStaleSessionsJob < ApplicationJob
  queue_as :maintenance

  def perform
    cutoff = 30.days.ago
    Session.where("last_active_at < ?", cutoff).in_batches.delete_all
    Rails.logger.info "Cleaned up sessions inactive since #{cutoff}"
  end
end

When passing args in the YAML, they map positionally to perform parameters:

# config/recurring.yml
  generate_monthly_report:
    class: GenerateReportJob
    schedule: "0 6 1 * *"    # 6am on the 1st of each month
    args: [monthly, { recipients: ["ops@example.com"] }]

# app/jobs/generate_report_job.rb
class GenerateReportJob < ApplicationJob
  def perform(period, options = {})
    recipients = options.fetch("recipients", [])
    ReportMailer.with(period: period, recipients: recipients).monthly.deliver_later
  end
end

One practical note: args from YAML are deserialized as plain Ruby objects — strings, integers, and hashes with string keys. Do not pass ActiveRecord objects or symbols; they will not survive the YAML round-trip correctly. Use IDs and look up records inside perform.

Missed Executions: What Happens When the Server Is Down

By default, Solid Queue does not backfill missed executions. If your app is down during a scheduled window, that execution is simply skipped. The next scheduled time fires normally.

For most scheduled jobs this is the correct behaviour. A weekly digest does not need to send twice because a deploy ran long. A nightly cleanup that missed one night is fine — it runs the next night.

When you genuinely need to recover a missed job, build that logic into the job itself rather than relying on the scheduler:

class GenerateInvoicesJob < ApplicationJob
  def perform(billing_date = Date.current)
    # Called on schedule with no args — billing_date defaults to today
    # Called manually with a specific date to backfill
    Invoice::Generator.run(for_date: billing_date)
  end
end

Then for recovery you simply enqueue with the missed date:

GenerateInvoicesJob.perform_later(Date.yesterday)

This is cleaner than automatic backfill because the job knows exactly what date it is processing. Automatic backfill in missed-window scenarios tends to produce duplicate work or race conditions when multiple processes detect the gap simultaneously.

Timezone Pitfalls

Cron runs in UTC. Solid Queue’s dispatcher runs in UTC. Rails’ config.time_zone has no effect on when recurring tasks fire.

The 9am weekly digest in the config above fires at 09:00 UTC. If your users are in Amsterdam (UTC+2 in summer, UTC+1 in winter), that digest arrives at 11am in summer and 10am in winter — the kind of inconsistency that generates complaints after the clocks change.

For time-sensitive schedules, shift the UTC offset explicitly in the cron expression and document it clearly. For anything that needs to account for DST properly, guard inside the job itself:

class SendWeeklyDigestJob < ApplicationJob
  def perform
    # Abort if we are outside the intended local delivery window
    local_hour = Time.current.in_time_zone("Amsterdam").hour
    return unless (8..10).cover?(local_hour)

    DigestMailer.weekly.deliver_later
  end
end

This idempotent guard means you can trigger the job from staging or a monitoring script without producing a real email outside business hours. It also makes the timezone intent explicit in Ruby rather than buried in a cron expression comment.

Testing Solid Queue Recurring Jobs

Test the job class independently of the schedule. The schedule is configuration; the job behaviour is code.

# spec/jobs/cleanup_stale_sessions_job_spec.rb
RSpec.describe CleanupStaleSessionsJob, type: :job do
  describe "#perform" do
    let!(:stale_session) { create(:session, last_active_at: 45.days.ago) }
    let!(:active_session) { create(:session, last_active_at: 5.days.ago) }

    it "deletes sessions inactive for more than 30 days" do
      expect { described_class.new.perform }.to change(Session, :count).by(-1)
      expect { stale_session.reload }.to raise_error(ActiveRecord::RecordNotFound)
    end

    it "preserves recently active sessions" do
      described_class.new.perform
      expect(active_session.reload).to be_persisted
    end
  end
end

To verify that your YAML schedule is syntactically valid and references real job classes, add a lightweight spec that loads the config at test time:

# spec/config/recurring_spec.rb
require "fugit"

RSpec.describe "config/recurring.yml" do
  let(:config) do
    YAML.load_file(Rails.root.join("config/recurring.yml"))[Rails.env] || {}
  end

  it "references valid job classes" do
    config.each_value do |task|
      expect { task["class"].constantize }.not_to raise_error,
        "#{task["class"]} is not a valid job class"
    end
  end

  it "uses valid cron expressions" do
    config.each do |name, task|
      cron = Fugit::Cron.parse(task["schedule"])
      expect(cron).not_to be_nil,
        "#{name} has an invalid cron expression: #{task["schedule"]}"
    end
  end
end

This catches the two most common recurring job mistakes — typos in class names and malformed cron strings — without needing a running dispatcher.

Observability: Seeing What Actually Ran

Solid Queue stores every execution in the database. You can query directly from a console:

# Last 20 recurring executions
SolidQueue::RecurringExecution.order(created_at: :desc).limit(20).each do |ex|
  puts "#{ex.task_key} — ran at #{ex.created_at.strftime("%Y-%m-%d %H:%M UTC")}"
end

# Failed jobs in the last 24 hours
SolidQueue::FailedExecution.where("created_at > ?", 24.hours.ago).count

For a proper dashboard, mount the Mission Control engine in config/routes.rb:

mount MissionControl::Jobs::Engine, at: "/jobs"

Secure it behind authentication. Quick HTTP basic auth via credentials:

# config/initializers/mission_control.rb
MissionControl::Jobs.http_basic_auth_credentials = {
  username: Rails.application.credentials.dig(:mission_control, :username),
  password: Rails.application.credentials.dig(:mission_control, :password)
}

Mission Control shows queued, running, scheduled, and failed jobs with the full exception and backtrace for failures. You can retry failed jobs from the UI without touching the console — essential when a 3am job fails and you want to recover without waking an engineer. The Solid Queue background jobs deep-dive covers the full Mission Control setup including gem dependencies and auth options.

Preventing Overlapping Executions

Solid Queue does not skip a scheduled execution if the previous one is still running. If CleanupStaleSessionsJob takes 90 minutes and fires every hour, you will have overlapping executions. Guard against this with a database-level advisory lock:

class CleanupStaleSessionsJob < ApplicationJob
  def perform
    # with_advisory_lock from the with_advisory_lock gem
    # timeout_seconds: 0 means bail immediately if the lock is taken
    with_advisory_lock("cleanup_stale_sessions", timeout_seconds: 0) do
      Session.where("last_active_at < ?", 30.days.ago).in_batches.delete_all
    end
  end
end

The with_advisory_lock gem acquires a Postgres advisory lock for the duration of the block. If another instance of the job holds the lock, the new execution exits cleanly rather than queueing behind it. The advisory locks post goes deep on this pattern, including the gem setup and timeout semantics.

FAQ

Can I use Solid Queue recurring jobs without Rails 8?

Solid Queue works as a standalone gem with Rails 7.1 and later. Add gem "solid_queue" to your Gemfile, run bin/rails generate solid_queue:install, and configure recurring tasks as described above. Rails 8 includes it by default and configures it as the queue adapter in new apps, but the recurring jobs feature is not Rails-8-exclusive.

How do I migrate from whenever to Solid Queue recurring jobs?

Remove the whenever gem and its config/schedule.rb. For each rake task or job defined in the schedule, create a corresponding ActiveJob class if one does not exist, then add an entry to config/recurring.yml. Remove the crontab entries on the server — whenever --clear-crontab — or via your deploy script. Finally, ensure the Solid Queue dispatcher is running. The Sidekiq to Solid Queue migration guide covers the dependency removal and deploy sequencing in detail.

Does Solid Queue’s recurring job scheduler survive a deploy?

Yes. The dispatcher reads config/recurring.yml at startup and reconciles it with what is stored in the database. Tasks that exist in the YAML but not in the database are inserted. Tasks that were removed from the YAML are deleted from the database. Tasks that are unchanged are left alone. A rolling deploy that restarts the worker process will reload the schedule cleanly without duplicating or dropping executions.

Does the dispatcher need to run on the same server as the workers?

No. The dispatcher and workers communicate only through the database. The dispatcher creates SolidQueue::ScheduledExecution records; workers pick them up. You can run the dispatcher on your web server and workers on dedicated job servers, or run everything together. The only constraint is that all processes share the same database.

Still running scheduled jobs via whenever and a brittle crontab? TTB Software helps Rails teams modernize their background job infrastructure with Solid Queue, Mission Control, and sensible production operations. We have been doing this for nineteen years.