Digging into Solid Queue and Rails

In the realm of Ruby on Rails and background job processing, Solid Queue is the newest contender. Solid Queue is a robust alternative to well-established solutions like good_job and Sidekiq. This guide explores Solid Queue, delving into its distinctive features and comparing it with its counterparts.

What is Solid Queue?

Solid Queue is an open-source background processing library tailored for Ruby on Rails applications. Its primary objective is to efficiently manage asynchronous tasks, ensuring your application's performance remains rock-solid even during peak traffic and demanding workloads.

Key Features of Solid Queue:

Reliability: Solid Queue places a premium on reliability and data integrity. It guarantees that jobs are processed precisely once, eliminating the risk of duplicates or data loss.
Transaction Support: Unlike some competing solutions, Solid Queue seamlessly integrates with database transactions, ensuring consistent data when both your application and background jobs interact.
Concurrency Control: Solid Queue offers granular control over concurrency, enabling you to specify the number of workers for each job type and optimize resource utilization.
Active Job Compatibility: If your Rails application relies on Active Job, you'll appreciate Solid Queue's effortless compatibility, simplifying migration and adoption.

How Solid Queue Sets Itself Apart

Data Integrity: Solid Queue is committed to data integrity, which is critical for applications handling sensitive information or intricate business logic. It guarantees that jobs are processed reliably, eliminating the possibility of duplicates.
Database Transactions: Integrating Solid Queue with database transactions is a standout feature. In contrast, GoodJob and Sidekiq may necessitate additional configuration through something like Redis and management to maintain transactional consistency.
Concurrency Control: While all three solutions offer concurrency control, Solid Queue's flexibility in specifying worker counts for different job types empowers you to fine-tune resource allocation for optimal performance.

Examples

To show how to use SolidQueue in a Ruby on Rails application, you'll need to take the following steps on a new Rails application.

Install the gem. Add the following to your Gemfile and run bundle install .

gem "solid_queue"

Run the installer

rails generate solid_queue:install

This adds a few migrations, replaces a few configurations in the test, development, and production environment, and adds a solid_queue.yml config file to the app. All the values are commented out by default.

gsub  config/environments/development.rb
        gsub  config/environments/test.rb
        gsub  config/environments/production.rb
      create  config/solid_queue.yml
       rails  railties:install:migrations FROM=solid_queue
Copied migration 20240102191754_create_solid_queue_tables.solid_queue.rb from solid_queue

The configuration wasn’t added to my development.rb environment file but made it to the production.rb file. You may need to append the following to your preferred environment configurations if you aren't adding it to config/application.rb

config.active_job.queue_adapter = :solid_queue

Inside the config/solid_queue.yml configuration file, be sure to uncomment the defaults.

default: &default
  dispatchers:
    - polling_interval: 1
      batch_size: 500
  workers:
    - queues: "*"
      threads: 5
      processes: 1
      polling_interval: 0.1

development:
 <<: *default

test:
 <<: *default

production:
 <<: *default

Be sure to configure Solid Queue as your Active Job queue adapter

# config/application.rb
config.active_job.queue_adapter = :solid_queue

Run the migrations

rails db:migrate

This creates several tables related to different states of a given job:

== 20240102191754 CreateSolidQueueTables: migrating ===========================
-- create_table(:solid_queue_jobs)
   -> 0.0024s
-- create_table(:solid_queue_scheduled_executions)
   -> 0.0009s
-- create_table(:solid_queue_ready_executions)
   -> 0.0012s
-- create_table(:solid_queue_claimed_executions)
   -> 0.0009s
-- create_table(:solid_queue_blocked_executions)
   -> 0.0009s
-- create_table(:solid_queue_failed_executions)
   -> 0.0007s
-- create_table(:solid_queue_pauses)
   -> 0.0007s
-- create_table(:solid_queue_processes)
   -> 0.0009s
-- create_table(:solid_queue_semaphores)
   -> 0.0012s
-- add_foreign_key(:solid_queue_blocked_executions, :solid_queue_jobs, {:column=>:job_id, :on_delete=>:cascade})
   -> 0.0062s
-- add_foreign_key(:solid_queue_claimed_executions, :solid_queue_jobs, {:column=>:job_id, :on_delete=>:cascade})
   -> 0.0054s
-- add_foreign_key(:solid_queue_failed_executions, :solid_queue_jobs, {:column=>:job_id, :on_delete=>:cascade})
   -> 0.0050s
-- add_foreign_key(:solid_queue_ready_executions, :solid_queue_jobs, {:column=>:job_id, :on_delete=>:cascade})
   -> 0.0061s
-- add_foreign_key(:solid_queue_scheduled_executions, :solid_queue_jobs, {:column=>:job_id, :on_delete=>:cascade})
   -> 0.0053s
== 20240102191754 CreateSolidQueueTables: migrated (0.0381s) ==================

Start Solid Queue’s supervisor to be able to run jobs. This should be a new terminal instance.

bundle exec rake solid_queue:start

Define and enqueue jobs as you would with Active Job:

# app/jobs/sample_job.rb
class SampleJob < ApplicationJob
  def perform
    # Your job logic here
  end
end

# Enqueue the job
SampleJob.perform_later

rails console
> SampleJob.perform_later

Pretty simple, right? I love that it lives within the framework and doesn’t make me learn a complicated new standard. I come from Sidekiq, and the workflow is highly similar, minus a few bits.

Handling Failed jobs and retries

Solid Queue does not have an automatic retry feature; instead, it depends on Active Job for this functionality. Any failed jobs will be stored in the system, and a record of the failed execution will be created in the solid_queue_failed_executions table. These failed jobs will remain in the system until manually discarded or re-enqueued.

The team plans to introduce a UI component called Mission Control that allows you to examine and diagnose any specific job or group of jobs efficiently.

For now, you can manually intervene with these methods:

failed_execution = SolidQueue::FailedExecution.find(...) # Find the failed execution related to your job
failed_execution.error # inspect the error

failed_execution.retry # This will re-enqueue the job as if it was enqueued for the first time
failed_execution.discard # This will delete the job from the system

This seems nice for the ability to manually retry if necessary in your app’s logic rather than having it be arbitrary. Some might argue the latter.

Adding some UI in the meantime

I noticed on Twitter a library called Panoptic. The simple UI was built with Bootstrap CSS to provide a GUI for Solid Queue.

You can add it much like Sidekiq if you’re familiar with that configuration style. It’s an engine that bolts on the app.

# add to your Gemfile
gem "panoptic"

Then in your config/routes.rb file, add the following one-liner

# config/routes.rb
Rails.application.routes.draw do
  mount Panoptic::Engine => "/panoptic"
  # more code
end

You can visit localhost:3000:/panoptic to see it in action.

Again, the team has stated that something called Mission Control is on the near horizon, so I wouldn't depend heavily on this library. It is something, though!

In pursuit of increased developer happiness with Puma

While running bundle exec rake solid_queue:start each time you need background job processing, it works fine; a handy Puma plugin integration in Solid Queue allows for some nice automations. When Puma is running (rails server), the processing of your job will kick in alongside it.

I need to add a one-liner to my puma.rb file.

# config/puma.rb
plugin :solid_queue

Conclusion

Solid Queue is a formidable choice for background job processing in Ruby on Rails applications. It prioritizes reliability, data integrity, and seamless database transaction integration.

While GoodJob and Sidekiq remain excellent alternatives, Solid Queue's unique attributes make it a compelling candidate for your next project without additional dependencies or resources. You will need to consider the extra database space it might reserve, but hopefully, it shouldn't be enough to cause too much concern.