GoodJob

GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails.

Inspired by {Delayed::Job}[https://github.com/collectiveidea/delayed_job] and {Que}[https://github.com/que-rb/que], GoodJob is designed for maximum compatibility with Ruby on Rails, ActiveJob, and Postgres to be simple and performant for most workloads.

For more of the story of GoodJob, read the introductory blog post.

<details markdown=β€œ1”> <summary><strong>πŸ“Š Comparison of GoodJob with other job queue backends (click to expand)</strong></summary>

Queues, priority, retries Database Concurrency Reliability/Integrity Latency
**GoodJob** βœ… Yes βœ… Postgres βœ… Multithreaded βœ… ACID, Advisory Locks βœ… Postgres LISTEN/NOTIFY
**Que** βœ… Yes πŸ”ΆοΈ Postgres, requires `structure.sql` βœ… Multithreaded βœ… ACID, Advisory Locks βœ… Postgres LISTEN/NOTIFY
**Delayed Job** βœ… Yes βœ… Postgres πŸ”΄ Single-threaded βœ… ACID, record-based πŸ”Ά Polling
**Sidekiq** βœ… Yes πŸ”΄ Redis βœ… Multithreaded πŸ”΄ Crashes lose jobs βœ… Redis BRPOP
**Sidekiq Pro** βœ… Yes πŸ”΄ Redis βœ… Multithreaded βœ… Redis RPOPLPUSH βœ… Redis RPOPLPUSH

</details>

Table of contents

Set up

  1. Add good_job to your application's Gemfile:

    gem 'good_job'
    
  2. Install the gem:

    $ bundle install
  3. Run the GoodJob install generator. This will generate a database migration to create a table for GoodJob's job records:

    $ bin/rails g good_job:install

    Run the migration:

    $ bin/rails db:migrate

Optional: If using Rails' multiple databases with the migrations_paths configuration option, use the --database option:

```bash
bin/rails g good_job:install --database animals
bin/rails db:migrate:animals
```
  1. Configure the ActiveJob adapter:

    # config/application.rb
    config.active_job.queue_adapter = :good_job
    
  2. Inside of your application, queue your job πŸŽ‰:

    YourJob.perform_later
    

    GoodJob supports all ActiveJob features:

    YourJob.set(queue: :some_queue, wait: 5.minutes, priority: 10).perform_later
    
  3. In development, GoodJob executes jobs immediately. In production, GoodJob provides different options:

    • By default, GoodJob separates job enqueuing from job execution so that jobs can be scaled independently of the web server. Use the GoodJob command-line tool to execute jobs:

      $ bundle exec good_job start

      Ideally the command-line tool should be run on a separate machine or container from the web process. For example, on Heroku:

      web: rails server
      worker: bundle exec good_job start

      The command-line tool supports a variety of options, see the reference below for command-line configuration.

    • GoodJob can also be configured to execute jobs within the web server process to save on resources. This is useful for low-workloads when economy is paramount.

      $ GOOD_JOB_EXECUTION_MODE=async rails server

      Additional configuration is likely necessary, see the reference below for configuration.

Compatibility

Configuration

Command-line options

There several top-level commands available through the good_job command-line tool.

Configuration options are available with help.

good_job start

good_job start executes queued jobs.

$ bundle exec good_job help start

Usage:
  good_job start

Options:
  [--max-threads=COUNT]        # Maximum number of threads to use for working jobs. (env var: GOOD_JOB_MAX_THREADS, default: 5)
  [--queues=QUEUE_LIST]        # Queues to work from. (env var: GOOD_JOB_QUEUES, default: *)
  [--poll-interval=SECONDS]    # Interval between polls for available jobs in seconds (env var: GOOD_JOB_POLL_INTERVAL, default: 1)
  [--max-cache=COUNT]          # Maximum number of scheduled jobs to cache in memory (env var: GOOD_JOB_MAX_CACHE, default: 10000)
  [--shutdown-timeout=SECONDS] # Number of seconds to wait for jobs to finish when shutting down before stopping the thread. (env var: GOOD_JOB_SHUTDOWN_TIMEOUT, default: -1 (forever))
  [--enable-cron]              # Whether to run cron process (default: false)
  [--daemonize]                # Run as a background daemon (default: false)
  [--pidfile=PIDFILE]          # Path to write daemonized Process ID (env var: GOOD_JOB_PIDFILE, default: tmp/pids/good_job.pid)

Executes queued jobs.

All options can be configured with environment variables.
See option descriptions for the matching environment variable name.

== Configuring queues

Separate multiple queues with commas; exclude queues with a leading minus;
separate isolated execution pools with semicolons and threads with colons.

good_job cleanup_preserved_jobs

good_job cleanup_preserved_jobs deletes preserved job records. See GoodJob.preserve_job_records for when this command is useful.

$ bundle exec good_job help cleanup_preserved_jobs

Usage:
  good_job cleanup_preserved_jobs

Options:
  [--before-seconds-ago=SECONDS] # Delete records finished more than this many seconds ago (env var:  GOOD_JOB_CLEANUP_PRESERVED_JOBS_BEFORE_SECONDS_AGO, default: 86400)

Deletes preserved job records.

By default, GoodJob deletes job records when the job is performed and this
command is not necessary.

However, when `GoodJob.preserve_job_records = true`, the jobs will be
preserved in the database. This is useful when wanting to analyze or
inspect job performance.

If you are preserving job records this way, use this command regularly
to delete old records and preserve space in your database.

Configuration options

To use GoodJob, you can set config.active_job.queue_adapter to a :good_job.

Additional configuration can be provided via config.good_job.OPTION = ... for example:

# config/application.rb

config.active_job.queue_adapter = :good_job

# Configure options individually...
config.good_job.execution_mode = :async
config.good_job.max_threads = 5
config.good_job.poll_interval = 30 # seconds
config.good_job.shutdown_timeout = 25 # seconds
config.good_job.enable_cron = true
config.good_job.cron = { example: { cron: '0 * * * *', class: 'ExampleJob'  } }
config.good_job.queues = '*'

# ...or all at once.
config.good_job = {
  execution_mode: :async,
  max_threads: 5,
  poll_interval: 30,
  shutdown_timeout: 25,
  enable_cron: true,
  cron: {
    example: {
      cron: '0 * * * *',
      class: 'ExampleJob'
    },
  },
  queues: '*',
}

Available configuration options are:

By default, GoodJob configures the following execution modes per environment:

# config/environments/development.rb
config.active_job.queue_adapter = :good_job
config.good_job.execution_mode = :async

# config/environments/test.rb
config.active_job.queue_adapter = :good_job
config.good_job.execution_mode = :inline

# config/environments/production.rb
config.active_job.queue_adapter = :good_job
config.good_job.execution_mode = :external

Global options

Good Job’s general behavior can also be configured via several attributes directly on the GoodJob module:

You’ll generally want to configure these in config/initializers/good_job.rb, like so:

# config/initializers/good_job.rb
GoodJob.active_record_parent_class = "ApplicationRecord"
GoodJob.preserve_job_records = true
GoodJob.retry_on_unhandled_error = false
GoodJob.on_thread_error = -> (exception) { Raven.capture_exception(exception) }

Dashboard

🚧 GoodJob's dashboard is a work in progress. Please contribute ideas and code on {Github}[https://github.com/bensheldon/good_job/issues].

GoodJob includes a Dashboard as a mountable Rails::Engine.

  1. Explicitly require the Engine code at the top of your config/application.rb file, immediately after Rails is required. This is necessary because the mountable engine is an optional feature of GoodJob.

    # config/application.rb
    require_relative 'boot'
    
    require 'rails/all'
    require 'good_job/engine' # <= Add this line
    # ...
    
  2. Mount the engine in your config/routes.rb file. The following will mount it at http://example.com/good_job.

    # config/routes.rb
    # ...
    mount GoodJob::Engine => 'good_job'
    

    Because jobs can potentially contain sensitive information, you should authorize access. For example, using Devise's authenticate helper, that might look like:

    # config/routes.rb
    # ...
    authenticate :user, ->(user) { user.admin? } do
      mount GoodJob::Engine => 'good_job'
    end
    

    Another option is using basic auth like this:

    # config/initializers/good_job.rb
    GoodJob::Engine.middleware.use(Rack::Auth::Basic) do |username, password|
      ActiveSupport::SecurityUtils.secure_compare(Rails.application.credentials.good_job_username, username) &&
        ActiveSupport::SecurityUtils.secure_compare(Rails.application.credentials.good_job_password, password)
    end
    

ActiveJob concurrency

GoodJob can extend ActiveJob to provide limits on concurrently running jobs, either at time of enqueue or at perform. Limiting concurrency can help prevent duplicate, double or unecessary jobs from being enqueued, or race conditions when performing, for example when interacting with 3rd-party APIs.

Note: Limiting concurrency at enqueue requires Rails 6.0+ because Rails 5.2 cannot halt ActiveJob callbacks.

class MyJob < ApplicationJob
  include GoodJob::ActiveJobExtensions::Concurrency

  good_job_control_concurrency_with(
    # Maximum number of unfinished jobs to allow with the concurrency key
    total_limit: 1,

    # Or, if more control is needed:
    # Maximum number of jobs with the concurrency key to be concurrently enqueued (excludes performing jobs)
    enqueue_limit: 2,
    # Maximum number of jobs with the concurrency key to be concurrently performed (excludes enqueued jobs)
    perform_limit: 1,

    # A unique key to be globally locked against.
    # Can be String or Lambda/Proc that is invoked in the context of the job.
    # Note: Arguments passed to #perform_later must be accessed through `arguments` method.
    key: -> { "Unique-#{arguments.first}" } #  MyJob.perform_later("Alice") => "Unique-Alice"
  )

  def perform(first_name)
    # do work
  end
end

When testing, the resulting concurrency key value can be inspected:

job = MyJob.perform_later("Alice")
job.good_job_concurrency_key #=> "Unique-Alice"

Cron-style repeating/recurring jobs

GoodJob can enqueue jobs on a recurring basis that can be used as a replacement for cron.

Cron-style jobs are run on every GoodJob process (e.g. CLI or async execution mode) when config.good_job.enable_cron = true; use GoodJob's ActiveJob concurrency extension to limit the number of jobs that are enqueued.

Cron-format is parsed by the {fugit} gem, which has support for seconds-level resolution (e.g. * * * * * *).

# config/environments/application.rb or a specific environment e.g. production.rb

# Enable cron in this process; e.g. only run on the first Heroku worker process
config.good_job.enable_cron = ENV['DYNO'] == 'worker.1' # or `true` or via $GOOD_JOB_ENABLE_CRON

# Configure cron with a hash that has a unique key for each recurring job
config.good_job.cron = {
  # Every 15 minutes, enqueue `ExampleJob.set(priority: -10).perform_later(42, name: "Alice")`
  frequent_task: { # each recurring job must have a unique key
    cron: "*/15 * * * *", # cron-style scheduling format by fugit gem
    class: "ExampleJob", # reference the Job class with a string
    args: [42, { name: "Alice" }], # arguments to pass; can also be a proc e.g. `-> { { when: Time.now } }`
    set: { priority: -10 }, # additional ActiveJob properties; can also be a lambda/proc e.g. `-> { { priority: [1,2].sample } }`
    description: "Something helpful", # optional description that appears in Dashboard (coming soon!)
  },
  another_task: {
    cron: "0 0,12 * * *",
    class: "AnotherJob",
  },
  # etc.
}

Updating

GoodJob follows semantic versioning, though updates may be encouraged through deprecation warnings in minor versions.

Upgrading minor versions

Upgrading between minor versions (e.g. v1.4 to v1.5) should not introduce breaking changes, but can introduce new deprecation warnings and database migration notices.

To perform upgrades to the GoodJob database tables:

  1. Generate new database migration files:

    bin/rails g good_job:update

Optional: If using Rails' multiple databases with the migrations_paths configuration option, use the --database option:

```bash
$ bin/rails g good_job:update --database animals
```
  1. Run the database migration locally

    bin/rails db:migrate
  2. Commit the migration files and resulting db/schema.rb changes.

  3. Deploy the code, run the migrations against the production database, and restart server/worker processes.

Upgrading v1 to v2

GoodJob v2 introduces a new Advisory Lock key format that is different than the v1 advisory lock key format; it's therefore necessary to perform a simple, but staged production upgrade. If you are already using >= v1.12+ no other changes are necessary.

  1. Upgrade your production environment to v1.99.x following the minor version upgrade process, including database migrations. v1.99 is a transitional release that is safely compatible with both v1.x and v2.0.0 because it uses both v1- and v2-formatted advisory locks.

  2. Address any deprecation warnings generated by v1.99.

  3. Upgrade your production environment to v1.99.x to v2.0.x again following the minor upgrade process.

Notable changes:

Go deeper

Exceptions, retries, and reliability

GoodJob guarantees that a completely-performed job will run once and only once. GoodJob fully supports ActiveJob's built-in functionality for error handling, retries and timeouts.

Exceptions

ActiveJob provides tools for rescuing and retrying exceptions, including retry_on, discard_on, rescue_from that will rescue exceptions before they get to GoodJob.

If errors do reach GoodJob, you can assign a callable to GoodJob.on_thread_error to be notified. For example, to log errors to an exception monitoring service like Sentry (or Bugsnag, Airbrake, Honeybadger, etc.):

# config/initializers/good_job.rb
GoodJob.on_thread_error = -> (exception) { Raven.capture_exception(exception) }

Retries

By default, GoodJob will automatically and immediately retry a job when an exception is raised to GoodJob.

However, ActiveJob can be configured to retry an infinite number of times, with an exponential backoff. Using ActiveJob's retry_on prevents exceptions from reaching GoodJob:

class ApplicationJob < ActiveJob::Base
  retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
  # ...
end

When using retry_on with a limited number of retries, the final exception will not be rescued and will raise to GoodJob. GoodJob can be configured to discard un-handled exceptions instead of retrying them:

# config/initializers/good_job.rb
GoodJob.retry_on_unhandled_error = false

Alternatively, pass a block to retry_on to handle the final exception instead of raising it to GoodJob:

class ApplicationJob < ActiveJob::Base
  retry_on StandardError, attempts: 5 do |_job, _exception|
    # Log error, do nothing, etc.
  end
  # ...
end

When using retry_on with an infinite number of retries, exceptions will never be raised to GoodJob, which means GoodJob.on_thread_error will never be called. To report log or report exceptions to an exception monitoring service (e.g. Sentry, Bugsnag, Airbrake, Honeybadger, etc), create an explicit exception wrapper. For example:

class ApplicationJob < ActiveJob::Base
  retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY

  retry_on SpecialError, attempts: 5 do |_job, exception|
    Raven.capture_exception(exception)
  end

  around_perform do |_job, block|
    block.call
  rescue StandardError => e
    Raven.capture_exception(e)
    raise
  end
  # ...
end

ActionMailer retries

Any configuration in ApplicationJob will have to be duplicated on ActionMailer::MailDeliveryJob (ActionMailer::DeliveryJob in Rails 5.2 or earlier) because ActionMailer uses a custom class, ActionMailer::MailDeliveryJob, which inherits from ActiveJob::Base, rather than your applications ApplicationJob.

You can use an initializer to configure ActionMailer::MailDeliveryJob, for example:

# config/initializers/good_job.rb
ActionMailer::MailDeliveryJob.retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY

# With Sentry (or Bugsnag, Airbrake, Honeybadger, etc.)
ActionMailer::MailDeliveryJob.around_perform do |_job, block|
  block.call
rescue StandardError => e
  Raven.capture_exception(e)
  raise
end

Note, that ActionMailer::MailDeliveryJob is a default since Rails 6.0. Be sure that your app is using that class, as it might also be configured to use (deprecated now) ActionMailer::DeliveryJob.

Timeouts

Job timeouts can be configured with an around_perform:

class ApplicationJob < ActiveJob::Base
  JobTimeoutError = Class.new(StandardError)

  around_perform do |_job, block|
    # Timeout jobs after 10 minutes
    Timeout.timeout(10.minutes, JobTimeoutError) do
      block.call
    end
  end
end

Optimize queues, threads, and processes

By default, GoodJob creates a single thread execution pool that will execute jobs from any queue. Depending on your application's workload, job types, and service level objectives, you may wish to optimize execution resources. For example, providing dedicated execution resources for transactional emails so they are not delayed by long-running batch jobs. Some options:

Keep in mind, queue operations and management is an advanced discipline. This stuff is complex, especially for heavy workloads and unique processing requirements. Good job πŸ‘

Database connections

Each GoodJob execution thread requires its own database connection that is automatically checked out from Rails’ connection pool. Allowing GoodJob to create more threads than available database connections can lead to timeouts and is not recommended. For example:

# config/database.yml
pool: <%= [ENV.fetch("RAILS_MAX_THREADS", 5).to_i, ENV.fetch("GOOD_JOB_MAX_THREADS", 4).to_i].max %>

Execute jobs async / in-process

GoodJob can execute jobs β€œasync” in the same process as the web server (e.g. bin/rails s). GoodJob's async execution mode offers benefits of economy by not requiring a separate job worker process, but with the tradeoff of increased complexity. Async mode can be configured in two ways:

Depending on your application configuration, you may need to take additional steps:

GoodJob is compatible with Puma's preload_app! method.

Migrate to GoodJob from a different ActiveJob backend

If your application is already using an ActiveJob backend, you will need to install GoodJob to enqueue and perform newly created jobs and finish performing pre-existing jobs on the previous backend.

  1. Enqueue newly created jobs on GoodJob either entirely by setting ActiveJob::Base.queue_adapter = :good_job or progressively via individual job classes:

    # jobs/specific_job.rb
    class SpecificJob < ApplicationJob
      self.queue_adapter = :good_job
      # ...
    end
    
  2. Continue running executors for both backends. For example, on Heroku it's possible to run two processes within the same dyno:

procfile # Procfile # ... worker: bundle exec que ./config/environment.rb & bundle exec good_job & wait -n

  1. Once you are confident that no unperformed jobs remain in the previous ActiveJob backend, code and configuration for that backend can be completely removed.

Monitor and preserve worked jobs

GoodJob is fully instrumented with {ActiveSupport::Notifications}.

By default, GoodJob will delete job records after they are run, regardless of whether they succeed or not (raising a kind of StandardError), unless they are interrupted (raising a kind of Exception).

To preserve job records for later inspection, set an initializer:

# config/initializers/good_job.rb
GoodJob.preserve_job_records = true

It is also necessary to delete these preserved jobs from the database after a certain time period:

PgBouncer compatibility

GoodJob is not compatible with PgBouncer in transaction mode, but is compatible with PgBouncer's connection mode. GoodJob uses connection-based advisory locks and LISTEN/NOTIFY, both of which require full database connections.

A workaround to this limitation is to make a direct database connection available to GoodJob. With Rails 6.0's support for multiple databases, a direct connection to the database can be configured:

  1. Define a direct connection to your database that is not proxied through PgBouncer, for example:

    # config/database.yml
    
    production:
      primary:
        url: postgres://pgbouncer_host/my_database
      primary_direct:
        url: postgres://database_host/my_database
  2. Create a new ActiveRecord base class that uses the direct database connection

    # app/models/application_direct_record.rb
    
    class ApplicationDirectRecord < ActiveRecord::Base
      self.abstract_class = true
      connects_to database: :primary_direct
    end
    
  3. Configure GoodJob to use the newly created ActiveRecord base class:

    # config/initializers/good_job.rb
    
    GoodJob.active_record_parent_class = "ApplicationDirectRecord"
    

Contribute

Contributions are welcomed and appreciated πŸ™

Gem development

To run tests:

# Clone the repository locally
$ git clone git@github.com:bensheldon/good_job.git

# Set up the local environment
$ bin/setup

# Run the tests
$ bin/rspec

This gem uses Appraisal to run tests against multiple versions of Rails:

# Install Appraisal(s) gemfiles
$ bundle exec appraisal

# Run tests
$ bundle exec appraisal bin/rspec

For developing locally within another Ruby on Rails project:

# Within Ruby on Rails directory...
$ bundle config local.good_job /path/to/local/git/repository

# Confirm that the local copy is used
$ bundle install

# => Using good_job 0.1.0 from https://github.com/bensheldon/good_job.git (at /Users/You/Projects/good_job@dc57fb0)

Release

Package maintainers can release this gem by running:

# Sign into rubygems
$ gem signin

# Add a .env file with the following:
# CHANGELOG_GITHUB_TOKEN= # Github Personal Access Token

# Update version number, changelog, and create git commit:
$ bundle exec rake release[minor] # major,minor,patch

# ..and follow subsequent directions.

License

The gem is available as open source under the terms of the MIT License.