module RocketJob

Convert to and from CSV, JSON, xlsx, and PSV files.

Example, Convert CSV file to JSON.

job = RocketJob::Jobs::ConversionJob.new
job.input_category.file_name  = "data.csv"
job.output_category.file_name = "data.json"
job.save!

Example, Convert JSON file to PSV and compress it with GZip.

job = RocketJob::Jobs::ConversionJob.new
job.input_category.file_name  = "data.json"
job.output_category.file_name = "data.psv.gz"
job.save!

Example, Read a CSV file that has been zipped from a remote website and the convert it to a GZipped json file.

job = RocketJob::Jobs::ConversionJob.new
job.input_category.file_name  = "https://example.org/file.zip"
job.output_category.file_name = "data.json.gz"
job.save!

Copy the source_url file/url/path to the target file/url/path.

Example: Upload a file to an SFTP Server:

RocketJob::Jobs::CopyFileJob.create!(

source_url:  "/exports/uploads/important.csv.pgp",
target_url:  "sftp://sftp.example.org/uploads/important.csv.pgp",
target_args: {
  username: "Jack",
  password: "OpenSesame",
  ssh_options: {
    IdentityFile: "~/.ssh/secondary"
  }
}

)

Notes:

Job to dynamically perform ruby code on demand as a Batch,

Often used for data correction or cleansing.

Example: Iterate over all rows in a table:

code = <<-CODE
  if user = User.find(row)
    user.cleanse_attributes!
    user.save(validate: false)
  end
CODE
job  = RocketJob::Jobs::OnDemandBatchJob.new(code: code, description: 'cleanse users')
arel = User.unscoped.all.order('updated_at DESC')
job.upload_arel(arel)
job.save!

Console Testing:

code = <<-CODE
  if user = User.find(row)
    user.cleanse_attributes!
    user.save(validate: false)
  end
CODE
job  = RocketJob::Jobs::OnDemandBatchJob.new(code: code, description: 'cleanse users')

# Run against a sub-set using a limit
arel = User.unscoped.all.order('updated_at DESC').limit(100)
job.upload_arel(arel)

# Run the subset directly within the console
job.perform_now
job.cleanup!

By default output is not collected, call the method `#collect_output` to collect output.

Example:

job = RocketJob::Jobs::OnDemandBatchJob(
  description:              'Fix data',
  code:                     code,
  throttle_running_workers: 5,
  priority:                 30
)
job.collect_output
job.save!

Example: Move the upload operation into a before_batch.

upload_code = <<-CODE
  arel = User.unscoped.all.order('updated_at DESC')
  upload_arel(arel)
CODE

code = <<-CODE
  if user = User.find(row)
    user.cleanse_attributes!
    user.save(validate: false)
  end
CODE

RocketJob::Jobs::OnDemandBatchJob.create!(
  upload_code: upload_code,
  code:        code,
  description: 'cleanse users'
)

Job to dynamically perform ruby code on demand,

Create or schedule a generalized job for one off fixes or cleanups.

Example: Iterate over all rows in a table:

code = <<~CODE
  User.unscoped.all.order('updated_at DESC').each |user|
    user.cleanse_attributes!
    user.save!
  end
CODE

RocketJob::Jobs::OnDemandJob.create!(
  code:          code,
  description:   'Cleanse users'
)

Example: Test job in a console:

code = <<~CODE
  User.unscoped.all.order('updated_at DESC').each |user|
    user.cleanse_attributes!
    user.save!
  end
CODE

job = RocketJob::Jobs::OnDemandJob.new(code: code, description: 'cleanse users')
job.perform_now

Example: Pass input data:

code = <<~CODE
  puts data['a'] * data['b']
CODE

RocketJob::Jobs::OnDemandJob.create!(
  code: code,
  data: {'a' => 10, 'b' => 2}
)

Example: Retain output:

code = <<~CODE
  data['result'] = data['a'] * data['b']
CODE

RocketJob::Jobs::OnDemandJob.create!(
  code:           code,
  data:           {'a' => 10, 'b' => 2}
)

Example: Schedule the job to run nightly at 2am Eastern:

RocketJob::Jobs::OnDemandJob.create!(
  cron_schedule: '0 2 * * * America/New_York',
  code:          code
)

Example: Change the job priority, description, etc.

RocketJob::Jobs::OnDemandJob.create!(
  code:          code,
  description:   'Cleanse users',
  priority:      30
)

Example: Automatically retry up to 5 times on failure:

RocketJob::Jobs::OnDemandJob.create!(
  retry_limit: 5
  code:        code
)

Worker behavior for a job

@formatter:off

Constants

VERSION

Public Class Methods

create_indexes() click to toggle source
# File lib/rocket_job/rocket_job.rb, line 2
def self.create_indexes
  # Ensure models with indexes are loaded into memory first
  Job.create_indexes
  Server.create_indexes
  DirmonEntry.create_indexes
end
rails!() click to toggle source
# File lib/rocket_job/rocket_job.rb, line 57
def self.rails!
  @rails = true
end
rails?() click to toggle source

When running inside a Rocket Job server process, returns true when Rails has been initialized.

# File lib/rocket_job/rocket_job.rb, line 16
def self.rails?
  @rails
end
seconds_as_duration(seconds) click to toggle source

Returns a human readable duration from the supplied [Float] number of seconds

# File lib/rocket_job/rocket_job.rb, line 27
def self.seconds_as_duration(seconds)
  return nil unless seconds

  if seconds >= 86_400.0 # 1 day
    "#{(seconds / 86_400).to_i}d #{Time.at(seconds).strftime('%-Hh %-Mm')}"
  elsif seconds >= 3600.0 # 1 hour
    Time.at(seconds).strftime("%-Hh %-Mm")
  elsif seconds >= 60.0 # 1 minute
    Time.at(seconds).strftime("%-Mm %-Ss")
  elsif seconds >= 1.0 # 1 second
    format("%.3fs", seconds)
  else
    duration = seconds * 1000
    if defined? JRuby
      "#{duration.to_i}ms"
    else
      duration < 10.0 ? format("%.3fms", duration) : format("%.1fms", duration)
    end
  end
end
server!() click to toggle source
# File lib/rocket_job/rocket_job.rb, line 53
def self.server!
  @server = true
end
server?() click to toggle source

Whether the current process is running inside a Rocket Job server process.

# File lib/rocket_job/rocket_job.rb, line 10
def self.server?
  @server
end
standalone?() click to toggle source

When running inside a Rocket Job server process, returns true when running standalone.

# File lib/rocket_job/rocket_job.rb, line 22
def self.standalone?
  !@rails
end