module RocketJob
Convert to and from CSV, JSON, xlsx, and PSV files.
Example, Convert CSV file to JSON.
job = RocketJob::Jobs::ConversionJob.new job.input_category.file_name = "data.csv" job.output_category.file_name = "data.json" job.save!
Example, Convert JSON file to PSV and compress it with GZip.
job = RocketJob::Jobs::ConversionJob.new job.input_category.file_name = "data.json" job.output_category.file_name = "data.psv.gz" job.save!
Example, Read a CSV file that has been zipped from a remote website and the convert it to a GZipped json file.
job = RocketJob::Jobs::ConversionJob.new job.input_category.file_name = "https://example.org/file.zip" job.output_category.file_name = "data.json.gz" job.save!
Copy the source_url file/url/path to the target file/url/path.
Example: Upload a file to an SFTP Server:
RocketJob::Jobs::CopyFileJob.create!(
source_url: "/exports/uploads/important.csv.pgp", target_url: "sftp://sftp.example.org/uploads/important.csv.pgp", target_args: { username: "Jack", password: "OpenSesame", ssh_options: { IdentityFile: "~/.ssh/secondary" } }
)
Notes:
-
The password is only encrypted when the Symmetric Encryption gem has been installed.
-
If `decrypt: true` then the file will be decrypted with Symmetric Encryption, prior to uploading to the sftp server.
Job
to dynamically perform ruby code on demand as a Batch
,
Often used for data correction or cleansing.
Example: Iterate over all rows in a table:
code = <<-CODE if user = User.find(row) user.cleanse_attributes! user.save(validate: false) end CODE job = RocketJob::Jobs::OnDemandBatchJob.new(code: code, description: 'cleanse users') arel = User.unscoped.all.order('updated_at DESC') job.upload_arel(arel) job.save!
Console Testing:
code = <<-CODE if user = User.find(row) user.cleanse_attributes! user.save(validate: false) end CODE job = RocketJob::Jobs::OnDemandBatchJob.new(code: code, description: 'cleanse users') # Run against a sub-set using a limit arel = User.unscoped.all.order('updated_at DESC').limit(100) job.upload_arel(arel) # Run the subset directly within the console job.perform_now job.cleanup!
By default output is not collected, call the method `#collect_output` to collect output.
Example:
job = RocketJob::Jobs::OnDemandBatchJob( description: 'Fix data', code: code, throttle_running_workers: 5, priority: 30 ) job.collect_output job.save!
Example: Move the upload operation into a before_batch.
upload_code = <<-CODE arel = User.unscoped.all.order('updated_at DESC') upload_arel(arel) CODE code = <<-CODE if user = User.find(row) user.cleanse_attributes! user.save(validate: false) end CODE RocketJob::Jobs::OnDemandBatchJob.create!( upload_code: upload_code, code: code, description: 'cleanse users' )
Job
to dynamically perform ruby code on demand,
Create or schedule a generalized job for one off fixes or cleanups.
Example: Iterate over all rows in a table:
code = <<~CODE User.unscoped.all.order('updated_at DESC').each |user| user.cleanse_attributes! user.save! end CODE RocketJob::Jobs::OnDemandJob.create!( code: code, description: 'Cleanse users' )
Example: Test job in a console:
code = <<~CODE User.unscoped.all.order('updated_at DESC').each |user| user.cleanse_attributes! user.save! end CODE job = RocketJob::Jobs::OnDemandJob.new(code: code, description: 'cleanse users') job.perform_now
Example: Pass input data:
code = <<~CODE puts data['a'] * data['b'] CODE RocketJob::Jobs::OnDemandJob.create!( code: code, data: {'a' => 10, 'b' => 2} )
Example: Retain output:
code = <<~CODE data['result'] = data['a'] * data['b'] CODE RocketJob::Jobs::OnDemandJob.create!( code: code, data: {'a' => 10, 'b' => 2} )
Example: Schedule the job to run nightly at 2am Eastern:
RocketJob::Jobs::OnDemandJob.create!( cron_schedule: '0 2 * * * America/New_York', code: code )
Example: Change the job priority, description, etc.
RocketJob::Jobs::OnDemandJob.create!( code: code, description: 'Cleanse users', priority: 30 )
Example: Automatically retry up to 5 times on failure:
RocketJob::Jobs::OnDemandJob.create!( retry_limit: 5 code: code )
Worker
behavior for a job
@formatter:off
Constants
- VERSION
Public Class Methods
# File lib/rocket_job/rocket_job.rb, line 2 def self.create_indexes # Ensure models with indexes are loaded into memory first Job.create_indexes Server.create_indexes DirmonEntry.create_indexes end
# File lib/rocket_job/rocket_job.rb, line 57 def self.rails! @rails = true end
When running inside a Rocket Job
server process, returns true when Rails has been initialized.
# File lib/rocket_job/rocket_job.rb, line 16 def self.rails? @rails end
Returns a human readable duration from the supplied [Float] number of seconds
# File lib/rocket_job/rocket_job.rb, line 27 def self.seconds_as_duration(seconds) return nil unless seconds if seconds >= 86_400.0 # 1 day "#{(seconds / 86_400).to_i}d #{Time.at(seconds).strftime('%-Hh %-Mm')}" elsif seconds >= 3600.0 # 1 hour Time.at(seconds).strftime("%-Hh %-Mm") elsif seconds >= 60.0 # 1 minute Time.at(seconds).strftime("%-Mm %-Ss") elsif seconds >= 1.0 # 1 second format("%.3fs", seconds) else duration = seconds * 1000 if defined? JRuby "#{duration.to_i}ms" else duration < 10.0 ? format("%.3fms", duration) : format("%.1fms", duration) end end end
# File lib/rocket_job/rocket_job.rb, line 53 def self.server! @server = true end
Whether the current process is running inside a Rocket Job
server process.
# File lib/rocket_job/rocket_job.rb, line 10 def self.server? @server end
When running inside a Rocket Job
server process, returns true when running standalone.
# File lib/rocket_job/rocket_job.rb, line 22 def self.standalone? !@rails end