class Chronicle::ETL::JobLog

A record of what happened in the running of a job. We're interested in tracking when it ran, if it was successful, and what the latest record we found is (to use as a cursor for the next time)

Attributes

finished_at[RW]
highest_timestamp[RW]
job[RW]
job_id[RW]
last_id[RW]
num_records_processed[RW]
started_at[RW]
success[RW]

Public Class Methods

new() { |self| ... } click to toggle source

Create a new JobLog for a given Job

# File lib/chronicle/etl/job_log.rb, line 23
def initialize
  @num_records_processed = 0
  @success = false
  yield self if block_given?
end

Private Class Methods

build_from_serialized(attrs) click to toggle source

Create a new JobLog and set its instance variables from a serialized hash

# File lib/chronicle/etl/job_log.rb, line 74
def self.build_from_serialized attrs
  attrs.delete(:id)
  new do |job_log|
    attrs.each do |key, value|
      setter = "#{key.to_s}=".to_sym
      job_log.send(setter, value)
    end
  end
end

Public Instance Methods

finish() click to toggle source

Indicate that a job has finished

# File lib/chronicle/etl/job_log.rb, line 48
def finish
  @finished_at = Time.now
  @success = true
end
job=(job) click to toggle source
# File lib/chronicle/etl/job_log.rb, line 53
def job= job
  @job = job
  @job_id = job.id
end
log_transformation(transformer) click to toggle source

Log the result of a single transformation in a job @param transformer [Chronicle::ETL::Tranformer] The transformer that ran

# File lib/chronicle/etl/job_log.rb, line 31
def log_transformation(transformer)
  @last_id = transformer.id if transformer.id

  # Save the highest timestamp that we've encountered so far
  @highest_timestamp = [transformer.timestamp, @highest_timestamp].compact.max if transformer.timestamp

  # TODO: a transformer might yield nil. We might also want certain transformers to explode
  # records into multiple new ones. Therefore, this this variable will need more subtle behaviour
  @num_records_processed += 1
end
serialize() click to toggle source

Take a JobLog's instance variables and turn them into a hash representation

# File lib/chronicle/etl/job_log.rb, line 59
def serialize
  {
    job_id: @job_id,
    last_id: @last_id,
    highest_timestamp: @highest_timestamp,
    num_records_processed: @num_records_processed,
    started_at: @started_at,
    finished_at: @finished_at,
    success: @success
  }
end
start() click to toggle source

Indicate that a job has started

# File lib/chronicle/etl/job_log.rb, line 43
def start
  @started_at = Time.now
end