class Chronicle::ETL::Extractor

Abstract class representing an Extractor for an ETL job

Public Class Methods

new(options = {}) click to toggle source

Construct a new instance of this extractor. Options are passed in from a Runner

Paramters:

options

Options for configuring this Extractor

# File lib/chronicle/etl/extractors/extractor.rb, line 13
def initialize(options = {})
  @options = options.transform_keys!(&:to_sym)
  handle_continuation
end

Public Instance Methods

extract() click to toggle source

Entrypoint for this Extractor. Called by a Runner. Expects a series of records to be yielded

# File lib/chronicle/etl/extractors/extractor.rb, line 19
def extract
  raise NotImplementedError
end
results_count() click to toggle source

An optional method to calculate how many records there are to extract. Used primarily for building the progress bar

# File lib/chronicle/etl/extractors/extractor.rb, line 25
def results_count; end

Private Instance Methods

handle_continuation() click to toggle source
# File lib/chronicle/etl/extractors/extractor.rb, line 29
def handle_continuation
  return unless @options[:continuation]

  @options[:load_since] = @options[:continuation].highest_timestamp if @options[:continuation].highest_timestamp
  @options[:load_after_id] = @options[:continuation].last_id if @options[:continuation].last_id
end