class DaimonSkycrawlers::Processor::Base

The base class of processor

A processor implementation can inherit this class and override `#call` in the class.

Attributes

storage[W]

@!attribute [w] storage

Set storage to crawler instance.
@return [void]

Public Class Methods

new() click to toggle source
Calls superclass method DaimonSkycrawlers::Callbacks::new
# File lib/daimon_skycrawlers/processor/base.rb, line 25
def initialize
  super
  @skipped = false

  setup_default_filters
end

Public Instance Methods

call(message) click to toggle source

Process message

Override this method in subclass

@param message [Hash] parameters for processor

# File lib/daimon_skycrawlers/processor/base.rb, line 62
def call(message)
  raise "Implement this method in subclass"
end
process(message) click to toggle source

Process processor sequence

  1. Run registered filters

  2. Process HTTP response from message

@param message [Hash] parameters for processor

# File lib/daimon_skycrawlers/processor/base.rb, line 45
def process(message)
  @skipped = false
  proceeding = run_before_process_callbacks(message)
  unless proceeding
    skip(message[:url])
    return
  end
  call(message)
end
skipped?() click to toggle source

@private

# File lib/daimon_skycrawlers/processor/base.rb, line 33
def skipped?
  @skipped
end
storage() click to toggle source

Retrieve storage instance

# File lib/daimon_skycrawlers/processor/base.rb, line 69
def storage
  @storage ||= DaimonSkycrawlers::Storage::RDB.new
end

Private Instance Methods

setup_default_filters() click to toggle source
# File lib/daimon_skycrawlers/processor/base.rb, line 75
def setup_default_filters
  before_process do |m|
    !m[:heartbeat]
  end
end
skip(url) click to toggle source
# File lib/daimon_skycrawlers/processor/base.rb, line 81
def skip(url)
  log.info("Skipped '#{url}' by '#{self.class}'")
  @skipped = true
end