class DaimonSkycrawlers::Processor::Base
The base class of processor
A processor implementation can inherit this class and override `#call` in the class.
Attributes
storage[W]
@!attribute [w] storage
Set storage to crawler instance. @return [void]
Public Class Methods
new()
click to toggle source
Calls superclass method
DaimonSkycrawlers::Callbacks::new
# File lib/daimon_skycrawlers/processor/base.rb, line 25 def initialize super @skipped = false setup_default_filters end
Public Instance Methods
call(message)
click to toggle source
Process message
Override this method in subclass
@param message [Hash] parameters for processor
# File lib/daimon_skycrawlers/processor/base.rb, line 62 def call(message) raise "Implement this method in subclass" end
process(message)
click to toggle source
Process processor sequence
-
Run registered filters
-
Process HTTP response from message
@param message [Hash] parameters for processor
# File lib/daimon_skycrawlers/processor/base.rb, line 45 def process(message) @skipped = false proceeding = run_before_process_callbacks(message) unless proceeding skip(message[:url]) return end call(message) end
skipped?()
click to toggle source
@private
# File lib/daimon_skycrawlers/processor/base.rb, line 33 def skipped? @skipped end
storage()
click to toggle source
Retrieve storage instance
# File lib/daimon_skycrawlers/processor/base.rb, line 69 def storage @storage ||= DaimonSkycrawlers::Storage::RDB.new end
Private Instance Methods
setup_default_filters()
click to toggle source
# File lib/daimon_skycrawlers/processor/base.rb, line 75 def setup_default_filters before_process do |m| !m[:heartbeat] end end
skip(url)
click to toggle source
# File lib/daimon_skycrawlers/processor/base.rb, line 81 def skip(url) log.info("Skipped '#{url}' by '#{self.class}'") @skipped = true end