class DaimonSkycrawlers::Storage::RDB

Storage for RDBMS

Public Class Methods

new(config_path = "config/database.yml") click to toggle source
Calls superclass method DaimonSkycrawlers::ConfigMixin::new
# File lib/daimon_skycrawlers/storage/rdb.rb, line 10
def initialize(config_path = "config/database.yml")
  super()
  Base.configurations = YAML.load(ERB.new(::File.read(config_path)).result)
  Base.establish_connection(DaimonSkycrawlers.env.to_sym)
end

Public Instance Methods

read(message = {}) click to toggle source

Fetch page identified by url

@param message [Hash] this hash can include `:url`, `:key` to find page

# File lib/daimon_skycrawlers/storage/rdb.rb, line 44
def read(message = {})
  url = message[:url]
  key = message[:key]
  if key
    Page.where(key: key).order(updated_at: :desc).limit(1).first
  else
    Page.where(url: url).order(updated_at: :desc).limit(1).first
  end
end
save(data) click to toggle source

Save data to RDB

@param data [Hash] data has following keys

* `:url`: URL
* `:message`: Given message
* `:response`: HTTP response
# File lib/daimon_skycrawlers/storage/rdb.rb, line 24
def save(data)
  url = data[:url]
  message = data[:message]
  key = message[:key] || url
  response = data[:response]
  headers = response.headers
  body = response.body
  Page.create(url: url,
              key: key,
              headers: JSON.generate(headers),
              body: body,
              last_modified_at: headers["last-modified"],
              etag: headers["etag"])
end