class Pupa::Processor::DocumentStore::RedisStore

Stores JSON documents in Redis.

Pupa flushes the JSON document store before scraping. If you use Redis, **DO NOT** share a Redis database with Pupa and other applications. You can select a different database than the default ‘0` for use with Pupa by passing an argument like `redis://localhost:6379/0`.

@note Redis support depends on the ‘redis-store` gem. You may optionally

use the `hiredis` gem to further improve performance.

Public Class Methods

new(address, pipelined: false) click to toggle source

@param [String] address the address (e.g. ‘redis://localhost:6379/0`)

in which to dump JSON documents

@param [Boolean] pipelined whether to enable

[pipelining](http://redis.io/topics/pipelining)
# File lib/pupa/processor/document_store/redis_store.rb, line 18
def initialize(address, pipelined: false)
  @pipelined = pipelined
  options = {marshalling: false}
  options.update(driver: :hiredis) if defined?(Hiredis)
  @redis = Redis::Store::Factory.create(address, options)
end

Public Instance Methods

clear() click to toggle source

Deletes all keys in the database.

# File lib/pupa/processor/document_store/redis_store.rb, line 93
def clear
  @redis.flushdb
end
delete(name) click to toggle source

Delete a key.

@param [String] name a key

# File lib/pupa/processor/document_store/redis_store.rb, line 88
def delete(name)
  @redis.del(name)
end
entries() click to toggle source

Returns all keys in the database.

@return [Array<String>] all keys in the store

# File lib/pupa/processor/document_store/redis_store.rb, line 36
def entries
  @redis.keys('*')
end
exist?(name) click to toggle source

Returns whether database contains an entry for the given key.

@param [String] name a key @return [Boolean] whether the store contains an entry for the given key

# File lib/pupa/processor/document_store/redis_store.rb, line 29
def exist?(name)
  @redis.exists(name)
end
pipelined() { || ... } click to toggle source

Collects commands to run all at once.

# File lib/pupa/processor/document_store/redis_store.rb, line 98
def pipelined
  if @pipelined
    @redis.pipelined do
      yield
    end
  else
    yield
  end
end
read(name) click to toggle source

Returns, as JSON, the value of the given key.

@param [String] name a key @return [Hash] the value of the given key

# File lib/pupa/processor/document_store/redis_store.rb, line 44
def read(name)
  Oj.load(@redis.get(name))
end
read_multi(names) click to toggle source

Returns, as JSON, the values of the given keys.

@param [String] names keys @return [Array<Hash>] the values of the given keys

# File lib/pupa/processor/document_store/redis_store.rb, line 52
def read_multi(names)
  @redis.mget(*names).map{|value| Oj.load(value)}
end
write(name, value) click to toggle source

Writes, as JSON, the value to a key.

@param [String] name a key @param [Hash] value a value

# File lib/pupa/processor/document_store/redis_store.rb, line 60
def write(name, value)
  @redis.set(name, Oj.dump(value, mode: :compat, time_format: :ruby))
end
write_multi(pairs) click to toggle source

Writes, as JSON, the values to keys.

@param [Hash] pairs key-value pairs

# File lib/pupa/processor/document_store/redis_store.rb, line 76
def write_multi(pairs)
  args = []
  pairs.each do |key,value|
    args << key
    args << Oj.dump(value, mode: :compat, time_format: :ruby)
  end
  @redis.mset(*args)
end
write_unless_exists(name, value) click to toggle source

Writes, as JSON, the value to a key, unless the key exists.

@param [String] name a key @param [Hash] value a value @return [Boolean] whether the key was set

# File lib/pupa/processor/document_store/redis_store.rb, line 69
def write_unless_exists(name, value)
  @redis.setnx(name, Oj.dump(value, mode: :compat, time_format: :ruby))
end