class Blat::ConsumingQueue

Similar to a queue, except that it explicitly calls a block in order to acquire new URLs.

This makes it suitable for use in producer/consumer patterns.

Public Instance Methods

consume(connections = @max_connects, &block) click to toggle source

Executes the given block in order to keep the curl pool working at its maximum capacity.

consume blocks as long as links are being downloaded, as it relies on Curl::Multi#perform

Note that blocks providing links must also perform their own configuration, e.g.:

q.consume do
  url = get_url
  if(url)
    c = Curl::Easy.new(url)
    c.follow_location = true
    c.on_complete{ |c| puts "Retrieved: #{c.body_str}" }
    c
  else
    nil
  end
end
# File lib/blat/queue.rb, line 153
def consume(connections = @max_connects, &block)
  perform do
    while request_count < connections && new_link = yield
      add(new_link) if new_link
    end
  end
end