class Blat::ConsumingQueue
Similar to a queue, except that it explicitly calls a block in order to acquire new URLs.
This makes it suitable for use in producer/consumer patterns.
Public Instance Methods
consume(connections = @max_connects, &block)
click to toggle source
Executes the given block in order to keep the curl pool working at its maximum capacity.
consume blocks as long as links are being downloaded, as it relies on Curl::Multi#perform
Note that blocks providing links must also perform their own configuration, e.g.:
q.consume do url = get_url if(url) c = Curl::Easy.new(url) c.follow_location = true c.on_complete{ |c| puts "Retrieved: #{c.body_str}" } c else nil end end
# File lib/blat/queue.rb, line 153 def consume(connections = @max_connects, &block) perform do while request_count < connections && new_link = yield add(new_link) if new_link end end end