class Blat::Queue
The Blat::Queue
class represents a download queue that handles requests using Curl::Multi. It, and its descendants, accept a large number of Curl::Easy objects and download them in parallel.
In order to know when each request has completed, use Curl::Easy::on_complete. This is made simpler by Queue#add
, which will yield to a block on completion of each download.
Attributes
Public Class Methods
Create a new Blat::Queue
with a given number of maximum connections.
The ‘pipeline’ options controls Curl::Multi’s pipelining feature, which tries to use the same http connection for many requests to the same server.
# File lib/blat/queue.rb, line 22 def initialize(max_connections, pipeline = true) @multi = Curl::Multi.new # Set properties @max_connects = max_connections.to_i @pipeline = (pipeline == true) @multi.max_connects = @max_connects @multi.pipeline = @pipeline # Keep track of activity @active = false @activity_mx = Mutex.new end
Public Instance Methods
Is this object currently actively downloading data?
# File lib/blat/queue.rb, line 115 def active? @activity_mx.synchronize { @active } end
Add a URL or a Curl::Easy object to the queue.
Optionally, provide a callback for calling when requests are complete, e.g.:
q.add('http://google.com') do |c| puts "Complete request: #{r}" end
# File lib/blat/queue.rb, line 45 def add(curl_or_link, &block) # Convert to curl if necessary curl = curl_or_link.is_a?(Curl::Easy) ? curl_or_link : Curl::Easy.new(curl_or_link) curl.on_complete { |c| block.yield(c) } if block_given? # Add @multi.add(curl) # Return return curl end
Cancel all requests currently queued or downloading
# File lib/blat/queue.rb, line 58 def cancel @multi.cancel! end
Is the queue idle?
# File lib/blat/queue.rb, line 120 def idle? @multi.idle? end
Run the queue, waiting for requests to finish (blocking).
If a block is given it is executed repeatedly whilst waiting, e.g.
q.perform { puts "Active downloads: #{q.request_count}" }
# File lib/blat/queue.rb, line 89 def perform(&block) raise 'Already actively performing requests' if active? @activity_mx.synchronize { @active = true } @multi.perform do yield if block_given? end ensure @activity_mx.synchronize { @active = false } end
Perform downloads in a nonblocking manner
Optionally run the block given, as with regular perform
# File lib/blat/queue.rb, line 103 def perform_nonblock(&block) raise 'Currently active' if @thread me = self @thread = Thread.new() do me.perform { yield if block_given? } end @thread.abort_on_exception = true end
Remove a request from the queue.
This needn’t be called if a request has completed.
# File lib/blat/queue.rb, line 77 def remove(curl) @multi.remove(curl) end
Returns the number of active requests
# File lib/blat/queue.rb, line 65 def request_count requests.length end
Returns a list of active requests (Curl::Easy objects)
# File lib/blat/queue.rb, line 70 def requests @multi.requests end