class Blat::Queue

The Blat::Queue class represents a download queue that handles requests using Curl::Multi. It, and its descendants, accept a large number of Curl::Easy objects and download them in parallel.

In order to know when each request has completed, use Curl::Easy::on_complete. This is made simpler by Queue#add, which will yield to a block on completion of each download.

Attributes

max_connections[R]
pipeline[R]

Public Class Methods

new(max_connections, pipeline = true) click to toggle source

Create a new Blat::Queue with a given number of maximum connections.

The ‘pipeline’ options controls Curl::Multi’s pipelining feature, which tries to use the same http connection for many requests to the same server.

# File lib/blat/queue.rb, line 22
def initialize(max_connections, pipeline = true)
  @multi = Curl::Multi.new

  # Set properties
  @max_connects         = max_connections.to_i
  @pipeline             = (pipeline == true)
  @multi.max_connects   = @max_connects
  @multi.pipeline       = @pipeline

  # Keep track of activity
  @active               = false
  @activity_mx          = Mutex.new
end

Public Instance Methods

active?() click to toggle source

Is this object currently actively downloading data?

# File lib/blat/queue.rb, line 115
def active?
  @activity_mx.synchronize { @active }
end
add(curl_or_link, &block) click to toggle source

Add a URL or a Curl::Easy object to the queue.

Optionally, provide a callback for calling when requests are complete, e.g.:

q.add('http://google.com') do |c|
  puts "Complete request: #{r}"
end
# File lib/blat/queue.rb, line 45
def add(curl_or_link, &block)
  # Convert to curl if necessary
  curl = curl_or_link.is_a?(Curl::Easy) ? curl_or_link : Curl::Easy.new(curl_or_link)
  curl.on_complete { |c| block.yield(c) } if block_given?

  # Add
  @multi.add(curl)

  # Return
  return curl
end
cancel() click to toggle source

Cancel all requests currently queued or downloading

# File lib/blat/queue.rb, line 58
def cancel
  @multi.cancel!
end
Also aliased as: cancel!
cancel!()
Alias for: cancel
idle?() click to toggle source

Is the queue idle?

# File lib/blat/queue.rb, line 120
def idle?
  @multi.idle?
end
perform() { || ... } click to toggle source

Run the queue, waiting for requests to finish (blocking).

If a block is given it is executed repeatedly whilst waiting, e.g.

q.perform {
  puts "Active downloads: #{q.request_count}"
}
# File lib/blat/queue.rb, line 89
def perform(&block)
  raise 'Already actively performing requests' if active?

  @activity_mx.synchronize { @active = true }
  @multi.perform do
    yield if block_given?
  end
ensure
  @activity_mx.synchronize { @active = false }
end
perform_nonblock() { || ... } click to toggle source

Perform downloads in a nonblocking manner

Optionally run the block given, as with regular perform

# File lib/blat/queue.rb, line 103
def perform_nonblock(&block)
  raise 'Currently active' if @thread

    me = self
    @thread = Thread.new() do
      me.perform { yield if block_given? }
    end
    @thread.abort_on_exception = true

end
remove(curl) click to toggle source

Remove a request from the queue.

This needn’t be called if a request has completed.

# File lib/blat/queue.rb, line 77
def remove(curl)
  @multi.remove(curl)
end
request_count() click to toggle source

Returns the number of active requests

# File lib/blat/queue.rb, line 65
def request_count
  requests.length
end
requests() click to toggle source

Returns a list of active requests (Curl::Easy objects)

# File lib/blat/queue.rb, line 70
def requests
  @multi.requests
end