module Blat::Batch

Public Class Methods

run(max_connections, links, pipeline = true) { |c| ... } click to toggle source

Blat::Batch::run takes a list of links and downloads them all before returning. It is a very simple interface to Curl::Multi for smallish tasks.

max_connections

Defines how many parallel connections to use

links

Is the list of strings or Curl::Easy objects to download. The list object must support map and each

pipeline

Indicates if Curl::Multi should pipeline its HTTP requests

&block

If given, this block is called to configure each Curl::Easy object prior to it being pushed into the queue.

# File lib/blat/batch.rb, line 18
def self.run(max_connections, links, pipeline = true, &block)
  multi = Curl::Multi.new

  # Set options
  multi.max_connects  = max_connections.to_i
  multi.pipeline      = (pipeline == true)

  curls = links.map do |l|
    c = l
    c = Curl::Easy.new(l) unless l.is_a?(Curl::Easy)
    c
  end

  # Pump links in
  curls.each do |c|
    yield(c) if block_given?
    multi.add(c)
  end

  # Wait
  multi.perform

  return curls
end