class Mechanize::HTTP::Agent

This monkey patch attempts to fix the insidious “too many connection resets” bug described here: github.com/sparklemotion/mechanize/issues/123

The code is taken and modified from this helpful blog article: scottwb.com/blog/2013/11/09/defeating-the-infamous-mechanize-too-many-connection-resets-bug/

Constants

IDEMPOTENT_HTTP_METHODS
MAX_CONNECTION_RESET_RETRIES

Public Instance Methods

fetch(uri, http_method = :get, headers = {}, params = [], referer = current_page, redirects = 0)
Also aliased as: fetch_without_retry
Alias for: fetch_with_retry
fetch_with_retry(uri, http_method = :get, headers = {}, params = [], referer = current_page, redirects = 0) click to toggle source

Replacement for +Mechanize::HTTP::Agent#fetch+. When a “too many connection resets” error is encountered, this method retries the request (upto {MAX_CONNECTION_RESET_RETRIES} times).

# File lib/grubby/mechanize/fetch_with_retry.rb, line 14
def fetch_with_retry(uri, http_method = :get, headers = {}, params = [], referer = current_page, redirects = 0)
  retry_count = 0
  begin
    fetch_without_retry(uri, http_method, headers, params, referer, redirects)
  rescue Net::HTTP::Persistent::Error => e
    # raise if different type of error
    raise unless e.message.include?("too many connection resets")
    # raise if non-idempotent http method
    raise unless IDEMPOTENT_HTTP_METHODS.include?(http_method)
    # raise if we've tried too many times
    raise if retry_count >= MAX_CONNECTION_RESET_RETRIES

    # otherwise, shutdown the persistent HTTP connection and try again
    retry_count += 1
    $log.warn("#{e.message} (#{e.class}).  Retry in #{retry_count} seconds.")
    sleep(retry_count) # incremental backoff to allow server to self-correct
    $log.warn("Retry #{http_method.to_s.upcase} #{uri}")
    retry
  end
end
Also aliased as: fetch
fetch_without_retry(uri, http_method = :get, headers = {}, params = [], referer = current_page, redirects = 0)
Alias for: fetch