module GScraper

GScraper - A web-scraping interface to various Google Services.

Copyright © 2007-2012 Hal Brodigan (postmodern.mod3 at gmail.com)

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

GScraper - A web-scraping interface to various Google Services.

Copyright © 2007-2012 Hal Brodigan (postmodern.mod3 at gmail.com)

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

GScraper - A web-scraping interface to various Google Services.

Copyright © 2007-2012 Hal Brodigan (postmodern.mod3 at gmail.com)

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

GScraper - A web-scraping interface to various Google Services.

Copyright © 2007-2012 Hal Brodigan (postmodern.mod3 at gmail.com)

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

GScraper - A web-scraping interface to various Google Services.

Copyright © 2007-2012 Hal Brodigan (postmodern.mod3 at gmail.com)

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

GScraper - A web-scraping interface to various Google Services.

Copyright © 2007-2012 Hal Brodigan (postmodern.mod3 at gmail.com)

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

GScraper - A web-scraping interface to various Google Services.

Copyright © 2007-2012 Hal Brodigan (postmodern.mod3 at gmail.com)

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

Constants

COMMON_PROXY_PORT

Common proxy port.

VERSION

The version of GScraper

Public Class Methods

proxy() click to toggle source

The proxy information.

@return [Hash]

# File lib/gscraper/gscraper.rb, line 34
def self.proxy
  @@gscraper_proxy ||= {
    :host     => nil,
    :port     => COMMON_PROXY_PORT,
    :user     => nil,
    :password => nil
  }
end
proxy_uri(proxy=self.proxy) click to toggle source

Creates a HTTP URI for the current proxy.

@param [Hash] proxy_info

The proxy information.

@option proxy_info [String] :host

The proxy host.

@option proxy_info [Integer] :port (COMMON_PROXY_PORT)

The proxy port.

@option proxy_info [String] :user

The user-name to login as.

@option proxy_info [String] :password

The password to login with.
# File lib/gscraper/gscraper.rb, line 61
def self.proxy_uri(proxy=self.proxy)
  if proxy[:host]
    return URI::HTTP.build(
      :host     => proxy[:host],
      :port     => proxy[:port],
      :userinfo => "#{proxy[:user]}:#{proxy[:password]}",
      :path     => '/'
    )
  end
end
user_agent() click to toggle source

The GScraper User-Agent.

@return [String]

# File lib/gscraper/gscraper.rb, line 86
def self.user_agent
  @@gscraper_user_agent ||= self.user_agent_aliases['Windows IE 6']
end
user_agent=(agent) click to toggle source

Sets the GScraper User-Agent.

@param [String] agent

The new User-Agent string.

@return [String]

The new User-Agent string.
# File lib/gscraper/gscraper.rb, line 99
def self.user_agent=(agent)
  @@gscraper_user_agent = agent
end
user_agent_alias=(name) click to toggle source

Sets the GScraper User-Agent.

@param [String] name

The User-Agent alias.

@return [String]

The new User-Agent string.
# File lib/gscraper/gscraper.rb, line 112
def self.user_agent_alias=(name)
  @@gscraper_user_agent = self.user_agent_aliases[name.to_s]
end
user_agent_aliases() click to toggle source

The supported GScraper User-Agent Aliases.

@return [Array<String>]

# File lib/gscraper/gscraper.rb, line 77
def self.user_agent_aliases
  Mechanize::AGENT_ALIASES
end
web_agent(options={}) { |agent| ... } click to toggle source

Creates a new Mechanize agent.

@param [Hash] options

Additional options.

@option options [String] :user_agent_alias

The User-Agent Alias to use.

@option options [String] :user_agent

The User-Agent string to use.

@option options [Hash] :proxy

The proxy information to use.

@option :proxy [String] :host

The proxy host.

@option :proxy [Integer] :port

The proxy port.

@option :proxy [String] :user

The user-name to login as.

@option :proxy [String] :password

The password to login with.

@example

GScraper.web_agent

@example

GScraper.web_agent(:user_agent_alias => 'Linux Mozilla')
GScraper.web_agent(:user_agent => 'Google Bot')
# File lib/gscraper/gscraper.rb, line 150
def self.web_agent(options={})
  agent = Mechanize.new

  if options[:user_agent_alias]
    agent.user_agent_alias = options[:user_agent_alias]
  elsif options[:user_agent]
    agent.user_agent = options[:user_agent]
  elsif user_agent
    agent.user_agent = self.user_agent
  end

  proxy = (options[:proxy] || self.proxy)
  if proxy[:host]
    agent.set_proxy(proxy[:host],proxy[:port],proxy[:user],proxy[:password])
  end

  yield agent if block_given?
  return agent
end