module GScraper
GScraper
- A web-scraping interface to various Google Services.
Copyright © 2007-2012 Hal Brodigan (postmodern.mod3 at gmail.com)
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
GScraper
- A web-scraping interface to various Google Services.
Copyright © 2007-2012 Hal Brodigan (postmodern.mod3 at gmail.com)
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
GScraper
- A web-scraping interface to various Google Services.
Copyright © 2007-2012 Hal Brodigan (postmodern.mod3 at gmail.com)
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
GScraper
- A web-scraping interface to various Google Services.
Copyright © 2007-2012 Hal Brodigan (postmodern.mod3 at gmail.com)
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
GScraper
- A web-scraping interface to various Google Services.
Copyright © 2007-2012 Hal Brodigan (postmodern.mod3 at gmail.com)
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
GScraper
- A web-scraping interface to various Google Services.
Copyright © 2007-2012 Hal Brodigan (postmodern.mod3 at gmail.com)
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
GScraper
- A web-scraping interface to various Google Services.
Copyright © 2007-2012 Hal Brodigan (postmodern.mod3 at gmail.com)
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
Constants
- COMMON_PROXY_PORT
Common proxy port.
- VERSION
The version of
GScraper
Public Class Methods
The proxy information.
@return [Hash]
# File lib/gscraper/gscraper.rb, line 34 def self.proxy @@gscraper_proxy ||= { :host => nil, :port => COMMON_PROXY_PORT, :user => nil, :password => nil } end
Creates a HTTP URI for the current proxy.
@param [Hash] proxy_info
The proxy information.
@option proxy_info [String] :host
The proxy host.
@option proxy_info [Integer] :port (COMMON_PROXY_PORT
)
The proxy port.
@option proxy_info [String] :user
The user-name to login as.
@option proxy_info [String] :password
The password to login with.
# File lib/gscraper/gscraper.rb, line 61 def self.proxy_uri(proxy=self.proxy) if proxy[:host] return URI::HTTP.build( :host => proxy[:host], :port => proxy[:port], :userinfo => "#{proxy[:user]}:#{proxy[:password]}", :path => '/' ) end end
The GScraper
User-Agent.
@return [String]
# File lib/gscraper/gscraper.rb, line 86 def self.user_agent @@gscraper_user_agent ||= self.user_agent_aliases['Windows IE 6'] end
Sets the GScraper
User-Agent.
@param [String] agent
The new User-Agent string.
@return [String]
The new User-Agent string.
# File lib/gscraper/gscraper.rb, line 99 def self.user_agent=(agent) @@gscraper_user_agent = agent end
Sets the GScraper
User-Agent.
@param [String] name
The User-Agent alias.
@return [String]
The new User-Agent string.
# File lib/gscraper/gscraper.rb, line 112 def self.user_agent_alias=(name) @@gscraper_user_agent = self.user_agent_aliases[name.to_s] end
The supported GScraper
User-Agent Aliases.
@return [Array<String>]
# File lib/gscraper/gscraper.rb, line 77 def self.user_agent_aliases Mechanize::AGENT_ALIASES end
Creates a new Mechanize agent.
@param [Hash] options
Additional options.
@option options [String] :user_agent_alias
The User-Agent Alias to use.
@option options [String] :user_agent
The User-Agent string to use.
@option options [Hash] :proxy
The proxy information to use.
@option :proxy [String] :host
The proxy host.
@option :proxy [Integer] :port
The proxy port.
@option :proxy [String] :user
The user-name to login as.
@option :proxy [String] :password
The password to login with.
@example
GScraper.web_agent
@example
GScraper.web_agent(:user_agent_alias => 'Linux Mozilla') GScraper.web_agent(:user_agent => 'Google Bot')
# File lib/gscraper/gscraper.rb, line 150 def self.web_agent(options={}) agent = Mechanize.new if options[:user_agent_alias] agent.user_agent_alias = options[:user_agent_alias] elsif options[:user_agent] agent.user_agent = options[:user_agent] elsif user_agent agent.user_agent = self.user_agent end proxy = (options[:proxy] || self.proxy) if proxy[:host] agent.set_proxy(proxy[:host],proxy[:port],proxy[:user],proxy[:password]) end yield agent if block_given? return agent end