class Scrapers::RubyTapas::DpdCart
DpdCart
is a remote service gateway object (Gateway Pattern) that provides a connection to rubytapas.dpdcart.com where the RubyTapas
episodes and download files are available, as well as the episode feed.
Constants
- CONTENT_PATH
- DPDCART_HOST_FORMAT
The subscription name will be filled in depending on which subscription I’m downloading from. This is a stock sprintf-type fill in where you pass in the subscription parameter with a value, thusly:
DPDCART_HOST % {subscription: "rubytapas"}
- ENV_DPDCART_PASSWORD_FORMAT
- ENV_DPDCART_USER_FORMAT
- FEED_PATH
- LOGIN_PATH
Attributes
Password for dpdcart acount
Subscription at dbdcart
User name for dpdcart account
Public Class Methods
Create a new instance of the DpdCart
gateway.
@param user [String] - the DpdCart
account name, typically an email address. @param password [String] - password associated with the account. @param subscription [String] - subscription name at DPD Cart (e.g. ‘rubytapas’ or ‘elixirsips’)
If the user and password are empty, the information will be obtained in the following order:
-
reading the environment variables ‘<subscriptiion>_USER` and
‘<subscription>_PASSWORD`
Note that <subscription> will be the subscription passed in above.
-
reading the user’s ‘$HOME/.netrc` file and pulling the
credentials that match the host name for the subscription account.
If no credentials can be found, it will raise an error: ‘NoCredentialsError`.
# File lib/scrapers/rubytapas/dpdcart.rb, line 74 def initialize(user=nil, password=nil, subscription='rubytapas', options={}) self.options = options self.subscription = subscription set_user_and_password(user, password) self.agent = Mechanize.new end
Public Instance Methods
# File lib/scrapers/rubytapas/dpdcart.rb, line 44 def debug ; @debug ||= options[:debug] ; end
Download
the file from dpdcart
# File lib/scrapers/rubytapas/dpdcart.rb, line 87 def download!(file) login warn "DEBUG: downloading #{file}" if debug if dry_run warn "DEBUG: download skipped for dry run" if dry_run filename = file body = "no body" else page = agent.get(file) filename = page.filename body = page.body end [ filename, body ] end
# File lib/scrapers/rubytapas/dpdcart.rb, line 41 def dpdcart_host ; @dpdcart_host ||= DPDCART_HOST_FORMAT % {subscription: subscription} ; end
# File lib/scrapers/rubytapas/dpdcart.rb, line 45 def dry_run ; @dry_run ||= options[:dry_run] ; end
# File lib/scrapers/rubytapas/dpdcart.rb, line 43 def env_dpdcart_password ; @env_dpdcart_password ||= ENV_DPDCART_PASSWORD_FORMAT % {subscription: subscription} ; end
# File lib/scrapers/rubytapas/dpdcart.rb, line 42 def env_dpdcart_user ; @env_dpdcart_user ||= ENV_DPDCART_PASSWORD_FORMAT % {subscription: subscription} ; end
Retreive the episode feed from dpdcart
# File lib/scrapers/rubytapas/dpdcart.rb, line 82 def feed! http_fetch(feed_url) end
# File lib/scrapers/rubytapas/dpdcart.rb, line 46 def feed_url ; @feed_url ||= URI("https://#{dpdcart_host}#{FEED_PATH}") ; end
# File lib/scrapers/rubytapas/dpdcart.rb, line 47 def login_url ; @login_url ||= URI("https://#{dpdcart_host}#{LOGIN_PATH}") ; end
Private Instance Methods
# File lib/scrapers/rubytapas/dpdcart.rb, line 118 def get_credentials_from_environment [ ENV[env_dpdcart_user], ENV[env_dpdcart_password] ] end
# File lib/scrapers/rubytapas/dpdcart.rb, line 122 def get_credentials_from_netrc creds = Netrc.read[dpdcart_host] if creds.nil? warn "Could not find credentials for #{dpdcart_host}" exit -1 end [ creds.login, creds.password ] end
# File lib/scrapers/rubytapas/dpdcart.rb, line 143 def http_fetch(uri) request = Net::HTTP::Get.new(uri) request.basic_auth user, password Net::HTTP.start(uri.host, uri.port, {:use_ssl => true}) {|http| http.request(request)}.body end
Login to dpdcart before downloading
# File lib/scrapers/rubytapas/dpdcart.rb, line 132 def login page = agent.get login_url page.form.field_with(name: "username").value = user page.form.field_with(name: "password").value = password page.form.submit unless agent.page.title.match(/Subscription Content/) raise "Could not log in" end agent end
# File lib/scrapers/rubytapas/dpdcart.rb, line 106 def set_user_and_password(user, password) if user && password @user = user @password = password else @user, @password = get_credentials_from_environment unless user && password @user, @password = get_credentials_from_netrc end end end