class RestScrapyd
Don’t define a ‘RestScrapyd::Client` and include it in the class to avoid yardoc choking (plus two newlines to avoid this getting into the documentation)
Attributes
@!attribute project Default project name (can also be passed as hash argument to constructor).
Public Instance Methods
Add a version to a project, creating the project if it doesn’t exist. @param version [String] Project version @param egg [String, IO] Filename or file to upload, must be a Scrapy egg @return [Hash<String, Object>] Response, right now just +{“spiders”: <n>}+ @see scrapyd.readthedocs.org/en/latest/api.html#addversion-json
# File lib/rest-scrapyd.rb, line 38 def addversion(version, egg, project=self.project) egg = File.open(egg, 'rb') if egg.is_a? String post('addversion.json', project: project, version: version, egg: egg) end
Cancel a scheduled or running job @param job [String] Jobid to cancel @param project [String] Project name @return [String] Previous state of job @see scrapyd.readthedocs.org/en/latest/api.html#cancel-json
# File lib/rest-scrapyd.rb, line 68 def cancel(job, project=self.project) post('cancel.json', project: project, job: job)['prevstate'] end
Delete a project and all its uploaded versions. @param project [String] Project name @see scrapyd.readthedocs.org/en/latest/api.html#delproject-json
# File lib/rest-scrapyd.rb, line 114 def delproject(project) post('delproject.json', project: project) end
Delete a project version. @param version [String] Project version @param project [String] Project name @see scrapyd.readthedocs.org/en/latest/api.html#delversion-json
# File lib/rest-scrapyd.rb, line 107 def delversion(version, project=self.project) post('delversion.json', project: project, version: version) end
Get the list of pending, running and finished jobs of some project. @param project [String] Project name @return [Hash<String, Array<Hash>>] Jobs for project @see scrapyd.readthedocs.org/en/latest/api.html#listjobs-json
# File lib/rest-scrapyd.rb, line 99 def listjobs(project=self.project) get('listjobs.json', project: project).reject{|k,v| k=='status'} end
Get the list of projects uploaded to this Scrapy server. @return [Array<String>] Available projects @see scrapyd.readthedocs.org/en/latest/api.html#listprojects-json
# File lib/rest-scrapyd.rb, line 75 def listprojects get('listprojects.json')['projects'] end
Get the list of spiders available in the last version of some project. @param project [String] Project name @return [Array<String>] Available spiders for project @see scrapyd.readthedocs.org/en/latest/api.html#listspiders-json
# File lib/rest-scrapyd.rb, line 91 def listspiders(project=self.project) get('listspiders.json', project: project)['spiders'] end
Get the list of versions available for some project. @param project [String] Project name @return [Array<String>] Available versions for project @see scrapyd.readthedocs.org/en/latest/api.html#listversions-json
# File lib/rest-scrapyd.rb, line 83 def listversions(project=self.project) get('listversions.json', project: project)['versions'] end
Schedule a spider run (also known as a job). @param spider [String] Spider name @param version [String] Project version @param settings [Hash<String, String>] Additional Scrapy settings @param arguments [Hash<String, String>] Additional spider arguments @option arguments [String] :project ({self.project}) Project name @return [String] Job id @see scrapyd.readthedocs.org/en/latest/api.html#schedule-json
# File lib/rest-scrapyd.rb, line 51 def schedule(spider, version, settings={}, arguments={}) # +setting+ may appear multiple times, so can't use hash params = {project: self.project, spider: spider}.to_a params += settings.to_a.map{|k,v| ["setting", "#{k}=#{v}"]} params += arguments.to_a # rest-core doesn't (yet) support array payload, encode ourselves params.map{|p| p[0] = p[0].to_s} # like stringify_keys params = RestCore::Middleware.percent_encode(params) s_headers = headers.merge({'Content-Type' => 'application/x-www-form-urlencoded'}) post('schedule.json', params, {}, headers: s_headers)['jobid'] end