class Autobuild::TarImporter
Constants
- Bzip
The tarball is compressed using bzip
- Gzip
The tarball is compressed with gzip
- Plain
rubocop:disable Naming/ConstantName The tarball is not compressed
- TAR_OPTION
rubocop:enable Naming/ConstantName
- VALID_URI_SCHEMES
Known URI schemes for
url
- WINDOWS_VALID_URI_SCHEMES
Known URI schemes for
url
on windows- Zip
Not a tarball but a zip
Attributes
Sets the directory in which files get cached
The number of time we should retry downloading if the underlying tool supports it (wget does).
It defaults to 1 as autobuild has its own retry mechanism
The timeout (in seconds) used during downloading.
With wget, it is the timeout used for DNS resolution, connection and idle time (time without receiving data)
It defaults to 10s
The directory in which remote files are cached
Defaults to ArchiveImporter.cachedir
The local file (either a downloaded file if url
is not local, or url
itself)
The SHA1 digest of the current cachefile. It is updated only once the cachefile has been downloaded
@return [String] hexadecimal SHA1 digest of the file
The filename that should be used locally (for remote files)
This is usually inferred by using the URL's basename, but some download URLs do not allow this (for instance bitbucket tarballs)
Change it by calling {relocate}
@retun [String]
The number of time we should retry downloading if the underlying tool supports it (wget does).
It defaults to the global ArchiveImporter.retries
The timeout (in seconds) used during downloading.
With wget, it is the timeout used for DNS resolution, connection and idle time (time without receiving data)
It defaults to the global ArchiveImporter.timeout
The source URL
Public Class Methods
# File lib/autobuild/import/archive.rb, line 103 def self.auto_update=(flag) @auto_update = flag end
Tells the importer that the checkout should be automatically deleted on update, without asking the user @return [Boolean] true if the archive importer should automatically
delete the current checkout when the archive changed, false otherwise. The default is to set it to true if the AUTOBUILD_ARCHIVE_AUTOUPDATE environment variable is set to 1, and to false in all other cases
# File lib/autobuild/import/archive.rb, line 99 def self.auto_update? @auto_update end
The directory in which downloaded files are saved
It defaults, if set, to the value returned by {Importer.cache_dirs} and falls back #{prefix}/cache
# File lib/autobuild/import/archive.rb, line 38 def cachedir if @cachedir then @cachedir elsif (cache_dirs = Importer.cache_dirs('archives')) @cachedir = cache_dirs.first else "#{Autobuild.prefix}/cache" end end
Returns the unpack mode from the file name
# File lib/autobuild/import/archive.rb, line 83 def self.filename_to_mode(filename) if (mode = find_mode_from_filename(filename)) mode else raise "cannot infer the archive type from '#{filename}', "\ "provide it explicitely with the mode: option" end end
Returns the unpack mode from the file name
@return [Integer,nil] either one of the pack constants (Zip
, Plain
,
...) or nil if it cannot be inferred
@see filename_to_mode
# File lib/autobuild/import/archive.rb, line 73 def self.find_mode_from_filename(filename) case filename when /\.zip$/ then Zip when /\.tar$/ then Plain when /\.tar\.gz$|\.tgz$/ then Gzip when /\.bz2$/ then Bzip end end
Creates a new importer which downloads url
in cachedir
and unpacks it. The following options are allowed:
- :cachedir
-
the cache directory. Defaults to “#{Autobuild.prefix}/cache”
- :archive_dir
-
the directory contained in the archive file. If set,
the importer will rename that directory to make it match Package#srcdir
- :no_subdirectory
-
the archive does not have the custom archive
subdirectory.
- :retries
-
The number of retries for downloading
- :timeout
-
The timeout (in seconds) used during downloading, it
defaults to 10s
- :filename
-
Rename the archive to this filename (in cache) – will be
also used to infer the mode
usually automatically inferred from the filename
# File lib/autobuild/import/archive.rb, line 387 def initialize(url, options = Hash.new) sourceopts, options = Kernel.filter_options( options, :source_id, :repository_id, :filename, :mode, :update_cached_file, :user, :password, :expected_digest ) super(options) @filename = nil @update_cached_file = false @cachedir = @options[:cachedir] || ArchiveImporter.cachedir @retries = @options[:retries] || ArchiveImporter.retries @timeout = @options[:timeout] || ArchiveImporter.timeout relocate(url, sourceopts) end
Public Instance Methods
Returns true if the archive that has been used to checkout this package is different from the one we are supposed to checkout now
# File lib/autobuild/import/archive.rb, line 482 def archive_changed?(package) checkout_digest = File.read(checkout_digest_stamp(package)).strip checkout_digest != cachefile_digest end
The directory contained in the archive. If not set, we assume that it is the same than the source dir
# File lib/autobuild/import/archive.rb, line 334 def archive_dir @options[:archive_dir] || tardir end
Changes the cache directory for this importer
# File lib/autobuild/import/archive.rb, line 322 def cachedir=(dir) @cachedir = dir relocate(@url.to_s) end
# File lib/autobuild/import/archive.rb, line 470 def checkout_digest_stamp(package) File.join(package.srcdir, "archive-autobuild-stamp") end
# File lib/autobuild/import/archive.rb, line 226 def download_from_url(package) FileUtils.mkdir_p(cachedir) begin if %w[http https].include?(@url.scheme) if File.file?(cachefile) return false unless update_cached_file? cached_mtime = File.lstat(cachefile).mtime end updated = download_http(package, @url, "#{cachefile}.partial", user: @user, password: @password, current_time: cached_mtime) return false unless updated elsif Autobuild.bsd? return false unless update_needed?(package) package.run(:import, Autobuild.tool('curl'), '-Lso', "#{cachefile}.partial", @url) else return false unless update_needed?(package) additional_options = [] if (timeout = self.timeout) additional_options << "--timeout" << timeout end if (retries = self.retries) additional_options << "--tries" << retries end package.run(:import, Autobuild.tool('wget'), '-q', '-P', cachedir, *additional_options, @url, '-O', "#{cachefile}.partial", retry: true) end rescue Exception FileUtils.rm_f "#{cachefile}.partial" raise end FileUtils.mv "#{cachefile}.partial", cachefile true end
# File lib/autobuild/import/archive.rb, line 114 def download_http(package, uri, filename, # rubocop:disable Metrics/ParameterLists user: nil, password: nil, current_time: nil) request = Net::HTTP::Get.new(uri) request['If-Modified-Since'] = current_time.rfc2822 if current_time request.basic_auth(user, password) if user Net::HTTP.start( uri.hostname, uri.port, use_ssl: uri.scheme == 'https') do |http| http.request(request) do |resp| case resp when Net::HTTPNotModified return false when Net::HTTPSuccess if current_time && (last_modified = resp['last-modified']) && (current_time >= Time.rfc2822(last_modified)) return false end if (length = resp['Content-Length']) length = Integer(length) expected_size = "/#{Autobuild.human_readable_size(length)}" end File.open(filename, 'wb') do |io| size = 0 next_update = Time.now resp.read_body do |chunk| io.write chunk size += chunk.size if size != 0 && (Time.now > next_update) formatted_size = Autobuild.human_readable_size(size) package.progress "downloading %s "\ "(#{formatted_size}#{expected_size})" next_update = Time.now + 1 end end formatted_size = Autobuild.human_readable_size(size) package.progress "downloaded %s "\ "(#{formatted_size}#{expected_size})" end when Net::HTTPRedirection if (location = resp['location']).start_with?('/') redirect_uri = uri.dup redirect_uri.path = resp['location'] else redirect_uri = location end return download_http(package, URI(redirect_uri), filename, user: user, password: password, current_time: current_time) else raise PackageException.new(package, 'import'), "failed download of #{package.name} from #{uri}: "\ "#{resp.class}" end end end true end
# File lib/autobuild/import/archive.rb, line 175 def extract_tar_gz(io, target) Gem::Package::TarReader.new(io).each do |entry| newname = File.join( target, File.basename(entry.full_name)) FileUtils.mkdir_p(newname) if entry.directory? if entry.file? dir = File.dirname(newname) FileUtils.mkdir_p(dir) unless File.directory?(dir) File.open(newname, "wb") do |file| file.write(entry.read) end end end end
Tests whether the archive's content is stored within a subdirectory or not
If it has a subdirectory, its name is assumed to be the package's basename, or the value returned by {archive_dir} if the archive_dir
option was given to {initialize}
# File lib/autobuild/import/archive.rb, line 368 def has_subdirectory? !@options[:no_subdirectory] end
# File lib/autobuild/import/archive.rb, line 284 def read_cachefile_digest Digest::SHA1.hexdigest File.read(cachefile) end
Changes the URL from which we should pick the archive
# File lib/autobuild/import/archive.rb, line 404 def relocate(url, options = Hash.new) parsed_url = URI.parse(url).normalize @url = parsed_url if !VALID_URI_SCHEMES.include?(@url.scheme) raise ConfigException, "invalid URL #{@url} (local files "\ "must be prefixed with file://)" elsif Autobuild.windows? unless WINDOWS_VALID_URI_SCHEMES.include?(@url.scheme) raise ConfigException, "downloading from a #{@url.scheme} URL "\ "is not supported on windows" end end @repository_id = options[:repository_id] || parsed_url.to_s @source_id = options[:source_id] || parsed_url.to_s @expected_digest = options[:expected_digest] @filename = options[:filename] || @filename || File.basename(url).gsub(/\?.*/, '') @update_cached_file = options[:update_cached_file] @mode = options[:mode] || ArchiveImporter.find_mode_from_filename(filename) || @mode if Autobuild.windows? && (mode != Gzip) raise ConfigException, "only gzipped tar archives "\ "are supported on Windows" end @user = options[:user] @password = options[:password] if @user && !%w[http https].include?(@url.scheme) raise ConfigException, "authentication is only supported for "\ "http and https URIs" end @cachefile = if @url.scheme == 'file' @url.path else File.join(cachedir, filename) end end
@deprecated use {#archive_dir} instead
# File lib/autobuild/import/archive.rb, line 328 def tardir @options[:tardir] end
Updates the downloaded file in cache only if it is needed
@return [Boolean] true if a new file was downloaded, false otherwise @raises ConfigException
if a expected digest was given in the
source.yml file and it doesn't match
# File lib/autobuild/import/archive.rb, line 272 def update_cache(package) updated = download_from_url(package) @cachefile_digest = read_cachefile_digest if @expected_digest && @expected_digest != @cachefile_digest raise ConfigException, "The archive #{@url} does not match the digest provided" end updated end
# File lib/autobuild/import/archive.rb, line 110 def update_cached_file? @update_cached_file end
# File lib/autobuild/import/archive.rb, line 190 def update_needed?(package) return true unless File.file?(cachefile) return false unless update_cached_file? cached_size = File.lstat(cachefile).size cached_mtime = File.lstat(cachefile).mtime size, mtime = nil if @url.scheme == "file" size = File.stat(@url.path).size mtime = File.stat(@url.path).mtime else # rubocop:disable Security/Open open @url, :content_length_proc => ->(v) { size = v } do |file| mtime = file.last_modified end # rubocop:enable Security/Open end if mtime && size size != cached_size || mtime > cached_mtime elsif mtime package.warn "%s: archive size is not available for #{@url}, "\ "relying on modification time" mtime > cached_mtime elsif size package.warn "%s: archive modification time "\ "is not available for #{@url}, relying on size" size != cached_size else package.warn "%s: neither the archive size nor its modification time "\ "are available for #{@url}, will always update" true end end
Fingerprint for archive importer, we are using its digest whether is calculated or expected @raises ConfigException
if no digest is present
# File lib/autobuild/import/archive.rb, line 291 def vcs_fingerprint(_package) if @cachefile_digest @cachefile_digest elsif File.file?(cachefile) read_cachefile_digest elsif @expected_digest @expected_digest else raise ConfigException, "There is no digest for archive #{@url}, make sure "\ "cache directories are configured correctly" end end
# File lib/autobuild/import/archive.rb, line 474 def write_checkout_digest_stamp(package) File.open(checkout_digest_stamp(package), 'w') do |io| io.write cachefile_digest end end