class Autobuild::Importer
Constants
- Hook
Attributes
The set of handlers registered by Importer.fallback
Changes whether this importer is interactive or not
@return [Hash] the original option hash as given to initialize
A list of hooks that are called after a successful checkout or update
They are added either at the instance level with {#add_post_hook} or globally for all importers of a given type with {Importer.add_post_hook}
Returns a string that identifies the remote repository uniquely
This can be used to check whether two importers are pointing to the same repository, regardless of e.g. the access protocol used. For instance, two git importers that point to the same repository but different branches would have the same repository_id
but different source_id
@return [String] @see source_id
Returns a string that identifies the remote source uniquely
This can be used to check whether two importers are pointing to the same code base inside the same repository. For instance, two git importers that point to the same repository but different branches would have the same repository_id
but different source_id
@return [String] @see repository_id
Public Class Methods
Define a post-import hook for all instances of this class
@yieldparam [Importer] importer the importer that finished @yieldparam [Package] package the package we're acting on @see Importer#add_post_hook
# File lib/autobuild/importer.rb, line 286 def self.add_post_hook(always: false, &hook) @post_hooks ||= Array.new @post_hooks << Hook.new(always, hook) nil end
The cache directories for the given importer type.
This is used by some importers to save disk space and/or avoid downloading the same things over and over again
The default global cache directory is initialized from the AUTOBUILD_CACHE_DIR environment variable. Per-importer cache directories can be overriden by setting AUTOBUILD_{TYPE}_CACHE_DIR (e.g. AUTOBUILD_GIT_CACHE_DIR)
The following importers use caches:
-
the archive importer saves downloaded files in the cache. They are saved under an archives/ subdirectory of the default cache if set, or to the value of AUTOBUILD_ARCHIVES_CACHE_DIR
-
the git importer uses the cache directories as alternates for the git checkouts
@param [String] type the importer type. If set, it Given a root cache
directory X, and importer specific cache is setup as a subdirectory of X with e.g. X/git or X/archives. The subdirectory name is defined by this argument
@return [nil,Array<String>]
@see .set_cache_dirs .default_cache_dirs .default_cache_dirs=
# File lib/autobuild/importer.rb, line 97 def self.cache_dirs(type) if @cache_dirs[type] || (env = ENV["AUTOBUILD_#{type.upcase}_CACHE_DIR"]) @cache_dirs[type] ||= env.split(":") elsif (dirs = default_cache_dirs) dirs.map { |d| File.join(d, type) } end end
Returns the default cache directory if there is one
@return [Array<String>,nil] @see .cache_dirs
# File lib/autobuild/importer.rb, line 109 def self.default_cache_dirs if @default_cache_dirs @default_cache_dirs elsif (from_env = ENV['AUTOBUILD_CACHE_DIR']) @default_cache_dirs = [from_env] end end
Sets the default cache directory
@param [Array<String>,String] the directories @see .cache_dirs
# File lib/autobuild/importer.rb, line 130 def self.default_cache_dirs=(dirs) @default_cache_dirs = Array(dirs) end
Enumerate the post-import hooks defined for all instances of this class
# File lib/autobuild/importer.rb, line 293 def self.each_post_hook(error: false) return enum_for(__method__) unless block_given? (@post_hooks ||= Array.new).each do |hook| yield(hook.callback) if hook.always || !error end end
If called, registers the given block as a fallback mechanism for failing imports.
Fallbacks are tried in reverse order with the failing importer object as argument. The first valid importer object that has been returned will be used instead.
It is the responsibility of the fallback handler to make sure that it does not do infinite recursions and stuff like that.
# File lib/autobuild/importer.rb, line 21 def self.fallback(&block) @fallback_handlers.unshift(block) end
Creates a new Importer
object. The options known to Importer
are:
- :patches
-
a list of patch to apply after import
More options are specific to each importer type.
# File lib/autobuild/importer.rb, line 149 def initialize(options) @options = options.dup @options[:retry_count] = Integer(@options[:retry_count] || 0) @repository_id = options[:repository_id] || "#{self.class.name}:#{object_id}" @interactive = options[:interactive] @source_id = options[:source_id] || @repository_id @post_hooks = Array.new end
Sets the cache directory for a given importer type
@param [String] type the importer type @param [String] dir the cache directory @see .cache_dirs
# File lib/autobuild/importer.rb, line 122 def self.set_cache_dirs(type, *dirs) @cache_dirs[type] = dirs end
Unset all cache directories
# File lib/autobuild/importer.rb, line 135 def self.unset_cache_dirs @cache_dirs = Hash.new @default_cache_dirs = nil end
Public Instance Methods
Add a block that should be called when the import has successfully finished
@yieldparam [Importer] importer the importer that finished @yieldparam [Package] package the package we're acting on @see Importer.add_post_hook
# File lib/autobuild/importer.rb, line 316 def add_post_hook(always: false, &hook) post_hooks << Hook.new(always, hook) end
# File lib/autobuild/importer.rb, line 529 def apply(package, path, patch_level = 0) call_patch(package, false, path, patch_level) end
# File lib/autobuild/importer.rb, line 523 def call_patch(package, reverse, file, patch_level) package.run(:patch, Autobuild.tool('patch'), "-p#{patch_level}", (reverse ? '-R' : nil), '--forward', input: file, working_directory: package.importdir) end
# File lib/autobuild/importer.rb, line 551 def currently_applied_patches(package) patches_file = patchlist(package) return parse_patch_list(package, patches_file) if File.exist?(patches_file) patches_file = File.join(package.importdir, "patches-autobuild-stamp") if File.exist?(patches_file) cur_patches = parse_patch_list(package, patches_file) save_patch_state(package, cur_patches) FileUtils.rm_f patches_file return currently_applied_patches(package) end [] end
Enumerate the post-import hooks for this importer
# File lib/autobuild/importer.rb, line 321 def each_post_hook(error: false, &block) return enum_for(__method__, error: false) unless block_given? self.class.each_post_hook(error: error, &block) post_hooks.each do |hook| yield(hook.callback) if hook.always || !error end end
@api private
Call the post-import hooks added with {#add_post_hook}
# File lib/autobuild/importer.rb, line 304 def execute_post_hooks(package, error: false) each_post_hook(error: error) do |block| block.call(self, package) end end
Tries to find a fallback importer because of the given error.
# File lib/autobuild/importer.rb, line 499 def fallback(error, package, *args, &block) Importer.fallback_handlers.each do |handler| fallback_importer = handler.call(package, self) if fallback_importer.kind_of?(Importer) begin return fallback_importer.send(*args, &block) rescue Exception raise error end end end raise error end
Returns a unique hash representing the state of the imported package as a whole unit, including its dependencies and patches
# File lib/autobuild/importer.rb, line 200 def fingerprint(package) vcs_fingerprint_string = vcs_fingerprint(package) return unless vcs_fingerprint_string patches_fingerprint_string = patches_fingerprint(package) if patches_fingerprint_string Digest::SHA1.hexdigest(vcs_fingerprint_string + patches_fingerprint_string) elsif patches.empty? vcs_fingerprint_string end end
Imports the given package
The importer will checkout or update code in package.importdir. No update will be done if {update?} returns false.
@raises ConfigException
if package.importdir exists and is not a directory
@option options [Boolean] :checkout_only (false) if true, the importer
will not update an already checked-out package.
@option options [Boolean] :only_local (false) if true, will only perform
actions that do not require network access. Importers that do not support this mode will simply do nothing
@option options [Boolean] :reset (false) if true, the importer's
configuration is interpreted as a hard state in which it should put the working copy. Otherwise, it tries to update the local repository with the remote information. For instance, a git importer for which a commit ID is given will, in this mode, reset the repository to the requested ID (if that does not involve losing commits). Otherwise, it will only ensure that the requested commit ID is present in the current HEAD.
# File lib/autobuild/importer.rb, line 461 def import( # rubocop:disable Metrics/ParameterLists package, *old_boolean, ignore_errors: false, checkout_only: false, allow_interactive: true, **options ) # Backward compatibility unless old_boolean.empty? old_boolean = old_boolean.first Autoproj.warn "calling #import with a boolean as second argument "\ "is deprecated, switch to the named argument interface instead" Autoproj.warn " e.g. call import(package, only_local: #{old_boolean})" Autoproj.warn " #{caller(1..1).first}" options[:only_local] = old_boolean end importdir = package.importdir if File.directory?(importdir) package.isolate_errors(mark_as_failed: false, ignore_errors: ignore_errors) do if !checkout_only && package.update? perform_update(package, checkout_only: false, **options) elsif Autobuild.verbose package.message "%s: not updating" end end elsif File.exist?(importdir) raise ConfigException.new(package, 'import'), "#{importdir} exists but is not a directory" else package.isolate_errors(mark_as_failed: true, ignore_errors: ignore_errors) do perform_checkout(package, allow_interactive: allow_interactive) true end end end
Whether this importer will need interaction with the user, for instance to give credentials
# File lib/autobuild/importer.rb, line 182 def interactive? @interactive end
# File lib/autobuild/importer.rb, line 537 def parse_patch_list(package, patches_file) File.readlines(patches_file).map do |line| line = line.rstrip if line =~ /^(.*)\s+(\d+)$/ path = File.expand_path($1, package.srcdir) level = Integer($2) else path = File.expand_path(line, package.srcdir) level = 0 end [path, level, File.read(path)] end end
# File lib/autobuild/importer.rb, line 566 def patch(package, patches = self.patches) # Get the list of already applied patches cur_patches = currently_applied_patches(package) cur_patches_state = cur_patches.map { |_, level, content| [level, content] } patches_state = patches.map { |_, level, content| [level, content] } return false if cur_patches_state == patches_state # Do not be smart, remove all already applied patches # and then apply the new ones begin apply_count = (patches - cur_patches).size unapply_count = (cur_patches - patches).size if apply_count > 0 && unapply_count > 0 package.message "patching %s: applying #{apply_count} and "\ "unapplying #{unapply_count} patch(es)" elsif apply_count > 0 package.message "patching %s: applying #{apply_count} patch(es)" elsif unapply_count > 0 package.message "patching %s: unapplying #{unapply_count} patch(es)" end while (p = cur_patches.last) p, level, = *p unapply(package, p, level) cur_patches.pop end patches.to_a.each do |new_patch, new_patch_level, content| apply(package, new_patch, new_patch_level) cur_patches << [new_patch, new_patch_level, content] end ensure save_patch_state(package, cur_patches) end true end
# File lib/autobuild/importer.rb, line 513 def patchdir(package) File.join(package.importdir, ".autobuild-patches") end
# File lib/autobuild/importer.rb, line 238 def patches patches = if @options[:patches].respond_to?(:to_ary) @options[:patches] elsif !@options[:patches] [] else [[@options[:patches], 0]] end single_patch = (patches.size == 2 && patches[0].respond_to?(:to_str) && patches[1].respond_to?(:to_int)) patches = [patches] if single_patch patches.map do |obj| if obj.respond_to?(:to_str) path = obj level = 0 elsif obj.respond_to?(:to_ary) path, level = obj else raise Arguments, "wrong patch specification #{obj.inspect}" end [path, level, File.read(path)] end end
fingerprint for patches associated to this package
# File lib/autobuild/importer.rb, line 222 def patches_fingerprint(package) cur_patches = currently_applied_patches(package) cur_patches.map(&:shift) # leave only level and source information if !patches.empty? && cur_patches Digest::SHA1.hexdigest(cur_patches.sort.flatten.join("")) end end
We assume that package.importdir already exists (checkout is supposed to have been called)
# File lib/autobuild/importer.rb, line 519 def patchlist(package) File.join(patchdir(package), "list") end
# File lib/autobuild/importer.rb, line 404 def perform_checkout(package, **options) last_error = nil package.progress_start "checking out %s", :done_message => 'checked out %s' do retry_count = 0 begin checkout(package, **options) execute_post_hooks(package) rescue Interrupt if last_error then raise last_error else raise end rescue ::Exception => e last_error = e retry_count = update_retry_count(e, retry_count) raise unless retry_count package.message "checkout of %s failed, "\ "deleting the source directory #{package.importdir} "\ "and retrying (#{retry_count}/#{self.retry_count})" FileUtils.rm_rf package.importdir retry end end patch(package) package.updated = true rescue Interrupt raise rescue ::Exception # rubocop:disable Lint/ShadowedException package.message "checkout of %s failed, "\ "deleting the source directory #{package.importdir}" FileUtils.rm_rf package.importdir raise rescue Autobuild::Exception => e FileUtils.rm_rf package.importdir fallback(e, package, :import, package) end
# File lib/autobuild/importer.rb, line 331 def perform_update(package, only_local = false) cur_patches = currently_applied_patches(package) needed_patches = patches patch_changed = cur_patches.map(&:last) != needed_patches.map(&:last) patch(package, []) if patch_changed last_error = nil retry_count = 0 package.progress_start "updating %s" begin begin did_update = update(package, only_local) execute_post_hooks(package, error: false) rescue ::Exception execute_post_hooks(package, error: true) raise end message = if did_update == false Autobuild.color('already up-to-date', :green) else Autobuild.color('updated', :yellow) end did_update rescue Interrupt message = Autobuild.color('interrupted', :red) if last_error raise last_error else raise end rescue ::Exception => e message = Autobuild.color('update failed', :red) last_error = e # If the package is patched, it might be that the update # failed because we needed to unpatch first. Try it out # # This assumes that importing data with conflict will # make the import fail, but not make the patch # un-appliable. Importers that do not follow this rule # will have to unpatch by themselves. cur_patches = currently_applied_patches(package) unless cur_patches.empty? package.progress_done package.message "update failed and some patches are applied, "\ "removing all patches and retrying" begin patch(package, []) return perform_update(package, only_local) rescue Interrupt raise rescue ::Exception raise e end end retry_count = update_retry_count(e, retry_count) raise unless retry_count package.message "update failed in #{package.importdir}, "\ "retrying (#{retry_count}/#{self.retry_count})" retry ensure package.progress_done "#{message} %s" end patch(package) package.updated = true did_update rescue Autobuild::Exception => e fallback(e, package, :import, package) end
The number of times update / checkout should be retried before giving up. The default is 0 (do not retry)
Set either with retry_count=
or by setting the :retry_count option when constructing this importer
# File lib/autobuild/importer.rb, line 194 def retry_count @options[:retry_count] || 0 end
Sets the number of times update / checkout should be retried before giving up. 0 (the default) disables retrying.
See also retry_count
# File lib/autobuild/importer.rb, line 234 def retry_count=(count) @options[:retry_count] = Integer(count) end
# File lib/autobuild/importer.rb, line 605 def save_patch_state(package, cur_patches) patch_dir = patchdir(package) FileUtils.mkdir_p patch_dir cur_patches = cur_patches.each_with_index. map do |(_path, level, content), idx| path = File.join(patch_dir, idx.to_s) File.open(path, 'w') do |patch_io| patch_io.write content end [path, level] end File.open(patchlist(package), 'w') do |f| patch_state = cur_patches.map do |path, level| path = Pathname.new(path). relative_path_from(Pathname.new(package.srcdir)).to_s "#{path} #{level}" end f.write(patch_state.join("\n")) end end
# File lib/autobuild/importer.rb, line 626 def supports_relocation? false end
# File lib/autobuild/importer.rb, line 533 def unapply(package, path, patch_level = 0) call_patch(package, true, path, patch_level) end
# File lib/autobuild/importer.rb, line 266 def update_retry_count(original_error, retry_count) return if !original_error.respond_to?(:retry?) || !original_error.retry? retry_count += 1 retry_count if retry_count <= self.retry_count end
basic fingerprint of the package and its dependencies
# File lib/autobuild/importer.rb, line 214 def vcs_fingerprint(package) # each importer type should implement its own Autoproj.warn "Fingerprint in #{package.name} has not been implemented "\ "for this type of packages, results should be discarded" nil end