class Autobuild::Importer

Constants

Hook

Attributes

fallback_handlers[R]

The set of handlers registered by Importer.fallback

interactive[W]

Changes whether this importer is interactive or not

options[R]

@return [Hash] the original option hash as given to initialize

post_hooks[R]

A list of hooks that are called after a successful checkout or update

They are added either at the instance level with {#add_post_hook} or globally for all importers of a given type with {Importer.add_post_hook}

repository_id[R]

Returns a string that identifies the remote repository uniquely

This can be used to check whether two importers are pointing to the same repository, regardless of e.g. the access protocol used. For instance, two git importers that point to the same repository but different branches would have the same repository_id but different source_id

@return [String] @see source_id

source_id[R]

Returns a string that identifies the remote source uniquely

This can be used to check whether two importers are pointing to the same code base inside the same repository. For instance, two git importers that point to the same repository but different branches would have the same repository_id but different source_id

@return [String] @see repository_id

Public Class Methods

add_post_hook(always: false, &hook) click to toggle source

Define a post-import hook for all instances of this class

@yieldparam [Importer] importer the importer that finished @yieldparam [Package] package the package we're acting on @see Importer#add_post_hook

# File lib/autobuild/importer.rb, line 286
def self.add_post_hook(always: false, &hook)
    @post_hooks ||= Array.new
    @post_hooks << Hook.new(always, hook)
    nil
end
cache_dirs(type) click to toggle source

The cache directories for the given importer type.

This is used by some importers to save disk space and/or avoid downloading the same things over and over again

The default global cache directory is initialized from the AUTOBUILD_CACHE_DIR environment variable. Per-importer cache directories can be overriden by setting AUTOBUILD_{TYPE}_CACHE_DIR (e.g. AUTOBUILD_GIT_CACHE_DIR)

The following importers use caches:

  • the archive importer saves downloaded files in the cache. They are saved under an archives/ subdirectory of the default cache if set, or to the value of AUTOBUILD_ARCHIVES_CACHE_DIR

  • the git importer uses the cache directories as alternates for the git checkouts

@param [String] type the importer type. If set, it Given a root cache

directory X, and importer specific cache is setup as a subdirectory of X
with e.g. X/git or X/archives. The subdirectory name is defined by this
argument

@return [nil,Array<String>]

@see .set_cache_dirs .default_cache_dirs .default_cache_dirs=

# File lib/autobuild/importer.rb, line 97
def self.cache_dirs(type)
    if @cache_dirs[type] || (env = ENV["AUTOBUILD_#{type.upcase}_CACHE_DIR"])
        @cache_dirs[type] ||= env.split(":")
    elsif (dirs = default_cache_dirs)
        dirs.map { |d| File.join(d, type) }
    end
end
default_cache_dirs() click to toggle source

Returns the default cache directory if there is one

@return [Array<String>,nil] @see .cache_dirs

# File lib/autobuild/importer.rb, line 109
def self.default_cache_dirs
    if @default_cache_dirs
        @default_cache_dirs
    elsif (from_env = ENV['AUTOBUILD_CACHE_DIR'])
        @default_cache_dirs = [from_env]
    end
end
default_cache_dirs=(dirs) click to toggle source

Sets the default cache directory

@param [Array<String>,String] the directories @see .cache_dirs

# File lib/autobuild/importer.rb, line 130
def self.default_cache_dirs=(dirs)
    @default_cache_dirs = Array(dirs)
end
each_post_hook(error: false) { |callback| ... } click to toggle source

Enumerate the post-import hooks defined for all instances of this class

# File lib/autobuild/importer.rb, line 293
def self.each_post_hook(error: false)
    return enum_for(__method__) unless block_given?

    (@post_hooks ||= Array.new).each do |hook|
        yield(hook.callback) if hook.always || !error
    end
end
Autobuild::Importer.fallback { |package, importer| ... } click to toggle source

If called, registers the given block as a fallback mechanism for failing imports.

Fallbacks are tried in reverse order with the failing importer object as argument. The first valid importer object that has been returned will be used instead.

It is the responsibility of the fallback handler to make sure that it does not do infinite recursions and stuff like that.

# File lib/autobuild/importer.rb, line 21
def self.fallback(&block)
    @fallback_handlers.unshift(block)
end
new(options) click to toggle source

Creates a new Importer object. The options known to Importer are:

:patches

a list of patch to apply after import

More options are specific to each importer type.

# File lib/autobuild/importer.rb, line 149
def initialize(options)
    @options = options.dup
    @options[:retry_count] = Integer(@options[:retry_count] || 0)
    @repository_id = options[:repository_id] || "#{self.class.name}:#{object_id}"
    @interactive = options[:interactive]
    @source_id = options[:source_id] || @repository_id
    @post_hooks = Array.new
end
set_cache_dirs(type, *dirs) click to toggle source

Sets the cache directory for a given importer type

@param [String] type the importer type @param [String] dir the cache directory @see .cache_dirs

# File lib/autobuild/importer.rb, line 122
def self.set_cache_dirs(type, *dirs)
    @cache_dirs[type] = dirs
end
unset_cache_dirs() click to toggle source

Unset all cache directories

# File lib/autobuild/importer.rb, line 135
def self.unset_cache_dirs
    @cache_dirs = Hash.new
    @default_cache_dirs = nil
end

Public Instance Methods

add_post_hook(always: false, &hook) click to toggle source

Add a block that should be called when the import has successfully finished

@yieldparam [Importer] importer the importer that finished @yieldparam [Package] package the package we're acting on @see Importer.add_post_hook

# File lib/autobuild/importer.rb, line 316
def add_post_hook(always: false, &hook)
    post_hooks << Hook.new(always, hook)
end
apply(package, path, patch_level = 0) click to toggle source
# File lib/autobuild/importer.rb, line 529
def apply(package, path, patch_level = 0)
    call_patch(package, false, path, patch_level)
end
call_patch(package, reverse, file, patch_level) click to toggle source
# File lib/autobuild/importer.rb, line 523
def call_patch(package, reverse, file, patch_level)
    package.run(:patch, Autobuild.tool('patch'),
                "-p#{patch_level}", (reverse ? '-R' : nil), '--forward',
                input: file, working_directory: package.importdir)
end
currently_applied_patches(package) click to toggle source
# File lib/autobuild/importer.rb, line 551
def currently_applied_patches(package)
    patches_file = patchlist(package)
    return parse_patch_list(package, patches_file) if File.exist?(patches_file)

    patches_file = File.join(package.importdir, "patches-autobuild-stamp")
    if File.exist?(patches_file)
        cur_patches = parse_patch_list(package, patches_file)
        save_patch_state(package, cur_patches)
        FileUtils.rm_f patches_file
        return currently_applied_patches(package)
    end

    []
end
each_post_hook(error: false) { |callback| ... } click to toggle source

Enumerate the post-import hooks for this importer

# File lib/autobuild/importer.rb, line 321
def each_post_hook(error: false, &block)
    return enum_for(__method__, error: false) unless block_given?

    self.class.each_post_hook(error: error, &block)

    post_hooks.each do |hook|
        yield(hook.callback) if hook.always || !error
    end
end
execute_post_hooks(package, error: false) click to toggle source

@api private

Call the post-import hooks added with {#add_post_hook}

# File lib/autobuild/importer.rb, line 304
def execute_post_hooks(package, error: false)
    each_post_hook(error: error) do |block|
        block.call(self, package)
    end
end
fallback(error, package, *args, &block) click to toggle source

Tries to find a fallback importer because of the given error.

# File lib/autobuild/importer.rb, line 499
def fallback(error, package, *args, &block)
    Importer.fallback_handlers.each do |handler|
        fallback_importer = handler.call(package, self)
        if fallback_importer.kind_of?(Importer)
            begin
                return fallback_importer.send(*args, &block)
            rescue Exception
                raise error
            end
        end
    end
    raise error
end
fingerprint(package) click to toggle source

Returns a unique hash representing the state of the imported package as a whole unit, including its dependencies and patches

# File lib/autobuild/importer.rb, line 200
def fingerprint(package)
    vcs_fingerprint_string = vcs_fingerprint(package)
    return unless vcs_fingerprint_string

    patches_fingerprint_string = patches_fingerprint(package)
    if patches_fingerprint_string
        Digest::SHA1.hexdigest(vcs_fingerprint_string +
                               patches_fingerprint_string)
    elsif patches.empty?
        vcs_fingerprint_string
    end
end
import( package, *old_boolean, ignore_errors: false, checkout_only: false, allow_interactive: true, **options ) click to toggle source

Imports the given package

The importer will checkout or update code in package.importdir. No update will be done if {update?} returns false.

@raises ConfigException if package.importdir exists and is not a directory

@option options [Boolean] :checkout_only (false) if true, the importer

will not update an already checked-out package.

@option options [Boolean] :only_local (false) if true, will only perform

actions that do not require network access. Importers that do not
support this mode will simply do nothing

@option options [Boolean] :reset (false) if true, the importer's

configuration is interpreted as a hard state in which it should put the
working copy. Otherwise, it tries to update the local repository with
the remote information. For instance, a git importer for which a commit
ID is given will, in this mode, reset the repository to the requested ID
(if that does not involve losing commits). Otherwise, it will only
ensure that the requested commit ID is present in the current HEAD.
# File lib/autobuild/importer.rb, line 461
def import( # rubocop:disable Metrics/ParameterLists
    package, *old_boolean,
    ignore_errors: false, checkout_only: false, allow_interactive: true, **options
)
    # Backward compatibility
    unless old_boolean.empty?
        old_boolean = old_boolean.first
        Autoproj.warn "calling #import with a boolean as second argument "\
            "is deprecated, switch to the named argument interface instead"
        Autoproj.warn "   e.g. call import(package, only_local: #{old_boolean})"
        Autoproj.warn "   #{caller(1..1).first}"
        options[:only_local] = old_boolean
    end

    importdir = package.importdir
    if File.directory?(importdir)
        package.isolate_errors(mark_as_failed: false,
                               ignore_errors: ignore_errors) do
            if !checkout_only && package.update?
                perform_update(package, checkout_only: false, **options)
            elsif Autobuild.verbose
                package.message "%s: not updating"
            end
        end

    elsif File.exist?(importdir)
        raise ConfigException.new(package, 'import'),
              "#{importdir} exists but is not a directory"
    else
        package.isolate_errors(mark_as_failed: true,
                               ignore_errors: ignore_errors) do
            perform_checkout(package, allow_interactive: allow_interactive)
            true
        end
    end
end
interactive?() click to toggle source

Whether this importer will need interaction with the user, for instance to give credentials

# File lib/autobuild/importer.rb, line 182
def interactive?
    @interactive
end
parse_patch_list(package, patches_file) click to toggle source
# File lib/autobuild/importer.rb, line 537
def parse_patch_list(package, patches_file)
    File.readlines(patches_file).map do |line|
        line = line.rstrip
        if line =~ /^(.*)\s+(\d+)$/
            path = File.expand_path($1, package.srcdir)
            level = Integer($2)
        else
            path = File.expand_path(line, package.srcdir)
            level = 0
        end
        [path, level, File.read(path)]
    end
end
patch(package, patches = self.patches) click to toggle source
# File lib/autobuild/importer.rb, line 566
def patch(package, patches = self.patches)
    # Get the list of already applied patches
    cur_patches = currently_applied_patches(package)

    cur_patches_state = cur_patches.map { |_, level, content| [level, content] }
    patches_state     = patches.map { |_, level, content| [level, content] }
    return false if cur_patches_state == patches_state

    # Do not be smart, remove all already applied patches
    # and then apply the new ones
    begin
        apply_count = (patches - cur_patches).size
        unapply_count = (cur_patches - patches).size
        if apply_count > 0 && unapply_count > 0
            package.message "patching %s: applying #{apply_count} and "\
                "unapplying #{unapply_count} patch(es)"
        elsif apply_count > 0
            package.message "patching %s: applying #{apply_count} patch(es)"
        elsif unapply_count > 0
            package.message "patching %s: unapplying #{unapply_count} patch(es)"
        end

        while (p = cur_patches.last)
            p, level, = *p
            unapply(package, p, level)
            cur_patches.pop
        end

        patches.to_a.each do |new_patch, new_patch_level, content|
            apply(package, new_patch, new_patch_level)
            cur_patches << [new_patch, new_patch_level, content]
        end
    ensure
        save_patch_state(package, cur_patches)
    end

    true
end
patchdir(package) click to toggle source
# File lib/autobuild/importer.rb, line 513
def patchdir(package)
    File.join(package.importdir, ".autobuild-patches")
end
patches() click to toggle source
# File lib/autobuild/importer.rb, line 238
def patches
    patches =
        if @options[:patches].respond_to?(:to_ary)
            @options[:patches]
        elsif !@options[:patches]
            []
        else
            [[@options[:patches], 0]]
        end

    single_patch = (patches.size == 2 &&
        patches[0].respond_to?(:to_str) &&
        patches[1].respond_to?(:to_int))

    patches = [patches] if single_patch
    patches.map do |obj|
        if obj.respond_to?(:to_str)
            path  = obj
            level = 0
        elsif obj.respond_to?(:to_ary)
            path, level = obj
        else
            raise Arguments, "wrong patch specification #{obj.inspect}"
        end
        [path, level, File.read(path)]
    end
end
patches_fingerprint(package) click to toggle source

fingerprint for patches associated to this package

# File lib/autobuild/importer.rb, line 222
def patches_fingerprint(package)
    cur_patches = currently_applied_patches(package)
    cur_patches.map(&:shift) # leave only level and source information
    if !patches.empty? && cur_patches
      Digest::SHA1.hexdigest(cur_patches.sort.flatten.join(""))
    end
end
patchlist(package) click to toggle source

We assume that package.importdir already exists (checkout is supposed to have been called)

# File lib/autobuild/importer.rb, line 519
def patchlist(package)
    File.join(patchdir(package), "list")
end
perform_checkout(package, **options) click to toggle source
# File lib/autobuild/importer.rb, line 404
def perform_checkout(package, **options)
    last_error = nil
    package.progress_start "checking out %s", :done_message => 'checked out %s' do
        retry_count = 0
        begin
            checkout(package, **options)
            execute_post_hooks(package)
        rescue Interrupt
            if last_error then raise last_error
            else raise
            end
        rescue ::Exception => e
            last_error = e
            retry_count = update_retry_count(e, retry_count)
            raise unless retry_count

            package.message "checkout of %s failed, "\
                "deleting the source directory #{package.importdir} "\
                "and retrying (#{retry_count}/#{self.retry_count})"
            FileUtils.rm_rf package.importdir
            retry
        end
    end

    patch(package)
    package.updated = true
rescue Interrupt
    raise
rescue ::Exception # rubocop:disable Lint/ShadowedException
    package.message "checkout of %s failed, "\
        "deleting the source directory #{package.importdir}"
    FileUtils.rm_rf package.importdir
    raise
rescue Autobuild::Exception => e
    FileUtils.rm_rf package.importdir
    fallback(e, package, :import, package)
end
perform_update(package, only_local = false) click to toggle source
# File lib/autobuild/importer.rb, line 331
def perform_update(package, only_local = false)
    cur_patches    = currently_applied_patches(package)
    needed_patches = patches
    patch_changed = cur_patches.map(&:last) != needed_patches.map(&:last)
    patch(package, []) if patch_changed

    last_error = nil
    retry_count = 0
    package.progress_start "updating %s"
    begin
        begin
            did_update = update(package, only_local)
            execute_post_hooks(package, error: false)
        rescue ::Exception
            execute_post_hooks(package, error: true)
            raise
        end

        message = if did_update == false
                      Autobuild.color('already up-to-date', :green)
                  else
                      Autobuild.color('updated', :yellow)
                  end

        did_update
    rescue Interrupt
        message = Autobuild.color('interrupted', :red)
        if last_error
            raise last_error
        else raise
        end
    rescue ::Exception => e
        message = Autobuild.color('update failed', :red)
        last_error = e
        # If the package is patched, it might be that the update
        # failed because we needed to unpatch first. Try it out
        #
        # This assumes that importing data with conflict will
        # make the import fail, but not make the patch
        # un-appliable. Importers that do not follow this rule
        # will have to unpatch by themselves.
        cur_patches = currently_applied_patches(package)
        unless cur_patches.empty?
            package.progress_done
            package.message "update failed and some patches are applied, "\
                "removing all patches and retrying"
            begin
                patch(package, [])
                return perform_update(package, only_local)
            rescue Interrupt
                raise
            rescue ::Exception
                raise e
            end
        end

        retry_count = update_retry_count(e, retry_count)
        raise unless retry_count

        package.message "update failed in #{package.importdir}, "\
            "retrying (#{retry_count}/#{self.retry_count})"
        retry
    ensure
        package.progress_done "#{message} %s"
    end

    patch(package)
    package.updated = true
    did_update
rescue Autobuild::Exception => e
    fallback(e, package, :import, package)
end
retry_count() click to toggle source

The number of times update / checkout should be retried before giving up. The default is 0 (do not retry)

Set either with retry_count= or by setting the :retry_count option when constructing this importer

# File lib/autobuild/importer.rb, line 194
def retry_count
    @options[:retry_count] || 0
end
retry_count=(count) click to toggle source

Sets the number of times update / checkout should be retried before giving up. 0 (the default) disables retrying.

See also retry_count

# File lib/autobuild/importer.rb, line 234
def retry_count=(count)
    @options[:retry_count] = Integer(count)
end
save_patch_state(package, cur_patches) click to toggle source
# File lib/autobuild/importer.rb, line 605
def save_patch_state(package, cur_patches)
    patch_dir = patchdir(package)
    FileUtils.mkdir_p patch_dir
    cur_patches = cur_patches.each_with_index.
        map do |(_path, level, content), idx|
            path = File.join(patch_dir, idx.to_s)
            File.open(path, 'w') do |patch_io|
                patch_io.write content
            end
            [path, level]
        end
    File.open(patchlist(package), 'w') do |f|
        patch_state = cur_patches.map do |path, level|
            path = Pathname.new(path).
                relative_path_from(Pathname.new(package.srcdir)).to_s
            "#{path} #{level}"
        end
        f.write(patch_state.join("\n"))
    end
end
supports_relocation?() click to toggle source
# File lib/autobuild/importer.rb, line 626
def supports_relocation?
    false
end
unapply(package, path, patch_level = 0) click to toggle source
# File lib/autobuild/importer.rb, line 533
def unapply(package, path, patch_level = 0)
    call_patch(package, true, path, patch_level)
end
update_retry_count(original_error, retry_count) click to toggle source
# File lib/autobuild/importer.rb, line 266
def update_retry_count(original_error, retry_count)
    return if !original_error.respond_to?(:retry?) || !original_error.retry?

    retry_count += 1
    retry_count if retry_count <= self.retry_count
end
vcs_fingerprint(package) click to toggle source

basic fingerprint of the package and its dependencies

# File lib/autobuild/importer.rb, line 214
def vcs_fingerprint(package)
    # each importer type should implement its own
    Autoproj.warn "Fingerprint in #{package.name} has not been implemented "\
                  "for this type of packages, results should be discarded"
    nil
end