class Omnibus::Builder
Attributes
@return [Software]
the software definition that created this builder
Public Class Methods
Create a new builder object for evaluation.
@param [Software] software
the software definition that created this builder
# File lib/omnibus/builder.rb, line 57 def initialize(software) @software = software end
Public Instance Methods
Execute the given appbundler command against the embedded Ruby’s appbundler. This command assumes the appbundle
gem is installed and in the embedded Ruby. You should add a dependency on the appbundler
software definition if you want to use this command.
@example
appbundle 'chef'
@param software_name [String]
The omnibus software definition name that you want to appbundle. We assume that this software definition is a gem that already has `bundle install` ran on it so it has a Gemfile.lock to appbundle.
@param (see command
) @return (see command
)
# File lib/omnibus/builder.rb, line 411 def appbundle(software_name, options = {}) build_commands << BuildCommand.new("appbundle `#{software_name}'") do bin_dir = "#{install_dir}/bin" appbundler_bin = embedded_bin("appbundler") lockdir = options.delete(:lockdir) gem = options.delete(:gem) without = options.delete(:without) extra_bin_files = options.delete(:extra_bin_files) lockdir ||= begin app_software = project.softwares.find do |p| p.name == software_name end if app_software.nil? raise "could not find software definition for #{software_name}, add a dependency to it, or pass a lockdir argument to appbundle command." end app_software.project_dir end command = [ appbundler_bin, "'#{lockdir}'", "'#{bin_dir}'" ] # This option is almost entirely for support of ChefDK and enables transitive gemfile lock construction in order # to be able to decouple the dev gems for all the different components of ChefDK. AKA: don't use it outside of # ChefDK. You should also explicitly specify the lockdir when going down this road. command << [ "'#{gem}'" ] if gem # FIXME: appbundler lacks support for this argument when not also specifying the gem (2-arg appbundling lacks support) # (if you really need this bug fixed, though, fix it in appbundler, don't try using the 3-arg version to try to # get `--without` support, you will likely wind up going down a sad path). command << [ "--without", without.join(",") ] unless without.nil? command << [ "--extra-bin-files", extra_bin_files.join(",") ] unless extra_bin_files.nil? || extra_bin_files.empty? # Ensure the main bin dir exists FileUtils.mkdir_p(bin_dir) shellout!(command.join(" "), options) end end
Execute the given Ruby block at runtime. The block is captured as-is and no validation is performed. As a general rule, you should avoid this method unless you know what you are doing.
@example
block do # Some complex operation end
@example
block 'Named operation' do # The above name can be used in log output to identify the operation end
@param (see command
) @return (see command
)
# File lib/omnibus/builder.rb, line 491 def block(name = "<Dynamic Ruby block>", &proc) build_commands << BuildCommand.new(name, &proc) end
Execute all the {BuildCommand} instances, in order, for this builder.
@return [void]
# File lib/omnibus/builder.rb, line 805 def build log.info(log_key) { "Starting build" } shasum # ensure shashum is calculated before build since the build can alter the shasum log.internal(log_key) { "Cached builder checksum before build: #{shasum}" } if software.overridden? log.info(log_key) do "Version overridden from #{software.default_version || "n/a"} to "\ "#{software.version}" end end measure("Build #{software.name}") do build_commands.each do |command| execute(command) end end log.info(log_key) { "Finished build" } end
Execute the given bundle command against the embedded Ruby’s bundler. This command assumes the bundler
gem is installed and in the embedded Ruby. You should add a dependency on the bundler
software definition if you want to use this command.
@example
bundle 'install'
@param (see command
) @return (see command
)
# File lib/omnibus/builder.rb, line 387 def bundle(command, options = {}) build_commands << BuildCommand.new("bundle `#{command}'") do bin = embedded_bin("bundle") shellout!("#{bin} #{command}", **options) end end
Execute the given command string or command arguments.
@example
command 'make install', env: { 'PATH' => '/my/custom/path' }
@param [String] command
the command to execute
@param [Hash] options
a list of options to pass to the +Mixlib::ShellOut+ instance when it is executed
@return [void]
# File lib/omnibus/builder.rb, line 81 def command(command, options = {}) warn_for_shell_commands(command) build_commands << BuildCommand.new("Execute: `#{command}'") do shellout!(command, **options) end end
(see Util#compiler_safe_path
)
Some compilers require paths to be formatted in certain ways. This helper takes in the standard Omnibus-style path and ensures that it is passed correctly.
@example
configure ["--prefix=#{compiler_safe_path(install_dir, "embedded")}"]
Omnibus::Util#compiler_safe_path
# File lib/omnibus/builder.rb, line 290 def compiler_safe_path(*pieces) super end
Run a prexisting “./configure” script that was generated by autotools. On windows, this will run configure within an msys bash shell with the given arguments. –build is also set on your behalf based on windows_arch. A default prefix of “#{install_bin}/embedded” is appended. It is important to set –build rather than –host because by default, –build also sets –host but it doesn’t trigger “cross-compilation” mode in most configure scripts. Triggering this mode can confuse certain software projects like Ruby which depend on the build platform in its mkmf scripts.
@example With no arguments
configure
On POSIX systems, this results in:
./configure --prefix=/path/to/embedded
On Windows 64-bit, this results in:
./configure --build=x86_64-w64-mingw32 --prefix=C:/path/to/embedded
Note that the windows case uses a windows compabile path with forward slashes - not an msys path. Ensure that the final Makefile is happy with this and doesn’t perform path gymnastics on it. Don’t pass \ paths unless you particularly enjoy discovering exactly how many times configure and the Makefile it generates sends your path back and forth through bash/sh, mingw32 native binaries and msys binaries and how many backslashes it takes for you to quit software development.
@example With custom arguments
configure '--enable-shared'
@example With custom location of configure bin
configure '--enable-shared', bin: '../foo/configure'
The path to configure must be a “unix-y” path for both windows and posix as this path is run under an msys bash shell on windows. Prefer relative paths lest you incur the wrath of the msys path gods for they are not kind, just or benevolent.
@param (see command
) @return (see command
)
# File lib/omnibus/builder.rb, line 167 def configure(*args) options = args.last.is_a?(Hash) ? args.pop : {} configure = options.delete(:bin) || "./configure" configure_cmd = [configure] # Pass the host platform as well. Different versions of config.guess # arrive at differently terrible wild ass guesses for what MSYSTEM=MINGW64 # means. This can be anything from x86_64-pc-mingw64 to i686-pc-mingw32 # which doesn't even make any sense... if windows? platform = windows_arch_i386? ? "i686-w64-mingw32" : "x86_64-w64-mingw32" configure_cmd << "--build=#{platform}" end # Accept a prefix override if provided. Can be set to '' to suppress # this functionality. prefix = options.delete(:prefix) || "#{install_dir}/embedded" configure_cmd << "--prefix=#{prefix}" if prefix && prefix != "" configure_cmd.concat args configure_cmd = configure_cmd.join(" ").strip options[:in_msys_bash] = true command(configure_cmd, options) end
Copy the given source to the destination. This method accepts a single file or a file pattern to match.
@param [String] source
the path on disk to copy from
@param [String] destination
the path on disk to copy to
@param (see mkdir
)
@return (see command
)
# File lib/omnibus/builder.rb, line 663 def copy(source, destination, options = {}) command = "copy `#{source}' to `#{destination}'" build_commands << BuildCommand.new(command) do Dir.chdir(software.project_dir) do files = FileSyncer.glob(source) if files.empty? log.warn(log_key) { "no matched files for glob #{command}" } else files.each do |file| FileUtils.cp_r(file, destination, **options) end end end end end
Delete the given file or directory on the system. This method uses the equivalent of +rm -rf+, so you may pass in a specific file or a glob of files.
@param [String] path
the path of the file to delete
@param (see mkdir
)
@return (see command
)
# File lib/omnibus/builder.rb, line 617 def delete(path, options = {}) build_commands << BuildCommand.new("delete `#{path}'") do Dir.chdir(software.project_dir) do FileSyncer.glob(path).each do |file| FileUtils.rm_rf(file, **options) end end end end
Render the erb template by the given name. This method will search all possible locations for an erb template (such as {Config#software_gems}).
@example
erb source: 'example.erb', dest: '/path/on/disk/to/render'
@example
erb source: 'example.erb', dest: '/path/on/disk/to/render', vars: { foo: 'bar' }, mode: '0755'
@param [Hash] options
the list of options
@option options [String] :source
the name of the patch to apply
@option options [String] :dest
the path on disk where the erb should be rendered
@option options [Hash] :vars
the list of variables to pass to the ERB rendering
@option options [String] :mode
the file mode for the rendered template (default varies by system)
@return (see command
)
# File lib/omnibus/builder.rb, line 524 def erb(options = {}) source = options.delete(:source) dest = options.delete(:dest) mode = options.delete(:mode) || 0644 vars = options.delete(:vars) || {} raise "Missing required option `:source'!" unless source raise "Missing required option `:dest'!" unless dest locations, source_path = find_file("config/templates", source) unless source_path raise MissingTemplate.new(source, locations) end erbs << source_path block "Render erb `#{source}'" do render_template(source_path, destination: dest, mode: mode, variables: vars) end end
Execute the given Rubygem command against the embedded Rubygems.
@example
gem 'install chef'
@param (see command
) @return (see command
)
# File lib/omnibus/builder.rb, line 367 def gem(command, options = {}) build_commands << BuildCommand.new("gem `#{command}'") do bin = embedded_bin("gem") shellout!("#{bin} #{command}", **options) end end
Execute the given Go command or script against the embedded Go.
@example
go 'build -o hello'
@param (see command
) @return (see command
)
# File lib/omnibus/builder.rb, line 315 def go(command, options = {}) build_commands << BuildCommand.new("go `#{command}'") do bin = windows? ? windows_safe_path("#{install_dir}/embedded/go/bin/go") : embedded_bin("go") # Check if we are building a go binary and then check if we are on # Red Hat or CentOS so we build the binary properly with a build-id if command.start_with?("build", " build") && rhel? command << " -ldflags=-linkmode=external" end shellout!("#{bin} #{command}", options) end end
Link the given source to the destination. This method accepts a single file or a file pattern to match
@param [String] source
the path on disk to link from
@param [String] destination
the path on disk to link to
@param (see mkdir
)
@return (see command
)
# File lib/omnibus/builder.rb, line 721 def link(source, destination, options = {}) command = "link `#{source}' to `#{destination}'" build_commands << BuildCommand.new(command) do Dir.chdir(software.project_dir) do if options.delete(:unchecked) FileUtils.ln_s(source, destination, **options) else files = FileSyncer.glob(source) if files.empty? log.warn(log_key) { "no matched files for glob #{command}" } else files.each do |file| FileUtils.ln_s(file, destination, **options) end end end end end end
Execute the given make command. When present, this method will prefer the use of gmake
over make
. If applicable, this method will also set the ‘MAKE=gmake` environment variable when gmake is to be preferred.
On windows you need to have the msys-base package (or some equivalent) before you can invoke this.
@example With no arguments
make
@example With arguments
make 'install'
@example With custom make bin
make 'install', bin: '/path/to/custom/make'
@param (see command
) @return (see command
)
# File lib/omnibus/builder.rb, line 110 def make(*args) options = args.last.is_a?(Hash) ? args.pop : {} make = options.delete(:bin) || # Prefer gmake on non-windows environments. if !windows? && Omnibus.which("gmake") env = options.delete(:env) || {} env = { "MAKE" => "gmake" }.merge(env) options[:env] = env "gmake" else "make" end options[:in_msys_bash] = true make_cmd = ([make] + args).join(" ").strip command(make_cmd, options) end
Make a directory at runtime. This method uses the equivalent of +mkdir -p+ under the covers.
@param [String] directory
the name or path of the directory to create
@param [Hash] options
the list of options to pass to the underlying +FileUtils+ call
@return (see command
)
# File lib/omnibus/builder.rb, line 575 def mkdir(directory, options = {}) build_commands << BuildCommand.new("mkdir `#{directory}'") do Dir.chdir(software.project_dir) do FileUtils.mkdir_p(directory, **options) end end end
Move the given source to the destination. This method accepts a single file or a file pattern to match
@param [String] source
the path on disk to move from
@param [String] destination
the path on disk to move to
@param (see mkdir
)
@return (see command
)
# File lib/omnibus/builder.rb, line 692 def move(source, destination, options = {}) command = "move `#{source}' to `#{destination}'" build_commands << BuildCommand.new(command) do Dir.chdir(software.project_dir) do files = FileSyncer.glob(source) if files.empty? log.warn(log_key) { "no matched files for glob #{command}" } else files.each do |file| FileUtils.mv(file, destination, **options) end end end end end
Apply the patch by the given name. This method will search all possible locations for a patch (such as {Config#software_gems}).
On windows, you must have the the patch package installed before you can invoke this.
@example
patch source: 'ncurses-clang.patch'
@example
patch source: 'patch-ad', plevel: 0
@param [Hash] options
the list of options
@option options [String] :source
the name of the patch to apply
@option options [Fixnum] :plevel
the level to apply the patch
@option options [String] :target
the destination to apply the patch
@return (see command
)
# File lib/omnibus/builder.rb, line 220 def patch(options = {}) source = options.delete(:source) plevel = options.delete(:plevel) || 1 target = options.delete(:target) locations, patch_path = find_file("config/patches", source) unless patch_path raise MissingPatch.new(source, locations) end # Using absolute paths to the patch when invoking patch from within msys # is going to end is tears and table-flips. Use relative paths instead. # It's windows - we don't reasonably expect symlinks to show up any-time # soon and if you're using junction points, you're on your own. clean_patch_path = patch_path if windows? clean_patch_path = Pathname.new(patch_path).relative_path_from( Pathname.new(software.project_dir) ).to_s end if target patch_cmd = "cat #{clean_patch_path} | patch -p#{plevel} #{target}" else patch_cmd = "patch -p#{plevel} -i #{clean_patch_path}" end patches << patch_path options[:in_msys_bash] = true build_commands << BuildCommand.new("Apply patch `#{source}'") do shellout!(patch_cmd, **options) end end
Execute the given Rake command against the embedded Ruby’s rake. This command assumes the rake
gem has been installed.
@example
rake 'test'
@param (see command
) @return (see command
)
# File lib/omnibus/builder.rb, line 465 def rake(command, options = {}) build_commands << BuildCommand.new("rake `#{command}'") do bin = embedded_bin("rake") shellout!("#{bin} #{command}", **options) end end
Execute the given Ruby command or script against the embedded Ruby.
@example
ruby 'setup.rb'
@param (see command
) @return (see command
)
# File lib/omnibus/builder.rb, line 350 def ruby(command, options = {}) build_commands << BuildCommand.new("ruby `#{command}'") do bin = embedded_bin("ruby") shellout!("#{bin} #{command}", **options) end end
The shasum for this builder object. The shasum is calculated using the following:
- The descriptions of all {BuildCommand} objects - The digest of all patch files on disk - The digest of all erb files on disk
@return [String]
# File lib/omnibus/builder.rb, line 835 def shasum @shasum ||= begin digest = Digest::SHA256.new build_commands.each do |build_command| update_with_string(digest, build_command.description) end patches.each do |patch_path| update_with_file_contents(digest, patch_path) end erbs.each do |erb_path| update_with_file_contents(digest, erb_path) end digest.hexdigest end end
Strip symbols from the given file or directory on the system. This method uses find and passes the matched files to strip through xargs, ignoring errors. So one may pass in a specific file/directory or a glob of files.
@param [String] path
the path of the file(s) to strip
@return (see command
)
# File lib/omnibus/builder.rb, line 638 def strip(path) regexp_ends = ".*(" + IGNORED_ENDINGS.map { |e| e.gsub(/\./, '\.') }.join("|") + ")$" regexp_patterns = IGNORED_SUBSTRINGS.map { |e| ".*" + e.gsub(%r{/}, '\/') + ".*" }.join("|") regexp = regexp_ends + "|" + regexp_patterns # Do not actually care if strip runs on non-strippable file, as its a no-op. Hence the `|| true` appended. # Do want to avoid stripping files unneccessarily so as not to slow down build process. find_command = "find #{path}/ -type f -regextype posix-extended ! -regex \"#{regexp}\" | xargs strip || true" options = { in_msys_bash: true } command(find_command, options) end
(see FileSyncer.sync
)
@example
sync "#{project_dir}/**/*.rb", "#{install_dir}/ruby_files"
@example
sync project_dir, "#{install_dir}/files", exclude: '.git'
# File lib/omnibus/builder.rb, line 751 def sync(source, destination, options = {}) build_commands << BuildCommand.new("sync `#{source}' to `#{destination}'") do Dir.chdir(software.project_dir) do FileSyncer.sync(source, destination, **options) end end end
Touch the given filepath at runtime. This method will also ensure the containing directory exists first.
@param [String] file
the path of the file to touch
@param (see mkdir
)
@return (see command
)
# File lib/omnibus/builder.rb, line 594 def touch(file, options = {}) build_commands << BuildCommand.new("touch `#{file}'") do Dir.chdir(software.project_dir) do parent = File.dirname(file) FileUtils.mkdir_p(parent) unless File.directory?(parent) FileUtils.touch(file, **options) end end end
Helper method to update config_guess in the software’s source directory. You should add a dependency on the config_guess
software definition if you want to use this command. @param [Hash] options
Supported options are: target [String] subdirectory under the software source to copy config.guess.to. Default: "." install [Array<Symbol>] parts of config.guess to copy. Default: [:config_guess, :config_sub]
# File lib/omnibus/builder.rb, line 771 def update_config_guess(target: ".", install: %i{config_guess config_sub}) build_commands << BuildCommand.new("update_config_guess `target: #{target} install: #{install.inspect}'") do config_guess_dir = "#{install_dir}/embedded/lib/config_guess" %w{config.guess config.sub}.each do |c| unless File.exist?(File.join(config_guess_dir, c)) raise "Can not find #{c}. Make sure you add a dependency on 'config_guess' in your software definition" end end destination = File.join(software.project_dir, target) FileUtils.mkdir_p(destination) FileUtils.cp_r("#{config_guess_dir}/config.guess", destination) if install.include? :config_guess FileUtils.cp_r("#{config_guess_dir}/config.sub", destination) if install.include? :config_sub end end
(see Util#windows_safe_path
)
Most internal Ruby methods will handle this automatically, but the command
method is unable to do so.
@example
command "#{windows_safe_path(install_dir)}\\embedded\\bin\\gem"
Omnibus::Util#windows_safe_path
# File lib/omnibus/builder.rb, line 275 def windows_safe_path(*pieces) super end
The maximum number of workers suitable for this system.
@see (Config#workers)
# File lib/omnibus/builder.rb, line 261 def workers Config.workers end
Private Instance Methods
The in-order list of {BuildCommand} for this builder.
@return [Array<BuildCommand>]
# File lib/omnibus/builder.rb, line 866 def build_commands @build_commands ||= [] end
The list of paths to erb files on disk. This is used in the calculation of the shasum.
@return [Array<String>]
# File lib/omnibus/builder.rb, line 886 def erbs @erbs ||= [] end
Execute the given command object. This method also wraps the following operations:
- Reset bundler's environment using {with_clean_env} - Instrument (time/measure) the individual command's execution - Retry failed commands in accordance with {Config#build_retries}
@param [BuildCommand] command
the command object to build
# File lib/omnibus/builder.rb, line 925 def execute(command) with_clean_env do measure(command.description) do with_retries do command.run(self) end end end end
Find a file amonst all local files, “remote” local files, and {Config#software_gems}.
@param [String] path
the path to find the file
@param [String] source
the source name of the file to find
@return [Array<Array<String>, String, nil>]
an array where the first entry is the list of candidate paths searched, and the second entry is the first occurence of the matched file (or +nil+) if one does not exist.
# File lib/omnibus/builder.rb, line 1021 def find_file(path, source) # Search for patches just like we search for software candidate_paths = Omnibus.possible_paths_for(path).map do |directory| File.join(directory, software.name, source) end file = candidate_paths.find { |path| File.exist?(path) } [candidate_paths, file] end
The log key for this class, overriden to incorporate the software name.
@return [String]
# File lib/omnibus/builder.rb, line 1037 def log_key @log_key ||= "#{super}: #{software.name}" end
The list of paths to patch files on disk. This is used in the calculation of the shasum.
@return [Array<String>]
# File lib/omnibus/builder.rb, line 876 def patches @patches ||= [] end
This is a helper method that wraps {Util#shellout!} for the purposes of setting the :cwd
value.
It also accepts an :in_msys_bash option which controls whether the given command is wrapped and run with bash.exe -c on windows.
@see (Util#shellout!
)
Omnibus::Util#shellout!
# File lib/omnibus/builder.rb, line 899 def shellout!(command_string, options = {}) # Make sure the PWD is set to the correct directory # Also make a clone of options so that we can mangle it safely below. options = { cwd: software.project_dir }.merge(options) # Set the log level to :info so users will see build commands options[:log_level] ||= :info # Set the live stream to :debug so users will see build output options[:live_stream] ||= log.live_stream(:debug) # Use Util's shellout super(command_string, **options) end
Inspect the given command and warn if the command “looks” like it is a shell command that has a DSL method. (like +command ‘cp’+ versus copy
).
@param [String] command
the command to check
@return [void]
# File lib/omnibus/builder.rb, line 1050 def warn_for_shell_commands(command) case command when /^cp /i log.warn(log_key) { "Detected command `cp'. Consider using the `copy' DSL method." } when /^rubocopy /i log.warn(log_key) { "Detected command `rubocopy'. Consider using the `sync' DSL method." } when /^mv /i log.warn(log_key) { "Detected command `mv'. Consider using the `move' DSL method." } when /^rm /i log.warn(log_key) { "Detected command `rm'. Consider using the `delete' DSL method." } when /^remove /i log.warn(log_key) { "Detected command `remove'. Consider using the `delete' DSL method." } when /^rsync /i log.warn(log_key) { "Detected command `rsync'. Consider using the `sync' DSL method." } when /^strip /i log.warn(log_key) { "Detected command `strip'. Consider using the `strip' DSL method." } end end
Execute the given command, removing any Ruby-specific environment variables. This is an “enhanced” version of Bundler.with_clean_env
, which only removes Bundler-specific values. We need to remove all values, specifically:
-
_ORIGINAL_GEM_PATH
-
GEM_PATH
-
GEM_HOME
-
GEM_ROOT
-
BUNDLE_BIN_PATH
-
BUNDLE_GEMFILE
-
RUBYLIB
-
RUBYOPT
-
RUBY_ENGINE
-
RUBY_ROOT
-
RUBY_VERSION
The original environment restored at the end of this call.
@param [Proc] block
the block to execute with the cleaned environment
# File lib/omnibus/builder.rb, line 993 def with_clean_env(&block) original = ENV.to_hash ENV.delete("_ORIGINAL_GEM_PATH") ENV.delete_if { |k, _| k.start_with?("BUNDLER_") } ENV.delete_if { |k, _| k.start_with?("BUNDLE_") } ENV.delete_if { |k, _| k.start_with?("GEM_") } ENV.delete_if { |k, _| k.start_with?("RUBY") } yield ensure ENV.replace(original.to_hash) end
Execute the given block with (n) reties defined by {Config#build_retries}. This method will only retry for the following exceptions:
- +CommandFailed+ - +CommandTimeout+
@param [Proc] block
the block to execute
# File lib/omnibus/builder.rb, line 945 def with_retries(&block) tries = Config.build_retries delay = 5 exceptions = [CommandFailed, CommandTimeout] begin yield rescue *exceptions => e if tries <= 0 raise e else delay *= 2 log.warn(log_key) do label = "#{(Config.build_retries - tries) + 1}/#{Config.build_retries}" "[#{label}] Failed to execute command. Retrying in #{delay} seconds..." end sleep(delay) tries -= 1 retry end end end