class Omnibus::Licensing

Constants

CACHE_DIRECTORY
OUTPUT_DIRECTORY
STANDARD_LICENSES

Attributes

dep_license_map[R]

Manifest data of transitive dependency licensing information

@return Hash

licensing_warnings[R]

The warnings encountered while preparing the licensing information

@return [Array<String>]

project[R]

The project to create licenses for.

@return [Project]

transitive_dependency_licensing_warnings[R]

The warnings encountered while preparing the licensing information for transitive dependencies.

@return [Array<String>]

Public Class Methods

create_incrementally(project) { |license_collector| ... } click to toggle source

Creates a new instance of Licensing, executes preparation steps, then yields control to a given block, and then creates a summary of the included licenses.

@example Building a project:

Licensing.create_incrementally(self) do |license_collector|
  softwares.each do |software|
    software.build_me([license_collector])
  end
end

@param [Project] project

The project being built.

@yieldparam [Licensing] license_collector

Yields an instance of Licensing. Call #execute_post_build to copy the
license files for a Software definition.

@return [Licensing]

# File lib/omnibus/licensing.rb, line 56
def create_incrementally(project)
  new(project).tap do |license_collector|

    license_collector.prepare
    license_collector.validate_license_info

    yield license_collector

    license_collector.process_transitive_dependency_licensing_info
    license_collector.create_project_license_file
    license_collector.raise_if_warnings_fatal!
  end
end
new(project) click to toggle source

@param [Project] project

the project to create licenses for.
# File lib/omnibus/licensing.rb, line 104
def initialize(project)
  @project = project
  @licensing_warnings = []
  @transitive_dependency_licensing_warnings = []
  @dep_license_map = {}
end

Public Instance Methods

cache_dir() click to toggle source

Cache directory where transitive dependency licenses will be collected in.

@return [String]

# File lib/omnibus/licensing.rb, line 375
def cache_dir
  File.expand_path(CACHE_DIRECTORY, project.install_dir)
end
cache_dir_gitkeep_file() click to toggle source

Path to a .gitkeep file we create in the cache dir so git caching doesn’t delete the directory.

@return [String]

# File lib/omnibus/licensing.rb, line 385
def cache_dir_gitkeep_file
  File.join(cache_dir, ".gitkeep")
end
components_license_summary() click to toggle source

Summary of the licenses included by the softwares of the project. It is in the form of: … This product bundles python 2.7.9, which is available under a “Python” License. For details, see: /opt/opscode/LICENSES/python-LICENSE …

@return [String]

# File lib/omnibus/licensing.rb, line 238
def components_license_summary
  out = "\n\n"

  license_map.keys.sort.each do |name|
    license = license_map[name][:license]
    license_files = license_map[name][:license_files]
    version = license_map[name][:version]

    out << "This product bundles #{name} #{version},\n"
    out << "which is available under a \"#{license}\" License.\n"
    unless license_files.empty?
      out << "For details, see:\n"
      license_files.each do |license_file|
        out << "#{license_package_location(name, license_file)}\n"
      end
    end
    out << "\n"
  end

  out
end
create_project_license_file() click to toggle source

Creates the top level license file for the project. Top level file is created at #{project.license_file_path} and contains the name of the project, version of the project, text of the license of the project and a summary of the licenses of the included software components.

@return [void]

# File lib/omnibus/licensing.rb, line 205
def create_project_license_file
  File.open(project.license_file_path, "w") do |f|
    f.puts "#{project.name} #{project.build_version} license: \"#{project.license}\""
    f.puts ""
    f.puts project_license_content
    f.puts ""
    f.puts components_license_summary
    f.puts ""
    f.puts dependencies_license_summary
  end
end
dependencies_license_summary() click to toggle source

Summary of the licenses of the transitive dependencies of the project. It is in the form of: … This product includes inifile 3.0.0 which is a ‘ruby_bundler’ dependency of ‘chef’, and which is available under a ‘MIT’ License. For details, see: /opt/opscode/LICENSES/ruby_bundler-inifile-3.0.0-README.md …

@return [String]

# File lib/omnibus/licensing.rb, line 273
def dependencies_license_summary
  out = "\n\n"

  dep_license_map.each do |dep_mgr_name, data|
    data.each do |dep_name, data|
      data.each do |dep_version, dep_data|
        projects = dep_data["dependency_of"].sort.map { |p| "'#{p}'" }.join(", ")
        files = dep_data["license_files"].map { |f| File.join(output_dir, f) }

        out << "This product includes #{dep_name} #{dep_version}\n"
        out << "which is a '#{dep_mgr_name}' dependency of #{projects},\n"
        out << "and which is available under a '#{dep_data["license"]}' License.\n"
        out << "For details, see:\n"
        out << files.join("\n")
        out << "\n\n"
      end
    end
  end

  out
end
execute_post_build(software) click to toggle source

Callback that gets called by Software#build_me after the build is done. Invokes license copying for the given software. This ensures that licenses are copied before a git cache snapshot is taken, so that the license files are correctly restored when a build is skipped due to a cache hit.

@param [Software] software

@return [void]

# File lib/omnibus/licensing.rb, line 145
def execute_post_build(software)
  collect_licenses_for(software)
  unless software.skip_transitive_dependency_licensing
    collect_transitive_dependency_licenses_for(software)
    check_transitive_dependency_licensing_errors_for(software)
  end
end
execute_pre_build(software) click to toggle source

Required callback to use instances of this class as a build wrapper for Software#build_me. Licensing doesn’t need to do anything pre-build, so this does nothing.

@param [Software] software

@return [void]

# File lib/omnibus/licensing.rb, line 133
def execute_pre_build(software); end
license_map() click to toggle source

Map that collects information about the licenses of the softwares included in the project.

@example {

...
"python" => {
  "license" => "Python",
  "license_files" => "LICENSE",
  "version" => "2.7.9",
  "project_dir" => "/var/cache/omnibus/src/python/Python-2.7.9/"
},
...

}

@return [Hash]

# File lib/omnibus/licensing.rb, line 313
def license_map
  @license_map ||= begin
    map = {}

    project.library.each do |component|
      # Some of the components do not bundle any software but contain
      # some logic that we use during the build. These components are
      # covered under the project's license and they do not need specific
      # license files.
      next if component.license == :project_license

      map[component.name] = {
        license: component.license,
        license_files: component.license_files,
        version: component.version,
        project_dir: component.project_dir,
      }
    end

    map
  end
end
license_package_location(component_name, where) click to toggle source

Returns the location where the license file should reside in the package. License file is named as <project_name>-<license_file_name> and created under the output licenses directory.

@return [String]

# File lib/omnibus/licensing.rb, line 343
def license_package_location(component_name, where)
  if local?(where)
    File.join(output_dir, "#{component_name}-#{File.split(where).last}")
  else
    u = URI(where)
    File.join(output_dir, "#{component_name}-#{File.basename(u.path)}")
  end
end
licensing_info(message) click to toggle source

Logs the given message as info.

This method should only be used for detecting in a license is known or not. In the future, we will introduce a configurable way to whitelist or blacklist the allowed licenses. Once we implement that we need to stop using this method.

@param [String] message

message to log as warning
# File lib/omnibus/licensing.rb, line 408
def licensing_info(message)
  log.info(log_key) { message }
end
licensing_warning(message) click to toggle source

Logs the given message as warning or fails the build depending on the :fatal_licensing_warnings configuration setting.

@param [String] message

message to log as warning
# File lib/omnibus/licensing.rb, line 418
def licensing_warning(message)
  licensing_warnings << message
  log.warn(log_key) { message }
end
local?(license) click to toggle source

Returns if the given path to a license is local or a remote url.

@return [Boolean]

# File lib/omnibus/licensing.rb, line 394
def local?(license)
  u = URI(license)
  u.scheme.nil?
end
output_dir() click to toggle source

Output directory to create the licenses in.

@return [String]

# File lib/omnibus/licensing.rb, line 357
def output_dir
  File.expand_path(OUTPUT_DIRECTORY, project.install_dir)
end
output_dir_gitkeep_file() click to toggle source

Path to a .gitkeep file we create in the output dir so git caching doesn’t delete the directory.

@return [String]

# File lib/omnibus/licensing.rb, line 367
def output_dir_gitkeep_file
  File.join(output_dir, ".gitkeep")
end
prepare() click to toggle source

Creates the required directories for licenses.

@return [void]

# File lib/omnibus/licensing.rb, line 116
def prepare
  FileUtils.rm_rf(output_dir)
  FileUtils.mkdir_p(output_dir)
  FileUtils.touch(output_dir_gitkeep_file)
  FileUtils.rm_rf(cache_dir)
  FileUtils.mkdir_p(cache_dir)
  FileUtils.touch(cache_dir_gitkeep_file)
end
process_transitive_dependency_licensing_info() click to toggle source
  1. Translate all transitive dependency licensing issues into omnibus warnings

  2. Parse all the licensing information for all software from ‘cache_dir’

  3. Merge and drop the duplicates

  4. Add these licenses to the main manifest, to be merged with the main

licensing information from software definitions.

# File lib/omnibus/licensing.rb, line 454
def process_transitive_dependency_licensing_info
  Dir.glob("#{cache_dir}/*/*-dependency-licenses.json").each do |license_manifest_path|
    license_manifest_data = FFI_Yajl::Parser.parse(File.read(license_manifest_path))
    project_name = license_manifest_data["project_name"]
    dependency_license_dir = File.dirname(license_manifest_path)

    license_manifest_data["dependency_managers"].each do |dep_mgr_name, dependencies|
      dep_license_map[dep_mgr_name] ||= {}

      dependencies.each do |dependency|
        # Copy dependency files
        dependency["license_files"].each do |f|
          license_path = File.join(dependency_license_dir, f)
          output_path = File.join(output_dir, f)
          FileUtils.cp(license_path, output_path)
        end

        dep_name = dependency["name"]
        dep_version = dependency["version"]

        # If we already have this dependency we do not need to add it again.
        if dep_license_map[dep_mgr_name][dep_name] && dep_license_map[dep_mgr_name][dep_name][dep_version]
          dep_license_map[dep_mgr_name][dep_name][dep_version]["dependency_of"] << project_name
        else
          dep_license_map[dep_mgr_name][dep_name] ||= {}
          dep_license_map[dep_mgr_name][dep_name][dep_version] = {
            "license" => dependency["license"],
            "license_files" => dependency["license_files"],
            "dependency_of" => [ project_name ],
          }
        end
      end
    end
  end

  FileUtils.rm_rf(cache_dir)
end
project_license_content() click to toggle source

Contents of the project’s license

@return [String]

# File lib/omnibus/licensing.rb, line 222
def project_license_content
  project.license_file.nil? ? "" : IO.read(File.join(Config.project_root, project.license_file))
end
raise_if_warnings_fatal!() click to toggle source
# File lib/omnibus/licensing.rb, line 434
def raise_if_warnings_fatal!
  warnings_to_raise = []
  if Config.fatal_licensing_warnings && !licensing_warnings.empty?
    warnings_to_raise << licensing_warnings
  end

  if Config.fatal_transitive_dependency_licensing_warnings && !transitive_dependency_licensing_warnings.empty?
    warnings_to_raise << transitive_dependency_licensing_warnings
    warnings_to_raise << "If you are encountering missing license or missing license file errors for **transitive** dependencies, you can provide overrides for the missing information at https://github.com/chef/license_scout/blob/1-stable/lib/license_scout/overrides.rb#L93. \n Promote license_scout to Rubygems with `/expeditor promote chef/license_scout:1-stable X.Y.Z` in slack."
  end

  warnings_to_raise.flatten!
  raise LicensingError.new(warnings_to_raise) unless warnings_to_raise.empty?
end
transitive_dependency_licensing_warning(message) click to toggle source

Logs the given message as warning or fails the build depending on the :fatal_transitive_dependency_licensing_warnings configuration setting.

@param [String] message

message to log as warning
# File lib/omnibus/licensing.rb, line 429
def transitive_dependency_licensing_warning(message)
  transitive_dependency_licensing_warnings << message
  log.warn(log_key) { message }
end
validate_license_info() click to toggle source

Inspects the licensing information for the project and the included software components. Logs the found issues to the log as warning.

@return [void]

# File lib/omnibus/licensing.rb, line 159
def validate_license_info
  # First check the project licensing information

  # Check existence of licensing information
  if project.license == "Unspecified"
    licensing_warning("Project '#{project.name}' does not contain licensing information.")
  end

  # Check license file exists
  if project.license != "Unspecified" && project.license_file.nil?
    licensing_warning("Project '#{project.name}' does not point to a license file.")
  end

  # Check used license is a standard license
  if project.license != "Unspecified" && !STANDARD_LICENSES.include?(project.license)
    licensing_info("Project '#{project.name}' is using '#{project.license}' which is not one of the standard licenses identified in https://opensource.org/licenses/alphabetical. Consider using one of the standard licenses.")
  end

  # Now let's check the licensing info for software components
  license_map.each do |software_name, license_info|
    # First check if the software specified a license
    if license_info[:license] == "Unspecified"
      licensing_warning("Software '#{software_name}' does not contain licensing information.")
    end

    # Check if the software specifies any license files
    if license_info[:license] != "Unspecified" && license_info[:license_files].empty?
      licensing_warning("Software '#{software_name}' does not point to any license files.")
    end

    # Check if the software license is one of the standard licenses
    if license_info[:license] != "Unspecified" && !STANDARD_LICENSES.include?(license_info[:license])
      licensing_info("Software '#{software_name}' uses license '#{license_info[:license]}' which is not one of the standard licenses identified in https://opensource.org/licenses/alphabetical. Consider using one of the standard licenses.")
    end
  end
end

Private Instance Methods

check_transitive_dependency_licensing_errors_for(software) click to toggle source

Checks transitive dependency licensing errors for the given software

# File lib/omnibus/licensing.rb, line 562
    def check_transitive_dependency_licensing_errors_for(software)
      reporter = LicenseScout::Reporter.new(license_output_dir(software))
      begin
        reporter.report.each { |i| transitive_dependency_licensing_warning(i) }
      rescue LicenseScout::Exceptions::InvalidOutputReport => e
        transitive_dependency_licensing_warning(<<-EOH)
Licensing output report at '#{license_output_dir(software)}' has errors:
#{e}
EOH
      end
      raise_if_warnings_fatal!
    end
collect_licenses_for(software) click to toggle source

Collect the license files for the software.

# File lib/omnibus/licensing.rb, line 581
def collect_licenses_for(software)
  return nil if software.license == :project_license

  software_name = software.name
  license_data = license_map[software_name]
  license_files = license_data[:license_files]

  license_files.each do |license_file|
    if license_file
      output_file = license_package_location(software_name, license_file)

      if local?(license_file)
        input_file = File.expand_path(license_file, license_data[:project_dir])
        if File.exist?(input_file)
          FileUtils.cp(input_file, output_file)
          File.chmod 0644, output_file unless windows?
        else
          licensing_warning("License file '#{input_file}' does not exist for software '#{software_name}'.")
          # If we got here, we need to fail now so we don't take a git
          # cache snapshot, or else the software build could be restored
          # from cache without fixing the license issue.
          raise_if_warnings_fatal!
        end
      else
        begin
          download_file!(license_file, output_file, enable_progress_bar: false)
          File.chmod 0644, output_file unless windows?
        rescue SocketError,
               Errno::ECONNREFUSED,
               Errno::ECONNRESET,
               Errno::ENETUNREACH,
               Timeout::Error,
               OpenURI::HTTPError,
               OpenSSL::SSL::SSLError
          licensing_warning("Can not download license file '#{license_file}' for software '#{software_name}'.")
          # If we got here, we need to fail now so we don't take a git
          # cache snapshot, or else the software build could be restored
          # from cache without fixing the license issue.
          raise_if_warnings_fatal!
        end
      end
    end
  end
end
collect_transitive_dependency_licenses_for(software) click to toggle source

Uses license_scout to collect the licenses for transitive dependencies into #{output_dir}/license-cache/#{software.name}

# File lib/omnibus/licensing.rb, line 496
    def collect_transitive_dependency_licenses_for(software)
      # We collect the licenses of the transitive dependencies of this software
      # with LicenseScout. We place these files under
      # /opt/project-name/license-cache for them to be cached in git_cache. Once
      # the build completes we will process these license files but we need to
      # perform this step after build, before git_cache to be able to operate
      # correctly with the git_cache.

      collector = LicenseScout::Collector.new(
        software.name,
        software.project_dir,
        license_output_dir(software),
        LicenseScout::Options.new(
          environment: software.with_embedded_path,
          ruby_bin: software.embedded_bin("ruby"),
          manual_licenses: software.dependency_licenses
        )
      )

      begin
        # We do not automatically collect dependency licensing information when
        # skip_transitive_dependency_licensing is set on the software.
        collector.run
      rescue LicenseScout::Exceptions::UnsupportedProjectType => e
        # Looks like this project is not supported by LicenseScout. Either the
        # language and the dependency manager used by the project is not
        # supported, or the software definition does not have any transitive
        # dependencies.  In the latter case software definition should set
        # 'skip_transitive_dependency_licensing' to 'true' to correct this
        # error.
        transitive_dependency_licensing_warning(<<-EOH)
Software '#{software.name}' is not supported project type for transitive \
dependency license collection. See https://github.com/chef/license_scout for \
the list of supported languages and dependency managers. If this project does \
not have any transitive dependencies, consider setting \
'skip_transitive_dependency_licensing' to 'true' in order to correct this error.
EOH
        # If we got here, we need to fail now so we don't take a git
        # cache snapshot, or else the software build could be restored
        # from cache without fixing the license issue.
        raise_if_warnings_fatal!
      rescue LicenseScout::Exceptions::Error => e
        transitive_dependency_licensing_warning(<<-EOH)
Can not automatically detect licensing information for '#{software.name}' using \
license_scout. Error is: '#{e}'
EOH
        # If we got here, we need to fail now so we don't take a git
        # cache snapshot, or else the software build could be restored
        # from cache without fixing the license issue.
        raise_if_warnings_fatal!
      rescue Exception => e
        # This catch all exception handling is here in order not to fail builds
        # until license_scout gets more stable. As we are adding support for more
        # and more dependency managers we discover unhandled edge cases which
        # requires us to have this. Remove this once license_scout is stable.
        transitive_dependency_licensing_warning(<<-EOH)
Unexpected error while running license_scout for '#{software.name}': '#{e}'
EOH
        # If we got here, we need to fail now so we don't take a git
        # cache snapshot, or else the software build could be restored
        # from cache without fixing the license issue.
        raise_if_warnings_fatal!
      end
    end
license_output_dir(software) click to toggle source

The directory to store the licensing information for the given software

# File lib/omnibus/licensing.rb, line 576
def license_output_dir(software)
  File.join(cache_dir, software.name)
end