class Omnibus::Licensing
Constants
- CACHE_DIRECTORY
- OUTPUT_DIRECTORY
- STANDARD_LICENSES
Attributes
Manifest
data of transitive dependency licensing information
@return Hash
The warnings encountered while preparing the licensing information
@return [Array<String>]
The project to create licenses for.
@return [Project]
The warnings encountered while preparing the licensing information for transitive dependencies.
@return [Array<String>]
Public Class Methods
Creates a new instance of Licensing
, executes preparation steps, then yields control to a given block, and then creates a summary of the included licenses.
@example Building a project:
Licensing.create_incrementally(self) do |license_collector| softwares.each do |software| software.build_me([license_collector]) end end
@param [Project] project
The project being built.
@yieldparam [Licensing] license_collector
Yields an instance of Licensing. Call #execute_post_build to copy the license files for a Software definition.
@return [Licensing]
# File lib/omnibus/licensing.rb, line 56 def create_incrementally(project) new(project).tap do |license_collector| license_collector.prepare license_collector.validate_license_info yield license_collector license_collector.process_transitive_dependency_licensing_info license_collector.create_project_license_file license_collector.raise_if_warnings_fatal! end end
@param [Project] project
the project to create licenses for.
# File lib/omnibus/licensing.rb, line 104 def initialize(project) @project = project @licensing_warnings = [] @transitive_dependency_licensing_warnings = [] @dep_license_map = {} end
Public Instance Methods
Cache directory where transitive dependency licenses will be collected in.
@return [String]
# File lib/omnibus/licensing.rb, line 375 def cache_dir File.expand_path(CACHE_DIRECTORY, project.install_dir) end
Path to a .gitkeep file we create in the cache dir so git caching doesn’t delete the directory.
@return [String]
# File lib/omnibus/licensing.rb, line 385 def cache_dir_gitkeep_file File.join(cache_dir, ".gitkeep") end
Summary of the licenses included by the softwares of the project. It is in the form of: … This product bundles python 2.7.9, which is available under a “Python” License. For details, see: /opt/opscode/LICENSES/python-LICENSE …
@return [String]
# File lib/omnibus/licensing.rb, line 238 def components_license_summary out = "\n\n" license_map.keys.sort.each do |name| license = license_map[name][:license] license_files = license_map[name][:license_files] version = license_map[name][:version] out << "This product bundles #{name} #{version},\n" out << "which is available under a \"#{license}\" License.\n" unless license_files.empty? out << "For details, see:\n" license_files.each do |license_file| out << "#{license_package_location(name, license_file)}\n" end end out << "\n" end out end
Creates the top level license file for the project. Top level file is created at #{project.license_file_path} and contains the name of the project, version of the project, text of the license of the project and a summary of the licenses of the included software components.
@return [void]
# File lib/omnibus/licensing.rb, line 205 def create_project_license_file File.open(project.license_file_path, "w") do |f| f.puts "#{project.name} #{project.build_version} license: \"#{project.license}\"" f.puts "" f.puts project_license_content f.puts "" f.puts components_license_summary f.puts "" f.puts dependencies_license_summary end end
Summary of the licenses of the transitive dependencies of the project. It is in the form of: … This product includes inifile 3.0.0 which is a ‘ruby_bundler’ dependency of ‘chef’, and which is available under a ‘MIT’ License. For details, see: /opt/opscode/LICENSES/ruby_bundler-inifile-3.0.0-README.md …
@return [String]
# File lib/omnibus/licensing.rb, line 273 def dependencies_license_summary out = "\n\n" dep_license_map.each do |dep_mgr_name, data| data.each do |dep_name, data| data.each do |dep_version, dep_data| projects = dep_data["dependency_of"].sort.map { |p| "'#{p}'" }.join(", ") files = dep_data["license_files"].map { |f| File.join(output_dir, f) } out << "This product includes #{dep_name} #{dep_version}\n" out << "which is a '#{dep_mgr_name}' dependency of #{projects},\n" out << "and which is available under a '#{dep_data["license"]}' License.\n" out << "For details, see:\n" out << files.join("\n") out << "\n\n" end end end out end
Callback that gets called by Software#build_me
after the build is done. Invokes license copying for the given software. This ensures that licenses are copied before a git cache snapshot is taken, so that the license files are correctly restored when a build is skipped due to a cache hit.
@param [Software] software
@return [void]
# File lib/omnibus/licensing.rb, line 145 def execute_post_build(software) collect_licenses_for(software) unless software.skip_transitive_dependency_licensing collect_transitive_dependency_licenses_for(software) check_transitive_dependency_licensing_errors_for(software) end end
Required callback to use instances of this class as a build wrapper for Software#build_me
. Licensing
doesn’t need to do anything pre-build, so this does nothing.
@param [Software] software
@return [void]
# File lib/omnibus/licensing.rb, line 133 def execute_pre_build(software); end
Map that collects information about the licenses of the softwares included in the project.
@example {
... "python" => { "license" => "Python", "license_files" => "LICENSE", "version" => "2.7.9", "project_dir" => "/var/cache/omnibus/src/python/Python-2.7.9/" }, ...
}
@return [Hash]
# File lib/omnibus/licensing.rb, line 313 def license_map @license_map ||= begin map = {} project.library.each do |component| # Some of the components do not bundle any software but contain # some logic that we use during the build. These components are # covered under the project's license and they do not need specific # license files. next if component.license == :project_license map[component.name] = { license: component.license, license_files: component.license_files, version: component.version, project_dir: component.project_dir, } end map end end
Returns the location where the license file should reside in the package. License file is named as <project_name>-<license_file_name> and created under the output licenses directory.
@return [String]
# File lib/omnibus/licensing.rb, line 343 def license_package_location(component_name, where) if local?(where) File.join(output_dir, "#{component_name}-#{File.split(where).last}") else u = URI(where) File.join(output_dir, "#{component_name}-#{File.basename(u.path)}") end end
Logs the given message as info.
This method should only be used for detecting in a license is known or not. In the future, we will introduce a configurable way to whitelist or blacklist the allowed licenses. Once we implement that we need to stop using this method.
@param [String] message
message to log as warning
# File lib/omnibus/licensing.rb, line 408 def licensing_info(message) log.info(log_key) { message } end
Logs the given message as warning or fails the build depending on the :fatal_licensing_warnings configuration setting.
@param [String] message
message to log as warning
# File lib/omnibus/licensing.rb, line 418 def licensing_warning(message) licensing_warnings << message log.warn(log_key) { message } end
Returns if the given path to a license is local or a remote url.
@return [Boolean]
# File lib/omnibus/licensing.rb, line 394 def local?(license) u = URI(license) u.scheme.nil? end
Output directory to create the licenses in.
@return [String]
# File lib/omnibus/licensing.rb, line 357 def output_dir File.expand_path(OUTPUT_DIRECTORY, project.install_dir) end
Path to a .gitkeep file we create in the output dir so git caching doesn’t delete the directory.
@return [String]
# File lib/omnibus/licensing.rb, line 367 def output_dir_gitkeep_file File.join(output_dir, ".gitkeep") end
Creates the required directories for licenses.
@return [void]
# File lib/omnibus/licensing.rb, line 116 def prepare FileUtils.rm_rf(output_dir) FileUtils.mkdir_p(output_dir) FileUtils.touch(output_dir_gitkeep_file) FileUtils.rm_rf(cache_dir) FileUtils.mkdir_p(cache_dir) FileUtils.touch(cache_dir_gitkeep_file) end
-
Translate all transitive dependency licensing issues into omnibus warnings
-
Parse all the licensing information for all software from ‘cache_dir’
-
Merge and drop the duplicates
-
Add these licenses to the main manifest, to be merged with the main
licensing information from software definitions.
# File lib/omnibus/licensing.rb, line 454 def process_transitive_dependency_licensing_info Dir.glob("#{cache_dir}/*/*-dependency-licenses.json").each do |license_manifest_path| license_manifest_data = FFI_Yajl::Parser.parse(File.read(license_manifest_path)) project_name = license_manifest_data["project_name"] dependency_license_dir = File.dirname(license_manifest_path) license_manifest_data["dependency_managers"].each do |dep_mgr_name, dependencies| dep_license_map[dep_mgr_name] ||= {} dependencies.each do |dependency| # Copy dependency files dependency["license_files"].each do |f| license_path = File.join(dependency_license_dir, f) output_path = File.join(output_dir, f) FileUtils.cp(license_path, output_path) end dep_name = dependency["name"] dep_version = dependency["version"] # If we already have this dependency we do not need to add it again. if dep_license_map[dep_mgr_name][dep_name] && dep_license_map[dep_mgr_name][dep_name][dep_version] dep_license_map[dep_mgr_name][dep_name][dep_version]["dependency_of"] << project_name else dep_license_map[dep_mgr_name][dep_name] ||= {} dep_license_map[dep_mgr_name][dep_name][dep_version] = { "license" => dependency["license"], "license_files" => dependency["license_files"], "dependency_of" => [ project_name ], } end end end end FileUtils.rm_rf(cache_dir) end
Contents of the project’s license
@return [String]
# File lib/omnibus/licensing.rb, line 222 def project_license_content project.license_file.nil? ? "" : IO.read(File.join(Config.project_root, project.license_file)) end
# File lib/omnibus/licensing.rb, line 434 def raise_if_warnings_fatal! warnings_to_raise = [] if Config.fatal_licensing_warnings && !licensing_warnings.empty? warnings_to_raise << licensing_warnings end if Config.fatal_transitive_dependency_licensing_warnings && !transitive_dependency_licensing_warnings.empty? warnings_to_raise << transitive_dependency_licensing_warnings warnings_to_raise << "If you are encountering missing license or missing license file errors for **transitive** dependencies, you can provide overrides for the missing information at https://github.com/chef/license_scout/blob/1-stable/lib/license_scout/overrides.rb#L93. \n Promote license_scout to Rubygems with `/expeditor promote chef/license_scout:1-stable X.Y.Z` in slack." end warnings_to_raise.flatten! raise LicensingError.new(warnings_to_raise) unless warnings_to_raise.empty? end
Logs the given message as warning or fails the build depending on the :fatal_transitive_dependency_licensing_warnings configuration setting.
@param [String] message
message to log as warning
# File lib/omnibus/licensing.rb, line 429 def transitive_dependency_licensing_warning(message) transitive_dependency_licensing_warnings << message log.warn(log_key) { message } end
Inspects the licensing information for the project and the included software components. Logs the found issues to the log as warning.
@return [void]
# File lib/omnibus/licensing.rb, line 159 def validate_license_info # First check the project licensing information # Check existence of licensing information if project.license == "Unspecified" licensing_warning("Project '#{project.name}' does not contain licensing information.") end # Check license file exists if project.license != "Unspecified" && project.license_file.nil? licensing_warning("Project '#{project.name}' does not point to a license file.") end # Check used license is a standard license if project.license != "Unspecified" && !STANDARD_LICENSES.include?(project.license) licensing_info("Project '#{project.name}' is using '#{project.license}' which is not one of the standard licenses identified in https://opensource.org/licenses/alphabetical. Consider using one of the standard licenses.") end # Now let's check the licensing info for software components license_map.each do |software_name, license_info| # First check if the software specified a license if license_info[:license] == "Unspecified" licensing_warning("Software '#{software_name}' does not contain licensing information.") end # Check if the software specifies any license files if license_info[:license] != "Unspecified" && license_info[:license_files].empty? licensing_warning("Software '#{software_name}' does not point to any license files.") end # Check if the software license is one of the standard licenses if license_info[:license] != "Unspecified" && !STANDARD_LICENSES.include?(license_info[:license]) licensing_info("Software '#{software_name}' uses license '#{license_info[:license]}' which is not one of the standard licenses identified in https://opensource.org/licenses/alphabetical. Consider using one of the standard licenses.") end end end
Private Instance Methods
Checks transitive dependency licensing errors for the given software
# File lib/omnibus/licensing.rb, line 562 def check_transitive_dependency_licensing_errors_for(software) reporter = LicenseScout::Reporter.new(license_output_dir(software)) begin reporter.report.each { |i| transitive_dependency_licensing_warning(i) } rescue LicenseScout::Exceptions::InvalidOutputReport => e transitive_dependency_licensing_warning(<<-EOH) Licensing output report at '#{license_output_dir(software)}' has errors: #{e} EOH end raise_if_warnings_fatal! end
Collect the license files for the software.
# File lib/omnibus/licensing.rb, line 581 def collect_licenses_for(software) return nil if software.license == :project_license software_name = software.name license_data = license_map[software_name] license_files = license_data[:license_files] license_files.each do |license_file| if license_file output_file = license_package_location(software_name, license_file) if local?(license_file) input_file = File.expand_path(license_file, license_data[:project_dir]) if File.exist?(input_file) FileUtils.cp(input_file, output_file) File.chmod 0644, output_file unless windows? else licensing_warning("License file '#{input_file}' does not exist for software '#{software_name}'.") # If we got here, we need to fail now so we don't take a git # cache snapshot, or else the software build could be restored # from cache without fixing the license issue. raise_if_warnings_fatal! end else begin download_file!(license_file, output_file, enable_progress_bar: false) File.chmod 0644, output_file unless windows? rescue SocketError, Errno::ECONNREFUSED, Errno::ECONNRESET, Errno::ENETUNREACH, Timeout::Error, OpenURI::HTTPError, OpenSSL::SSL::SSLError licensing_warning("Can not download license file '#{license_file}' for software '#{software_name}'.") # If we got here, we need to fail now so we don't take a git # cache snapshot, or else the software build could be restored # from cache without fixing the license issue. raise_if_warnings_fatal! end end end end end
Uses license_scout to collect the licenses for transitive dependencies into #{output_dir}/license-cache/#{software.name}
# File lib/omnibus/licensing.rb, line 496 def collect_transitive_dependency_licenses_for(software) # We collect the licenses of the transitive dependencies of this software # with LicenseScout. We place these files under # /opt/project-name/license-cache for them to be cached in git_cache. Once # the build completes we will process these license files but we need to # perform this step after build, before git_cache to be able to operate # correctly with the git_cache. collector = LicenseScout::Collector.new( software.name, software.project_dir, license_output_dir(software), LicenseScout::Options.new( environment: software.with_embedded_path, ruby_bin: software.embedded_bin("ruby"), manual_licenses: software.dependency_licenses ) ) begin # We do not automatically collect dependency licensing information when # skip_transitive_dependency_licensing is set on the software. collector.run rescue LicenseScout::Exceptions::UnsupportedProjectType => e # Looks like this project is not supported by LicenseScout. Either the # language and the dependency manager used by the project is not # supported, or the software definition does not have any transitive # dependencies. In the latter case software definition should set # 'skip_transitive_dependency_licensing' to 'true' to correct this # error. transitive_dependency_licensing_warning(<<-EOH) Software '#{software.name}' is not supported project type for transitive \ dependency license collection. See https://github.com/chef/license_scout for \ the list of supported languages and dependency managers. If this project does \ not have any transitive dependencies, consider setting \ 'skip_transitive_dependency_licensing' to 'true' in order to correct this error. EOH # If we got here, we need to fail now so we don't take a git # cache snapshot, or else the software build could be restored # from cache without fixing the license issue. raise_if_warnings_fatal! rescue LicenseScout::Exceptions::Error => e transitive_dependency_licensing_warning(<<-EOH) Can not automatically detect licensing information for '#{software.name}' using \ license_scout. Error is: '#{e}' EOH # If we got here, we need to fail now so we don't take a git # cache snapshot, or else the software build could be restored # from cache without fixing the license issue. raise_if_warnings_fatal! rescue Exception => e # This catch all exception handling is here in order not to fail builds # until license_scout gets more stable. As we are adding support for more # and more dependency managers we discover unhandled edge cases which # requires us to have this. Remove this once license_scout is stable. transitive_dependency_licensing_warning(<<-EOH) Unexpected error while running license_scout for '#{software.name}': '#{e}' EOH # If we got here, we need to fail now so we don't take a git # cache snapshot, or else the software build could be restored # from cache without fixing the license issue. raise_if_warnings_fatal! end end
The directory to store the licensing information for the given software
# File lib/omnibus/licensing.rb, line 576 def license_output_dir(software) File.join(cache_dir, software.name) end