modules.helpers package

Submodules

modules.helpers.commandline module

checkCmdLineArgs(cmd_line_parser: argparse.ArgumentParser, cmdline_args: object)buildnis.modules.helpers.commandline_arguments.CommandlineArguments

Sets all needed attributes to a default value if they aren’t present.

Parameters
  • cmd_line_parser (argparse.ArgumentParser) – the argparse.ArgumentParser instance to use

  • cmdline_args (object) – the object returned by cmd_line_parser.parse_args

Returns

the checked and filled CommandlineArguments instance

Return type

CommandlineArguments

parseCommandLine()buildnis.modules.helpers.commandline_arguments.CommandlineArguments

Parses the command line arguments.

Parses the command line arguments, exits the program if an illegal argument has been given.

Parameters

arguments – the command line arguments passed to the program

Returns

An CommandlineArguments instance containing the command line arguments as attributes.

modules.helpers.commandline_arguments module

class CommandlineArguments(src: object)

Bases: object

Holds information about the command line arguments passed to the program.

Its attributes are the possible command line arguments of the program.

project_config_file

the path to the project config file to use

Type

FilePath

conf_dir

the path to the directory to write generated configurations to

Type

FilePath

conf_scripts_dir

the path to the directory to search for additional build tool configure scripts.

Type

FilePath

log_file

the path to the log file to write.

Type

FilePath

log_level

the minimum log level

Type

int

do_configure

run only the configure phase of the build

Type

bool

do_build

run only the build phase of the build

Type

bool

build_targets

list of build targets that should be build

Type

List[str]

do_install

only run the install phase of the build

Type

bool

install_targets

the list of targets to install

Type

List[str]

do_clean

delete all files generated by the build phase

Type

bool

do_distclean

delete all generated files (build and configuration)

Type

bool

do_check_what_to_do

do everything that has not been done yet.

Type

bool

checkTargetArgs(name: str)None

Checks the list stored in the attribute with the given name and flattens it to a single list if it contains another list.

Parameters

name (str) – the name of the attribute to check the stored list of

static doAppendTarget(tmp_targets: List, target: object)None

Appends a target to the list of targets.

Parameters
  • tmp_targets (List) – the list of targets to append

  • target (object) – the target to check if it is a list or a single element.

flattenList(name: str, tmp_targets: List)

Flattens the given list to a single list.

Parameters
  • name (str) – The attribute’s name, the list to flatten.

  • tmp_targets (List) – Where to sore the flattened list.

initAttribs(src: object)None

Initializes all attributes to default values if not set from the command-line, that means, that attribute isn’t an attribute of src.

Parameters

src (object) – The object holding the parsed command-line arguments.

setCleanStages(src: object)None

Set arguments for the clean stages of the build.

Parameters

src (object) – The original object holding the command line arguments.

setConfigs(src: object)None

Set configuration related arguments.

Parameters

src (object) – The original object holding the command line arguments.

setStages(src: object)None

Set arguments for the stages of the build.

Parameters

src (object) – The original object holding the command line arguments.

deleteConfigs(commandline_args: buildnis.modules.helpers.commandline_arguments.CommandlineArguments, logger: logging.Logger, list_of_generated_files: List[str], list_of_generated_dirs: List[str])

Deletes all configuration files and directories.

Parameters
  • commandline_args (object) – Command line argument object instance

  • logger (logging.Logger) – The logger to use and stop

  • list_of_generated_files (List[FilePath]) – The list of files to delete

  • list_of_generated_dirs (List[FilePath]) – The list of directories to delete. Attention: each directory must be empty!

deleteLogfiles(commandline_args: buildnis.modules.helpers.commandline_arguments.CommandlineArguments)None

Deletes the log file, if one has been configured from the command-line.

Parameters

commandline_args (CommandlineArguments) – Instance holding the command-line arguments, to check, if a log file has been used.

doDistClean(commandline_args: buildnis.modules.helpers.commandline_arguments.CommandlineArguments, logger: logging.Logger, list_of_generated_files: List[str], list_of_generated_dirs: List[str])None

Helper: if argument distclean is set, delete all generated files.

WARNING: Shuts down the logging mechanism, no more logging after this function!

Parameters
  • commandline_args (object) – Command line argument object instance

  • logger (logging.Logger) – The logger to use and stop

  • list_of_generated_files (List[FilePath]) – The list of files to delete

  • list_of_generated_dirs (List[FilePath]) – The list of directories to delete. Attention: each directory must be empty!

setupLogger(commandline_args: buildnis.modules.helpers.commandline_arguments.CommandlineArguments)logging.Logger

Sets up the logger.

Parameters

commandline_args (CommandlineArguments) – The object holding the command line arguments.

Returns

The commandline object instance to use.

Return type

logging.Logger

modules.helpers.config_parser module

expandItem(item: str, parents: List[object])object

Parses the given item, if it contains a placeholder, that placeholder is expanded. If the item doesn’t contain a placeholder, the item’s unaltered string is returned.

Parameters
  • item (str) – The item to parse and expand its placeholder

  • parents (List[object]) – The parents of the item to search for the placeholder’s content.

Returns

The expanded string if the item contained a placeholder, the

original string else. If the placeholder points to another object that is not a string, this object is returned.

Return type

object

getPlaceholder(parents: List[object], parent_to_use_id: int, placeholder: str)object
Returns the expanded placeholder. Searches for the attribute with name

placeholder or the value of the key placeholder in the parent.

Parameters
  • parents (List[object]) – The list of parents to search the expansion in.

  • parent_to_use_id (int) – The id of the parent in the list, to use for the replacement.

  • placeholder (str) – The string to replace with an element of the same name

Raises

Exception – if the replacement of the placeholder from the parent element throws an exception.

Returns

The replacement for the placeholder.

Return type

object

parseConfigElement(element: object, parents: Optional[List[object]] = None)object

Parses the given config element and replaces placeholders. Placeholders are strings of the form ${PLACEHOLDER}, with start with a dollar sign followed by an opening curly brace and end with a curly brace. The string between the two curly braces is changed against it’s value.

Parameters
  • element (object) – The configuration element to parse and expand.

  • parent (List[object], optional) – The parent and the parent’s parent and it’s

  • as a list (parent) –

  • with the parent as first element. Defaults to None. (starting) –

Returns

The parsed and expanded object.

Return type

object

parseList(element: List[object], local_parents: List[object])List[object]

Parses the items of a list.

Parameters
  • element (List[object]) – The list to parse

  • local_parents (List[object]) – The list of parents

Returns

The parsed and, if applicable, expanded, list of items.

Return type

List[object]

modules.helpers.execute module

class EnvArgs(script: FilePath = '', args: List[str] = None, do_source: bool = False)

Bases: NamedTuple

Class to hold the arguments needed for the environment script of a command to run.

script

The path to the environment script to run or source.

Type

FilePath

args

The arguments to pass to the environment script.

Type

List[str]

do_source

If this is true, the environment script is sourced in the current command interpreter and not executed.

Type

bool

args: List[str]

Alias for field number 1

do_source: bool

Alias for field number 2

script: str

Alias for field number 0

class ExeArgs(exe: FilePath = '', args: List[str] = None)

Bases: NamedTuple

Class to hold the arguments needed to run a command.

exe

The path to the executable to call.

Type

FilePath

args

The list of arguments to pass to the executable.

Type

List[str]

args: List[str]

Alias for field number 1

exe: str

Alias for field number 0

exception ExecuteException

Bases: Exception

The Exception is thrown if the execution of the given commandline fails.

class RunRegex(regex: str = '', group: int = 0)

Bases: NamedTuple

Class to hold a regex tuple to parse a command output with.

regex

The actual regex to use for parsing the output of the command.

Type

str

group

The match group of the regex to use for the result.

Type

int

group: int

Alias for field number 1

regex: str

Alias for field number 0

doesExecutableWork(exe_args: buildnis.modules.helpers.execute.ExeArgs, check_regex: buildnis.modules.helpers.execute.RunRegex, env_args: buildnis.modules.helpers.execute.EnvArgs = EnvArgs(script='', args=None, do_source=False))str

Checks if the given command line works.

Tries to run the command with the given arguments (see runCommand) and parses the output of the program, tries to match the given regex check_regex in the output (stdout and stderr) of the command. If the match group regex_group (defaults to 0, the whole regex) is found in the output, the function returns the matched string.

Parameters
  • exe_args (ExeArgs) – The name of the executable or path to the executable to call and the arguments to pass to the executable.

  • env_args (EnvArgs) – The arguments needed for the environment script, if applicable. Holds the command to call the environment script in env_args.script, the arguments to call the environment script with in env_args.args and if the environment script has to be sourced instead of running it, env_args.do_source ir True.

  • env_args – (EnvArgs): The arguments needed to setup the environment for the executable, if applicable. Defaults to (“”, None, False).

Raises

ExecuteException – if something goes wrong

Returns

the matched string if the regex matches the output, the empty string

’’ otherwise.

Return type

str

runCommand(exe_args: buildnis.modules.helpers.execute.ExeArgs, env_args: buildnis.modules.helpers.execute.EnvArgs = EnvArgs(script='', args=None, do_source=False))buildnis.modules.config.CmdOutput

Executes the given command with the given arguments.

The argument exe_args.exe is the executable’s name or path, needed arguments can be passed in the list exe_args.args. In env_args the command to set up an environment can be given, with needed arguments to this environment script in the list env_args.args.

Parameters
  • exe_args (ExeArgs) – The name of the executable or path to the executable to call and the arguments to pass to the executable.

  • env_args (EnvArgs) – The arguments needed for the environment script, if applicable. Holds the command to call the environment script in env_args.script, the arguments to call the environment script with in env_args.args and if the environment script has to be sourced instead of running it, env_args.do_source ir True.

Raises

ExecuteException – if something goes wrong

Returns

The output of the executed command as tuple (stdout, stderr)

Return type

CmdOutput

setEnv(exe_args: buildnis.modules.helpers.execute.ExeArgs, env_args: buildnis.modules.helpers.execute.EnvArgs, exe_args_real: List[str], env_args_real: List[str], cmd_line_args: List[str])None

Set up the command line arguments for the environment script.

Parameters
  • exe_args (ExeArgs) – The object holding the needed arguments for the executable itself.

  • env_args (EnvArgs) – The object holding the needed arguments to set the environment for the executable.

  • exe_args_real (List[str]) – The real list of executable arguments to use.

  • env_args_real (List[str]) – The real list of environment arguments to use.

  • cmd_line_args (List[str]) – The list of arguments to pass to the command interpreter.

modules.helpers.file_compare module

class FileCompare(file: str)

Bases: object

Holds information about a file to compare it to another version of itself to see if something has changed.

Does the following to steps:
  • compares the file sizes, if they are the same

  • compares the Blake2 hashes of both files

path

path to the file as string

Type

FilePath

path_obj

the Path object of the file

Type

pathlib.Path

size

the size of the file in bytes (symlinks don’t have a meaningful size)

Type

int

hash

the BLAKE2 hash of the file’s contend as a hex string.

Type

str

isSame(bool)

returns true if self ‘is’ the same as the given file.

generateHash(str)

generates the hash of the file, saves it in hash and returns it

generateHash()str

Rehashes the file, saves and returns the hash as hex string.

The hash replaces the old one in the attribute hash.

Raises

FileCompareException – if something goes wrong

Returns

The hash of the file with path path as hex string.

Return type

str

hasChanged(not_exist_is_excp: bool = False)bool

Checks if the stored file has changed on disk since taking the last checksum.

If the file has changed (another file size or checksum) or it doesn’t exist any more, True is returned. If the file still has the same checksum as the stored one, False is returned.

Parameters

not_exist_is_excp (bool, optional) – Should an exception be raised if the file doesn’t exist anymore? Defaults to False.

Raises

FileCompareException – if something went wrong

Returns

True, if the file has changed since calculating the checksum,

False else.

Return type

bool

isSame(file: str, not_exist_is_excp: bool = False)bool

Checks whether self and the file with path file are the same.

Problem: a symlink to the file and the file itself are NOT the same using this.

Parameters
  • file (FilePath) – path to the file to check

  • not_exist_is_excp (bool) – if this is True an exception is raised if file does not exist. If this is False, False is returned - the files data is not the same. Default: False

Raises

FileCompareException – if something goes wrong

Returns

True, if the file is the same

False else

Return type

bool

isSameFile(other: buildnis.modules.helpers.file_compare.FileCompare)bool

Checks whether two FileComare instances hold the same file content.

Compares self against another FileCompare instance, comparing file size and hash.

Returns True if both files have the same content.

Attention: does NOT compare the filenames, only file size and hash of the files are compared.

Parameters

other (FileCompare) – the instance to compare self to

Raises

FileCompareException – if something doesn’t work out well ….

Returns

True, if both instances have the same filesize and hash.

False else

Return type

bool

areHashesSame(file1: str, file2: str, not_exist_is_excp: bool = False)bool

Compares the BLAKE2 hashes of the given files.

Returns True if the contents of both files are the same (have the same hash).

Parameters
  • file1 (FilePath) – first file to compare

  • file2 (FilePath) – second file to compare

  • not_exist_is_excp (bool) – if this is True an exception is raised if a file does not exist. If this is False, False is returned - the files hashes are not the same. Default: False

Raises

FileCompareException – if something goes wrong

Returns

True, if both files’ content have the same BLAKE2 hash value.

False else

Return type

bool

modules.helpers.files module

exception FileCompareException

Bases: buildnis.modules.BuildnisException

Exception raised if a given file path can’t be accessed or read to generate the hash.

checkIfExists(file: str)bool

Returns True if the given file exists.

Parameters

file (FilePath) – Path to the file to test

Raises

FileCompareException – if something went wrong

Returns

True`, if the file exists

False else

Return type

bool

checkIfIsDir(directory: str)bool

Returns True if the given file exists and is a directory.

Parameters

directory (FilePath) – Path to the directory to test

Raises

FileCompareException – if something went wrong

Returns

True`, if the file exists and is a directory

False else

Return type

bool

checkIfIsFile(file: str)bool

Returns True if the given file exists and is a file.

Parameters

file (FilePath) – Path to the file to test

Raises

FileCompareException – if something went wrong

Returns

True`, if the file exists and is a file

False else

Return type

bool

Returns True if the given file exists and is a symlink.

Parameters

link (FilePath) – Path to the file to test

Raises

FileCompareException – if something went wrong

Returns

True`, if the file exists and is a symlink

False else

Return type

bool

deleteDirs(logger: logging.Logger, list_of_dirs: List[str])None

Deletes all directories in the given list of directories.

Attention: directory has to be empty!

Raises

FileCompareException – if something goes wrong

Parameters
  • logger (logging.Logger) – The logger instance to use for logging.

  • list_of_dirs (List[FilePath]) – The list of directories to delete. As a list of paths to the directories to delete.

deleteFiles(logger, list_of_files: List[str])None

Deletes all files in the given list of files to delete.

Raises

FileCompareException – if something goes wrong

Parameters
  • logger ([type]) – The logger instance to use for logging.

  • list_of_files (List[FilePath]) – The list of files to delete. As a list of file paths to the files to delete.

hashFile(file: str)str

Generates a BLAKE2 hash of the file with the given path.

Returns the hash as a hex string. If something goes wrong, it returns an FileCompareException instance.

Raises

FileCompareException – if something goes wrong

Parameters

file (FilePath) – the file to return the BLAKE2 hash of

Returns

the hex hash of the file’s contend

Return type

str

makeDirIfNotExists(directory: str)None

Creates the directory directory if it doesn’t exist yet.

Parameters

directory (FilePath) – the directory to create

Raises

FileCompareException – if something goes wrong

returnExistingFile(file_list: List[str])str

Returns the first existing path in the list of given paths, and “” the empty string, if none of the paths points to an existing file.

Raises

FileCompareException – if something goes wrong

Parameters

file_list (List[FilePath]) – The list of file paths to check for existence.

Returns

The first of the given file paths that exists as a file, the

empty string (“”) if none exists.

Return type

FilePath

modules.helpers.json module

checkConfigName(json_path: str, conf_file_name: str, ret_val: object)None

Check the name of the JSON configuration file.

Parameters
  • json_path (FilePath) – The path of the JSOn file to check.

  • conf_file_name (str) – The configuration name of the JSON file.

  • ret_val (object) – The deserialized JSON file.

checkConfigVersion(json_path: str, ret_val: object)None

Check the version of the JSON configuration file.

Parameters
  • json_path (FilePath) – The path to the JSON file.

  • ret_val (object) – The deserialized JSON configuration.

getJSONDict(src: object, to_ignore: Optional[List[str]] = None)Dict

Returns a dictionary suitable to pass to json.dump(s).

Attention: only works with simple classes obtained from json.load(s).

Parameters
  • src (object) – The class to serialize.

  • to_ignore (List[str]) – The list of attribute names to ignore.

Returns

The dictionary suitable to pass to json.dump(s).

Return type

Dict

parseItem(src: object, to_ignore: List[str], ret_val: Dict[str, object], item: object)None

Parses an item of src’s dictionary of attributes.

Parameters
  • src (object) – The object to serialize.

  • to_ignore (List[str]) – The list of attributes to ignore and not serialize.

  • ret_val (Dict[str, object]) – The dictionary to return, the serialized object src.

  • item (object) – The current item to serialize.

parseList(src, ret_val, item)None

Parse the elements of a list.

Parameters
  • src (object) – The object to serialize.

  • ret_val (Dict[str, object]) – The dictionary to return, the serialized object src.

  • item (object) – The current item to serialize.

readJSON(json_path: str, file_text: str = '', conf_file_name: str = '')object

Reads the JSON from the given file and saves it to a class object with the JSON elements as attributes.

If an error occurs, the program is exited with an error message! The JSON must have an element file_version that has a value of at least CFG_VERSION, if not, the program is exited with an error message.

Parameters
  • json_path (FilePath) – The path to the JSON file to read.

  • file_text (str, optional) – The name of the JSON configuration file for logging proposes. Defaults to “”, which will be logged as ‘a’, like in “Writing _a_ JSON configuration file …”.

  • conf_file_name (str, optional) – The string that has to be the value of file_name in the JSON file, if not, the program exits. Defaults to “”.

Returns

A class instance with the JSON elements as attributes.

Return type

object

setFileCompare(src: object, ret_val: Dict[str, object], item: object)None

Set the attributes of the FileCompare instance in the JSON dictionary.

Parameters
  • src (object) – The object to serialize.

  • ret_val (Dict[str, object]) – The dictionary to return, the serialized object src.

  • item (object) – The current item to serialize.

setOrigFile(json_path: str, ret_val: object)None

Set the FileCompare instance orig_file to the original JSON configuration.

Parameters
  • json_path (FilePath) – The path to the JSON configuration file.

  • ret_val (object) – The deserialized JSON file.

writeJSON(json_dict: Dict, json_path: str, file_text: str = '', conf_file_name: str = '')None

Writes the information contained in the dictionary json_dict as JSON.

If an error occurs, the program is exited with an error message!

Parameters
  • json_dict (Dict) – The JSON serializeable dict to generate the JSON of

  • json_path (FilePath) – Path to the JSON file to write

  • file_text (str, optional) – The name of the JSON configuration file for logging proposes. Defaults to “”, which will be logged as ‘a’, like in “Writing _a_ JSON configuration file …”.

  • conf_file_name (str, optional) – The string that has to be the value of file_name in the JSON file, if not, the program exits. Defaults to “”.

modules.helpers.logging module

getProgramLogger(level: int, logfile: str)logging.Logger

Returns the logger to use for the program.

Always logs DEBUG, INFO and WARNING to stdout, ERROR and CRITICAL go to stderr. If a logfile is given, everything is logged to this file too. level is the minimum log level to actually output messages.

When you want to use this logger, simply call

my_logger = modules.helpers.logging.getProgramLogger()

my_logger.info (“Info level log”) my_logger.error (“Error level log”)

Parameters
  • level (int) – minimumm log level to output

  • logfile (FilePath) – if this is not None or the empty string “”, log to this file too.

Returns

the logging.Logger instance to use to log

Return type

logging.Logger

modules.helpers.placeholder_regex module

current_date_regex = re.compile('\\$\\{(DATE)\\}')

Regex to find the placeholder ${DATE}.

current_day_regex = re.compile('\\$\\{(DAY)}')

Regex to find the placeholder ${DAY}.

current_month_regex = re.compile('\\$\\{(MONTH)}')

Regex to find the placeholder ${MONTH}.

current_time_regex = re.compile('\\$\\{(TIME)\\}')

Regex to find the placeholder ${TIME}.

current_year_regex = re.compile('\\$\\{(YEAR)}')

Regex to find the placeholder ${YEAR}.

host_cpu_arch_regex = re.compile('\\$\\{(HOST_CPU_ARCH)\\}')

Regex to find the placeholder ${HOST_CPU_ARCH}.

host_name_regex = re.compile('\\$\\{(HOST_NAME)\\}')

Regex to find the placeholder ${HOST_NAME}.

host_num_cores_regex = re.compile('\\$\\{(HOST_NUM_CORES)\\}')

Regex to find the placeholder ${HOST_NUM_CORES}.

host_num_log_cores_regex = re.compile('\\$\\{(HOST_NUM_LOG_CORES)\\}')

Regex to find the placeholder ${HOST_NUM_LOG_CORES}.

host_os_regex = re.compile('\\$\\{(HOST_OS)\\}')

Regex to find the placeholder ${HOST_OS}.

os_name_linux_regex = re.compile('\\$\\{(OS_NAME_LINUX)\\}')

Regex to find the placeholder ${OS_NAME_LINUX}.

os_name_osx_regex = re.compile('\\$\\{(OS_NAME_OSX)\\}')

Regex to find the placeholder ${OS_NAME_OSX}.

os_name_windows_regex = re.compile('\\$\\{(OS_NAME_WINDOWS)\\}')

Regex to find the placeholder ${OS_NAME_WINDOWS}.

placeholder_regex = re.compile('\\$\\{(.*)\\}')

Regex to find general placefolders, of the form ${STRING}, where string is to be substituted for a value of another configuration item.

project_author_regex = re.compile('\\$\\{(PROJECT_AUTHOR)\\}')

Regex to find the placeholder ${PROJECT_AUTHOR}.

project_cfg_dir_regex = re.compile('\\$\\{(PROJECT_CONFIG_DIR_PATH)\\}')

Regex to find the placeholder ${PROJECT_CONFIG_DIR_PATH}.

project_company_regex = re.compile('\\$\\{(PROJECT_COMPANY)\\}')

Regex to find the placeholder ${PROJECT_COMPANY}.

Regex to find the placeholder ${PROJECT_COPYRIGHT_INFO}.

project_email_regex = re.compile('\\$\\{(PROJECT_EMAIL)\\}')

Regex to find the placeholder ${PROJECT_EMAIL}.

project_name_regex = re.compile('\\$\\{(PROJECT_NAME)\\}')

Regex to find the placeholder ${PROJECT_NAME}.

project_root_regex = re.compile('\\$\\{(PROJECT_ROOT)\\}')

Regex to find the placeholder ${PROJECT_ROOT}.

project_version_regex = re.compile('\\$\\{(PROJECT_VERSION)\\}')

Regex to find the placeholder ${PROJECT_VERSION}.

project_web_url_regex = re.compile('\\$\\{(PROJECT_WEB_URL)\\}')

Regex to find the placeholder ${PROJECT_WEB_URL}.

replaceConstants(item: str)str

Replaces all known constants defined in config_values.py in the given string.

These are placeholders like ${PROJECT_ROOT}, ${PROJECT_NAME}, …

Parameters

item (str) – The string to parse for known constants.

Returns

The substitution if a placeholder has been found, the unaltered

string else.

Return type

str

replaceDateTimeConstants(ret_val: str)str

Replaces date and time placeholders with the current date and/or time.

Parameters

ret_val (str) – The string to parse.

Returns

The replaced date and or time if a match has been found, the original

string ret_val else.

Return type

str

replaceHostConstants(ret_val: str)str

Replaces host placeholders with the corresponding values.

Parameters

ret_val (str) – The string to parse.

Returns

The replaced values if a match has been found, the original

string ret_val else.

Return type

str

replaceProjectConstants(ret_val: str)str

Replaces all project constant placeholders.

Parameters

ret_val (str) – The string to parse for placeholders.

Returns

If a project placeholder has been found, the replaced value, the original

string ret_val else.

Return type

str

modules.helpers.web module

exception WebException

Bases: Exception

Exception that is raised from functions in this module.

doDownload(url: str, to: str = '', use_proxy: bool = False)None

Download data from the given URL to the given path.

Parameters
  • url (str) – The URL to download from

  • to (str, optional) – The path to save the download to. Defaults to “”.

  • use_proxy (bool, optional) – Should a proxy be used. Defaults to False.

Module contents