Developer Guide

Deploy for development

This project uses pip to manage the package. If you want to work on the project yourself you can create the necessary links via:

$ pip3 install --user -e .

That will install a backlink ~/.local/bin/kas to this project. Now you are able to call it from anywhere. For local development, use the run-kas wrapper from the project root directory. In this case, replace kas with path/to/run-kas.

Making Changes

These sections provide an overview of common modifications along with the required steps.

Changes of the project configuration

When changing the project configuration, you need to update the json configuration schema (schema-kas.json). Further, a short description of the changes needs to be added to Configuration Format Changes. After making the changes, you need to update the minimum and maximum values of header.version. If the version was already updated after the last release, the version bump is not required.

Add a new CLI option

Options that take a parameter (e.g. --format json) must be handled in kas-container as well. To keep the handling in kas-container simple, try to choose a unique option name across all plugins.

Add a new sub-command (plugin)

To add a new sub-command, you need to create a new python file in the kas/plugins directory. It then needs to be imported and registered in kas/plugins/__init__.py. Further, it needs to be registered in kas-container, as well as in the container-entrypoint.

Each sub-command must be documented and have its own man page. The documentation is generated from the docstrings of the sub-command file and must be registered in docs/userguide/plugins.rst. In addition, a manpage should be added in docs/_man/kas-plugin-<name> and registered in docs/conf.py (as kas-<name>.1).

Add support for new credentials

Credentials are passed using environment variables. For details, see Credential Handling. These can either contain the credential directly or point to a credential file (e.g. .netrc). To add support for a new credential, the following steps are required:

• document the variable in Environment Variables

• add the variable to the ENV_VARS list in libcmds.py::SetupHome

• add a forward of the variable in kas-container

• add the variable to the test_environment_variables.py test

For variables pointing to a credential file, the following applies in addition:

• the variable should end in _FILE (exceptions may apply)

• kas-container

  • bind-mount the variable into /var/kas/userdata/<credential file>

  • rewrite the variable to the path inside the container

If the variable does not end in _FILE, manual processing in the container-entrypoint script is needed to support it under rootless docker.

Container image build

To build the container images kas provides, there is a script provided for your convenience. It uses docker buildx and requires BuildKit 0.13.0 or newer. To start the build both container variants, invoke:

$ scripts/build-container.sh

You can limit the target type to either Yocto/OE (kas) or isar (kas-isar) via the --target options. See the script help for more options.

Since release 4.3, the containers officially provided via ghcr.io are fully reproducible. To test this, you can use the following script, e.g. to validate that release:

$ scripts/reproduce-container.sh kas:4.3

Both scripts also support building/checking of the arm64 container images. See the help of both scripts for more details.

Testing

The kas project has an extensive test suite. When adding new features or fixing bugs, it is recommended to add a test. The tests are written using the pytest framework.

Decoupling from the calling environment

Please make sure to decouple the tests from your local environment. To simplify this, we provide the monkeykas fixture to clean up the environment prior to each test. When adding new kas environment variables, make sure to add these to the cleanup handler as well.

When writing tests, no assumptions about the values of the following environment variables should be made (they also can be unset):

• KAS_WORK_DIR

• KAS_BUILD_DIR

As tests might want to check data in the work or build dir, we provide the following helpers to safely access the corresponding paths (by reading the value from the environment variable at call time):

• monkeykas.get_kwd(): absolute path to the current kas work dir

• monkeykas.get_kbd(): absolute path to the current kas build dir

• monkeykas.move_to_kwd(path): move the path to into the KAS_WORK_DIR, if needed

Tests that explicitly check for correct handling of the directory layout are encouraged to parameterize these paths by temporarily setting them via monkeykas.setenv(). Tests that rely on features that might be subject to the current working directory should be marked with the dirsfromenv marker. By that, various combinations of KAS_WORK_DIR and KAS_BUILD_DIR are tested.

Executing the testsuite

Note

The menu plugin tests require the snack package to be installed. On most distros this is packaged in python3-newt, on Arch Linux it is part of libnewt.

To run the tests, invoke:

$ python3 -m pytest

Online and offline testing

Some tests require internet access to fetch resources. These tests are marked with the online marker. To run these tests, invoke:

$ python3 -m pytest -m online

To run all tests except the online tests, invoke:

$ python3 -m pytest -m "not online"

When adding new tests, please consider whether they require internet access or not and mark them accordingly. In general, we prefer offline tests.

Measure code coverage

To measure the code coverage of the unit tests, the pytest-cov package is required. On Debian systems, this is provided in python3-pytest-cov. Once installed, run:

$ python3 -m pytest --cov --cov-report html

The coverage in HTML format can then be found in htmlcov.

Community Resources

Project home:

• https://github.com/siemens/kas

Source code:

• https://github.com/siemens/kas.git

• git@github.com:siemens/kas.git

Documentation:

• https://kas.readthedocs.org

Mailing list:

• kas-devel@googlegroups.com

• Subscription:

  • kas-devel+subscribe@googlegroups.com

  • https://groups.google.com/forum/#!forum/kas-devel/join

• Archives

  • https://groups.google.com/forum/#!forum/kas-devel

  • https://www.mail-archive.com/kas-devel@googlegroups.com/

Class reference documentation

kas.kas Module

This module is the main entry point for kas, setup tool for bitbake based projects. In case of user errors (e.g. invalid configuration, repo fetch failure) kas exits with error code 2, while exiting with 1 for internal errors. When cancelled by SIGINT, kas exits with 130. For details on error handling, see kas.kasusererror.

class kas.kas.ArgumentChoicesHelpFormatter(prog, indent_increment=2, max_help_position=24, width=None)

Help message formatter which adds choices to argument help.

If the default METAVAR is used, this will do nothing, as the default METAVAR shows the available choices already. If the METAVAR is overridden, and %(choice)s is not present in the help string, add them.

kas.kas.cleanup_logger()

Cleanup the logging environment

kas.kas.create_logger()

Setup the logging environment

kas.kas.interruption()

Gracefully cancel all tasks in the event loop

kas.kas.kas(argv)

The actual main entry point of kas.

kas.kas.kas_get_argparser()

Creates an argparser for kas with all plugins.

kas.kas.main()

The main function that operates as a wrapper around kas.

kas.kas.register_signal_handlers(loop)

Register the signal handlers which should be handled by the event loop. Implemented as function to monkey-patch it out in tests.

kas.kas.set_global_loglevel(level)

Configure the global log level. Implemented as function to monkey-patch it out in tests.

kas.kas.shutdown_loop(loop)

Waits for completion of the event loop but ignores any exceptions. The tasks are either already cancelled or will be transitively cancelled shortly. As this is the final cleanup, we cannot check for exceptions as these might lead in an unclosed event loops.

kas.kas.termination()

Forcefully terminate the process

kas.libkas Module

This module contains the core implementation of kas.

exception kas.libkas.EnvNotValidError

The caller environment is not suited for the requested operation

class kas.libkas.ExtendConstAction(option_strings, dest, const, default=None, required=False, help=None, metavar=None)

Add an ‘extend_const’ action similar to ‘append_const’.

Based on the existing ‘append_const’ and ‘extend’ actions.

exception kas.libkas.InitBuildEnvError

Error related to the OE / ISAR environment setup scripts

class kas.libkas.LogOutput(live)

Handles the log output of executed applications

log_stderr(line)

This method is called when a line is received over stderr.

log_stdout(line)

This method is called when a line is received over stdout.

exception kas.libkas.TaskExecError(command, ret_code)

Similar to kas.kasusererror.CommandExecError but for kas internal tasks

kas.libkas.add_cachedir_tag(dir, comment=None)

Create a CACHEDIR.TAG below dir. If a comment is provided, add this to the tag as well.

kas.libkas.find_program(paths, name)

Find a file within the paths array and returns its path.

kas.libkas.get_build_environ(build_system)

Creates the build environment variables.

kas.libkas.repos_apply_patches(repos)

Applies the patches to the repositories.

Note

termination point of the asyncio event loop.

kas.libkas.repos_fetch(repos)

Fetches the list of repositories to the kas_work_dir.

Note

termination point of the asyncio event loop.

kas.libkas.run_cmd(cmd, cwd, env=None, fail=True, capture_stderr=False)

Runs a command synchronously.

async kas.libkas.run_cmd_async(cmd, cwd, env=None, fail=True, liveupdate=False, capture_stderr=False)

Run a command asynchronously.

kas.libkas.ssh_add_key(env, key)

Adds an ssh key to the ssh-agent

kas.libkas.ssh_add_key_file(env, key_path)

Adds an ssh key file to the ssh-agent

kas.libkas.ssh_cleanup_agent()

Removes the identities and stops the ssh-agent instance

kas.libkas.ssh_no_host_key_check()

Disables ssh host key check

kas.libkas.ssh_setup_agent(envkeys=None)

Starts the ssh-agent

kas.libcmds Module

This module contains common commands used by kas plugins.

class kas.libcmds.CleanupSSHAgent

Removes all the identities and stops the ssh-agent instance.

execute(ctx)

This method executes the command.

class kas.libcmds.Command

An abstract class that defines the interface of a command.

execute(ctx)

This method executes the command.

class kas.libcmds.FinishSetupRepos

Finalizes the repo setup loop

execute(ctx)

This method executes the command.

class kas.libcmds.InitSetupRepos

Prepares setting up repos including the include logic

execute(ctx)

This method executes the command.

class kas.libcmds.Loop(name)

A class that defines a set of commands as a loop.

add(command)

Appends a command to the loop.

execute(ctx)

Executes the loop.

class kas.libcmds.Macro(use_common_setup=True)

Contains commands and provides method to run them.

add(command)

Appends commands to the command list.

run(ctx, skip=None)

Runs a command from the command list with respect to the configuration.

class kas.libcmds.MakeInteractive

Make execution environment interactive

execute(ctx)

This method executes the command.

class kas.libcmds.MakeNonInteractive

Make execution environment non-interactive

execute(ctx)

This method executes the command.

class kas.libcmds.ReposApplyPatches

Applies the patches defined in the configuration to the repositories.

execute(ctx)

This method executes the command.

class kas.libcmds.ReposCheckout

Ensures that the right revision of each repo is checked out.

execute(ctx)

This method executes the command.

class kas.libcmds.SetupDir

Creates the build directory.

execute(ctx)

This method executes the command.

class kas.libcmds.SetupEnviron

Sets up the kas environment.

execute(ctx)

This method executes the command.

class kas.libcmds.SetupHome

Sets up the home directory of kas.

execute(ctx)

This method executes the command.

class kas.libcmds.SetupReposStep

Single step of the checkout repos loop

execute(ctx)

This method executes the command.

class kas.libcmds.SetupSSHAgent

Sets up the ssh agent configuration.

execute(ctx)

This method executes the command.

class kas.libcmds.WriteBBConfig

Writes bitbake configuration files into the build directory.

execute(ctx)

This method executes the command.

kas.config Module

This module contains the implementation of the kas configuration.

class kas.config.Config(ctx, filename, target=None, task=None)

Implements the kas configuration based on config files.

find_missing_repos(repo_paths={})

Returns repos that are in config but not on disk and updates the internal config dictionary.

get_artifacts(missing_ok=True)

Returns the found artifacts after glob expansion, relative to the build_dir as a list of tuples (name, path). If missing_ok=False, raises an ArtifactNotFoundError if no artifact for a given name is found.

get_bblayers_conf_header()

Returns the bblayers.conf header

get_bitbake_targets()

Returns a list of bitbake targets

get_bitbake_task()

Returns the bitbake task

get_build_system()

Returns the pre-selected build system

get_buildtools()

Returns the buildtools keys: version, download URL and archive filename. These are provided so kas knows which buildtools archive to fetch and from what source.

get_config(remove_includes=False, apply_overrides=False)

Returns a copy of the config dict

get_distro()

Returns the distro

get_environment()

Returns the configured environment variables from the configuration file with possible overwritten values from the environment.

get_local_conf_header()

Returns the local.conf header

get_lockfiles()

Returns the list of in-use lockfiles.

get_machine()

Returns the machine

get_multiconfig()

Returns the multiconfig array as bitbake string

get_repo(name)

Returns a Repo instance for the configuration with the key name.

get_repos()

Returns the list of repos.

get_repos_config()

Returns the repository configuration

get_signers_config(keytype=None)

Returns the keys from the configuration

kas.repos Module

This module contains the Repo class.

class kas.repos.GitRepo(name, url, path, commit, tag, branch, refspec, layers, patches, signers, disable_operations)

Provides the git functionality for a Repo.

static get_type()

Repo type as defined in spdx-spec/v2.3/package-information

class kas.repos.MercurialRepo(name, url, path, commit, tag, branch, refspec, layers, patches, signers, disable_operations)

Provides the hg functionality for a Repo.

static get_type()

Repo type as defined in spdx-spec/v2.3/package-information

exception kas.repos.PatchApplyError(msg, cmd=None, out=None, err=None)

The provided patch file could not be applied

exception kas.repos.PatchFileNotFound

The requested patch file was not found

exception kas.repos.PatchMappingError

The requested patch can not be related to a repo

class kas.repos.Repo(name, url, path, commit, tag, branch, refspec, layers, patches, signers, disable_operations)

Represents a repository in the kas configuration.

static factory(name, repo_config, repo_defaults, repo_fallback_path, repo_overrides={})

Returns a Repo instance depending on parameters. This factory function is referential transparent.

static get_root_path(path, fallback=True)

Checks if path is under version control and returns its root path. If the repo is a submodule, the root path of the super-repository is returned.

static get_type()

Repo type as defined in spdx-spec/v2.3/package-information

exception kas.repos.RepoFetchError(repo, output)

An error occurred during repo fetching

class kas.repos.RepoImpl(name, url, path, commit, tag, branch, refspec, layers, patches, signers, disable_operations)

Provides a generic implementation for a Repo.

async apply_patches_async()

Applies patches to a repository asynchronously.

checkout()

Checks out the correct revision of the repo.

async fetch_async()

Starts asynchronous repository fetch.

class kas.repos.RepoLayer(name: str, priority: int, repo_name: str, repo_path: Path)

Standalone definition of a single bitbake layer.

exception kas.repos.RepoRefError

The requested repo reference is invalid, missing or could not be found

class kas.repos.SignatureValidator

Handles key loading and signature validation of repos based on the configuration.

exception kas.repos.UnsupportedRepoTypeError

The requested repo type is unsupported / not implemented

kas.includehandler Module

This module implements how includes of configuration files are handled in kas.

exception kas.includehandler.IncludeException

Class for exceptions that appear in the include mechanism.

class kas.includehandler.IncludeHandler(top_files, use_lock=True)

Implements a handler where every configuration file should contain a dictionary as the base type with and ‘includes’ key containing a list of includes.

The includes can be specified in two ways: as a string containing the path, relative to the repository root from the current file, or as a dictionary. The dictionary must have a ‘file’ key containing the path to the include file and a ‘repo’ key containing the key of the repository. The path is interpreted relative to the repository root path, which is lazy resolved by the first access of a method.

The includes are read and merged from the deepest level upwards.

In case use_lock is True, kas checks if a file <file>.lock.<ext> exists next to the first entry in top_files. This filename is then appended to the list of top_files.

ensure_from_same_repo()

Ensure that all concatenated config files belong to the same repository

get_config(repos=None)

Parameters:

repos – A dictionary that maps repo names to directory paths

Returns:

(config, repos)

config – A dictionary containing the configuration repos – A list of missing repo names that are needed to create a complete configuration

get_lock_filename(kasfile=None)

Returns the lockfile name for the given kas config file.

get_lockfiles()

Returns a list of lockfiles in the order the configuration files were parsed.

sanitize_include_path(base_path, include)

Ensure the created path is with the base_path. Returns the resolved path of base_path / include.

property top_repo_path

Lazy resolve top repo path as we might need a prepared environment

exception kas.includehandler.LoadConfigException(message, filename)

Class for exceptions that appear while loading a configuration file.

kas.kasusererror Module

This module provides a common base class for all exceptions which are related to user or configuration errors. These exceptions should be caught and reported to the user using a meaningful message instead of a stacktrace.

When handling errors in KAS, never return directly using sys.exit, but instead throw an exception derived from KasUserError (for user errors), or one derived from Exception for internal errors. These are then handled centrally, mapped to correct return codes and pretty printed.

exception kas.kasusererror.ArgsCombinationError(message)

Invalid combination of CLI arguments provided

exception kas.kasusererror.ArtifactNotFoundError(name, artifact)

A configured artifact is not found (or the glob matches 0 elements).

exception kas.kasusererror.CommandExecError(command, ret_code, forward_ret_code=False)

Failure in execution of a shell command. The forward_error_code parameter can be used to request the receiver of the exception to sys.exit with that code instead of a generic one. Only use this in special cases, where the return code can actually be related to a single shell command.

exception kas.kasusererror.EnvSetButNotFoundError(env_name, path)

A environment variable pointing to a file or directory is set, but the path it points to does not exist.

exception kas.kasusererror.KasUserError

User or input error. Derive all user error exceptions from this class.

exception kas.kasusererror.MissingModuleError(module, operation)

An optional module is missing for the requested operation

kas.plugins Module

This module contains and manages kas plugins

kas.plugins.all()

Get a list of all loaded kas plugin classes

kas.plugins.get(name)

Lookup a kas plugin class by name

kas.plugins.load()

Import all kas plugins

kas.plugins.register_plugins(mod)

Register all kas plugins found in a module