Developer Guide

Deploy for development

This project uses pip to manage the package. If you want to work on the project yourself you can create the necessary links via:

$ pip3 install --user -e .

That will install a backlink ~/.local/bin/kas to this project. Now you are able to call it from anywhere. For local development, use the run-kas wrapper from the project root directory. In this case, replace kas with path/to/run-kas.

Changes of the project configuration

When changing the project configuration, you need to update the json configuration schema (schema-kas.json). Further, a short description of the changes needs to be added to Configuration Format Changes. After making the changes, you need to update the minimum and maximum values of header.version. If the version was already updated after the last release, the version bump is not required.

Add a new sub-command (plugin)

To add a new sub-command, you need to create a new python file in the kas/plugins directory. It then needs to be imported and registered in kas/plugins/__init__.py. Further, it needs to be registered in kas-container, as well as in the container-entrypoint.

Each sub-command must be documented and have its own man page. The documentation is generated from the docstrings of the sub-command file and must be registered in docs/userguide/plugins.rst. In addition, a manpage should be added in docs/_man/kas-plugin-<name> and registered in docs/conf.py (as kas-<name>.1).

Container image build

To build the container images kas provides, there is a script provided for your convenience. It uses docker buildx and requires BuildKit 0.13.0 or newer. To start the build both container variants, invoke:

$ scripts/build-container.sh

You can limit the target type to either Yocto/OE (kas) or isar (kas-isar) via the --target options. See the script help for more options.

Since release 4.3, the containers officially provided via ghcr.io are fully reproducible. To test this, you can use the following script, e.g. to validate that release:

$ scripts/reproduce-container.sh kas:4.3

Both scripts also support building/checking of the arm64 container images. See the help of both scripts for more details.

Testing

The kas project has an extensive test suite. When adding new features or fixing bugs, it is recommended to add a test. The tests are written using the pytest framework. Please make sure to decouple the tests from your local environment. To simplify this, we provide the monkeykas fixture to clean up the environment prior to each test. When adding new kas environment variables, make sure to add these to the cleanup handler as well.

Note

The menu plugin tests require the snack package to be installed. On most distros this is packaged in python3-newt, on Arch Linux it is part of libnewt.

To run the tests, invoke:

$ python3 -m pytest

Online and offline testing

Some tests require internet access to fetch resources. These tests are marked with the online marker. To run these tests, invoke:

$ python3 -m pytest -m online

To run all tests except the online tests, invoke:

$ python3 -m pytest -m "not online"

When adding new tests, please consider whether they require internet access or not and mark them accordingly. In general, we prefer offline tests.

Measure code coverage

To measure the code coverage of the unit tests, the pytest-cov package is required. On Debian systems, this is provided in python3-pytest-cov. Once installed, run:

$ python3 -m pytest --cov --cov-report html

The coverage in HTML format can then be found in htmlcov.

Community Resources

Project home:

• https://github.com/siemens/kas

Source code:

• https://github.com/siemens/kas.git

• git@github.com:siemens/kas.git

Documentation:

• https://kas.readthedocs.org

Mailing list:

• kas-devel@googlegroups.com

• Subscription:

  • kas-devel+subscribe@googlegroups.com

  • https://groups.google.com/forum/#!forum/kas-devel/join

• Archives

  • https://groups.google.com/forum/#!forum/kas-devel

  • https://www.mail-archive.com/kas-devel@googlegroups.com/

Class reference documentation

kas.kas Module

This module is the main entry point for kas, setup tool for bitbake based projects. In case of user errors (e.g. invalid configuration, repo fetch failure) kas exits with error code 2, while exiting with 1 for internal errors. When cancelled by SIGINT, kas exits with 130. For details on error handling, see kas.kasusererror.

class kas.kas.ArgumentChoicesHelpFormatter(prog, indent_increment=2, max_help_position=24, width=None)

Help message formatter which adds choices to argument help.

If the default METAVAR is used, this will do nothing, as the default METAVAR shows the available choices already. If the METAVAR is overridden, and %(choice)s is not present in the help string, add them.

kas.kas.cleanup_logger()

Cleanup the logging environment

kas.kas.create_logger()

Setup the logging environment

kas.kas.interruption()

Gracefully cancel all tasks in the event loop

kas.kas.kas(argv)

The actual main entry point of kas.

kas.kas.kas_get_argparser()

Creates an argparser for kas with all plugins.

kas.kas.main()

The main function that operates as a wrapper around kas.

kas.kas.shutdown_loop(loop)

Waits for completion of the event loop

kas.kas.termination()

Forcefully terminate the process

kas.libkas Module

This module contains the core implementation of kas.

exception kas.libkas.EnvNotValidError

The caller environment is not suited for the requested operation

class kas.libkas.ExtendConstAction(option_strings, dest, const, default=None, required=False, help=None, metavar=None)

Add an ‘extend_const’ action similar to ‘append_const’.

Based on the existing ‘append_const’ and ‘extend’ actions.

exception kas.libkas.InitBuildEnvError

Error related to the OE / ISAR environment setup scripts

class kas.libkas.LogOutput(live)

Handles the log output of executed applications

log_stderr(line)

This method is called when a line is received over stderr.

log_stdout(line)

This method is called when a line is received over stdout.

exception kas.libkas.TaskExecError(command, ret_code)

Similar to kas.kasusererror.CommandExecError but for kas internal tasks

kas.libkas.find_program(paths, name)

Find a file within the paths array and returns its path.

kas.libkas.get_build_environ(build_system)

Creates the build environment variables.

kas.libkas.repos_apply_patches(repos)

Applies the patches to the repositories.

Note

termination point of the asyncio event loop.

kas.libkas.repos_fetch(repos)

Fetches the list of repositories to the kas_work_dir.

Note

termination point of the asyncio event loop.

kas.libkas.run_cmd(cmd, cwd, env=None, fail=True)

Runs a command synchronously.

async kas.libkas.run_cmd_async(cmd, cwd, env=None, fail=True, liveupdate=False)

Run a command asynchronously.

kas.libkas.ssh_add_key(env, key)

Adds an ssh key to the ssh-agent

kas.libkas.ssh_add_key_file(env, key_path)

Adds an ssh key file to the ssh-agent

kas.libkas.ssh_cleanup_agent()

Removes the identities and stops the ssh-agent instance

kas.libkas.ssh_no_host_key_check()

Disables ssh host key check

kas.libkas.ssh_setup_agent(envkeys=None)

Starts the ssh-agent

kas.libcmds Module

This module contains common commands used by kas plugins.

class kas.libcmds.CleanupSSHAgent

Removes all the identities and stops the ssh-agent instance.

execute(ctx)

This method executes the command.

class kas.libcmds.Command

An abstract class that defines the interface of a command.

execute(ctx)

This method executes the command.

class kas.libcmds.FinishSetupRepos

Finalizes the repo setup loop

execute(ctx)

This method executes the command.

class kas.libcmds.InitSetupRepos

Prepares setting up repos including the include logic

execute(ctx)

This method executes the command.

class kas.libcmds.Loop(name)

A class that defines a set of commands as a loop.

add(command)

Appends a command to the loop.

execute(ctx)

Executes the loop.

class kas.libcmds.Macro(use_common_setup=True)

Contains commands and provides method to run them.

add(command)

Appends commands to the command list.

run(ctx, skip=None)

Runs a command from the command list with respect to the configuration.

class kas.libcmds.ReposApplyPatches

Applies the patches defined in the configuration to the repositories.

execute(ctx)

This method executes the command.

class kas.libcmds.ReposCheckout

Ensures that the right revision of each repo is checked out.

execute(ctx)

This method executes the command.

class kas.libcmds.SetupDir

Creates the build directory.

execute(ctx)

This method executes the command.

class kas.libcmds.SetupEnviron

Sets up the kas environment.

execute(ctx)

This method executes the command.

class kas.libcmds.SetupHome

Sets up the home directory of kas.

execute(ctx)

This method executes the command.

class kas.libcmds.SetupReposStep

Single step of the checkout repos loop

execute(ctx)

This method executes the command.

class kas.libcmds.SetupSSHAgent

Sets up the ssh agent configuration.

execute(ctx)

This method executes the command.

class kas.libcmds.WriteBBConfig

Writes bitbake configuration files into the build directory.

execute(ctx)

This method executes the command.

kas.config Module

This module contains the implementation of the kas configuration.

class kas.config.Config(ctx, filename, target=None, task=None)

Implements the kas configuration based on config files.

find_missing_repos(repo_paths={})

Returns repos that are in config but not on disk

get_artifacts(missing_ok=True)

Returns the found artifacts after glob expansion, relative to the build_dir as a list of tuples (name, path). If missing_ok=False, raises an ArtifactNotFoundError if no artifact for a given name is found.

get_bblayers_conf_header()

Returns the bblayers.conf header

get_bitbake_targets()

Returns a list of bitbake targets

get_bitbake_task()

Returns the bitbake task

get_build_system()

Returns the pre-selected build system

get_config()

Returns the config dict.

get_distro()

Returns the distro

get_environment()

Returns the configured environment variables from the configuration file with possible overwritten values from the environment.

get_local_conf_header()

Returns the local.conf header

get_machine()

Returns the machine

get_multiconfig()

Returns the multiconfig array as bitbake string

get_repo(name)

Returns a Repo instance for the configuration with the key name.

get_repos()

Returns the list of repos.

get_repos_config()

Returns the repository configuration

kas.repos Module

This module contains the Repo class.

class kas.repos.GitRepo(name, url, path, commit, tag, branch, refspec, layers, patches, disable_operations)

Provides the git functionality for a Repo.

static get_type()

Repo type as defined in spdx-spec/v2.3/package-information

class kas.repos.MercurialRepo(name, url, path, commit, tag, branch, refspec, layers, patches, disable_operations)

Provides the hg functionality for a Repo.

static get_type()

Repo type as defined in spdx-spec/v2.3/package-information

exception kas.repos.PatchApplyError

The provided patch file could not be applied

exception kas.repos.PatchFileNotFound

The requested patch file was not found

exception kas.repos.PatchMappingError

The requested patch can not be related to a repo

class kas.repos.Repo(name, url, path, commit, tag, branch, refspec, layers, patches, disable_operations)

Represents a repository in the kas configuration.

static factory(name, repo_config, repo_defaults, repo_fallback_path, repo_overrides={})

Returns a Repo instance depending on params. This factory function is referential transparent.

static get_root_path(path, fallback=True)

Checks if path is under version control and returns its root path. If the repo is a submodule, the root path of the super-repository is returned.

static get_type()

Repo type as defined in spdx-spec/v2.3/package-information

class kas.repos.RepoImpl(name, url, path, commit, tag, branch, refspec, layers, patches, disable_operations)

Provides a generic implementation for a Repo.

async apply_patches_async()

Applies patches to a repository asynchronously.

checkout()

Checks out the correct revision of the repo.

async fetch_async()

Starts asynchronous repository fetch.

exception kas.repos.RepoRefError

The requested repo reference is invalid, missing or could not be found

exception kas.repos.UnsupportedRepoTypeError

The requested repo type is unsupported / not implemented

kas.includehandler Module

This module implements how includes of configuration files are handled in kas.

exception kas.includehandler.IncludeException

Class for exceptions that appear in the include mechanism.

class kas.includehandler.IncludeHandler(top_files, top_repo_path, use_lock=True)

Implements a handler where every configuration file should contain a dictionary as the base type with and ‘includes’ key containing a list of includes.

The includes can be specified in two ways: as a string containing the path, relative to the repository root from the current file, or as a dictionary. The dictionary must have a ‘file’ key containing the path to the include file and a ‘repo’ key containing the key of the repository. The path is interpreted relative to the repository root path.

The includes are read and merged from the deepest level upwards.

In case use_lock is True, kas checks if a file <file>.lock.<ext> exists next to the first entry in top_files. This filename is then appended to the list of top_files.

get_config(repos=None)

Parameters:

repos – A dictionary that maps repo names to directory paths

Returns:

(config, repos)

config – A dictionary containing the configuration repos – A list of missing repo names that are needed to create a complete configuration

exception kas.includehandler.LoadConfigException(message, filename)

Class for exceptions that appear while loading a configuration file.

kas.includehandler.load_config(filename)

Load the configuration file and test if version is supported.

kas.kasusererror Module

This module provides a common base class for all exceptions which are related to user or configuration errors. These exceptions should be caught and reported to the user using a meaningful message instead of a stacktrace.

When handling errors in KAS, never return directly using sys.exit, but instead throw an exception derived from KasUserError (for user errors), or one derived from Exception for internal errors. These are then handled centrally, mapped to correct return codes and pretty printed.

exception kas.kasusererror.ArgsCombinationError(message)

Invalid combination of CLI arguments provided

exception kas.kasusererror.ArtifactNotFoundError(name, artifact)

A configured artifact is not found (or the glob matches 0 elements).

exception kas.kasusererror.CommandExecError(command, ret_code, forward_ret_code=False)

Failure in execution of a shell command. The forward_error_code parameter can be used to request the receiver of the exception to sys.exit with that code instead of a generic one. Only use this in special cases, where the return code can actually be related to a single shell command.

exception kas.kasusererror.KasUserError

User or input error. Derive all user error exceptions from this class.

exception kas.kasusererror.MissingModuleError(module, operation)

An optional module is missing for the requested operation

kas.plugins Module

This module contains and manages kas plugins

kas.plugins.all()

Get a list of all loaded kas plugin classes

kas.plugins.get(name)

Lookup a kas plugin class by name

kas.plugins.load()

Import all kas plugins

kas.plugins.register_plugins(mod)

Register all kas plugins found in a module