Test Runner (Twister) — Zephyr Project Documentation (2024)

Board Configuration

To build tests for a specific board and to execute some of the tests on realhardware or in an emulation environment such as QEMU a board configuration fileis required which is generic enough to be used for other tasks that require aboard inventory with details about the board and its configuration that is onlyavailable during build time otherwise.

The board metadata file is located in the board directory and is structuredusing the YAML markup language. The example below shows a board with a datarequired for best test coverage for this specific board:

identifier: frdm_k64fname: NXP FRDM-K64Ftype: mcuarch: armtoolchain: - zephyr - gnuarmemb - xtoolssupported: - arduino_gpio - arduino_i2c - netif:eth - adc - i2c - nvs - spi - gpio - usb_device - watchdog - can - pwmtesting: default: true

identifier:

A string that matches how the board is defined in the build system. This samestring is used when building, for example when calling west build orcmake:

# with westwest build -b reel_board# with cmakecmake -DBOARD=reel_board ..

name:

The actual name of the board as it appears in marketing material.

type:

Type of the board or configuration, currently we support 2 types: mcu, qemu

simulation:

Simulator used to simulate the platform, e.g. qemu.

arch:

Architecture of the board

toolchain:

The list of supported toolchains that can build this board. This should matchone of the values used for ZEPHYR_TOOLCHAIN_VARIANT when building on the command line

ram:

Available RAM on the board (specified in KB). This is used to match testcaserequirements. If not specified we default to 128KB.

flash:

Available FLASH on the board (specified in KB). This is used to match testcaserequirements. If not specified we default to 512KB.

supported:

A list of features this board supports. This can be specified as a single wordfeature or as a variant of a feature class. For example:

supported: - pci

This indicates the board does support PCI. You can make a testcase build orrun only on such boards, or:

supported: - netif:eth - sensor:bmi16

A testcase can both depend on ‘eth’ to only test ethernet or on ‘netif’ to runon any board with a networking interface.

testing:

testing relating keywords to provide best coverage for the features of thisboard.

default: [True|False]:

This is a default board, it will tested with the highest priority and iscovered when invoking the simplified twister without any additionalarguments.

ignore_tags:

Do not attempt to build (and therefore run) tests marked with this list oftags.

only_tags:

Only execute tests with this list of tags on a specific platform.

timeout_multiplier: <float> (default 1)

Multiply each test case timeout by specified ratio. This option allows to tune timeouts onlyfor required platform. It can be useful in case naturally slow platform I.e.: HW board withpower-efficient but slow CPU or simulation platform which can perform instruction accuratesimulation but does it slowly.

env:

A list of environment variables. Twister will check if all these environment variables are set,and otherwise skip this platform. This allows the user to define a platform which should beused, for example, only if some required software or hardware is present, and to signal thatpresence to twister using these environment variables.

Test Cases

Test cases are detected by the presence of a testcase.yaml or a sample.yamlfiles in the application’s project directory. This file may contain one or moreentries in the test section each identifying a test scenario.

The name of each testcase needs to be unique in the context of the overalltestsuite and has to follow basic rules:

1. The format of the test identifier shall be a string without any spaces orspecial characters (allowed characters: alphanumeric and [_=]) consistingof multiple sections delimited with a dot (.).

2. Each test identifier shall start with a section followed by a subsectionseparated by a dot. For example, a test that covers semaphores in the kernelshall start with kernel.semaphore.

3. All test identifiers within a testcase.yaml file need to be unique. Forexample a testcase.yaml file covering semaphores in the kernel can have:

   • kernel.semaphore: For general semaphore tests

   • kernel.semaphore.stress: Stress testing semaphores in the kernel.

4. Depending on the nature of the test, an identifier can consist of at leasttwo sections:

   • Ztest tests: The individual testcases in the ztest testsuite will beconcatenated to identifier in the testcase.yaml file generating uniqueidentifiers for every testcase in the suite.

   • Standalone tests and samples: This type of test should at least have 3sections in the test identifier in the testcase.yaml (or sample.yaml) file.The last section of the name shall signify the test itself.

Test cases are written using the YAML syntax and share the same structure assamples. The following is an example test with a few options that areexplained in this document.

tests: bluetooth.gatt: build_only: true platform_allow: qemu_cortex_m3 qemu_x86 tags: bluetooth bluetooth.gatt.br: build_only: true extra_args: CONF_FILE="prj_br.conf" filter: not CONFIG_DEBUG platform_exclude: up_squared platform_allow: qemu_cortex_m3 qemu_x86 tags: bluetooth

A sample with tests will have the same structure with additional informationrelated to the sample and what is being demonstrated:

sample: name: hello world description: Hello World sample, the simplest Zephyr applicationtests: sample.basic.hello_world: build_only: true tags: tests min_ram: 16 sample.basic.hello_world.singlethread: build_only: true extra_args: CONF_FILE=prj_single.conf filter: not CONFIG_BT tags: tests min_ram: 16

The full canonical name for each test case is:<path to test case>/<test entry>

Each test block in the testcase meta data can define the following key/valuepairs:

tags: <list of tags> (required)

A set of string tags for the testcase. Usually pertains tofunctional domains but can be anything. Command line invocationsof this script can filter the set of tests to run based on tag.

skip: <True|False> (default False)

skip testcase unconditionally. This can be used for broken tests.

slow: <True|False> (default False)

Don’t run this test case unless --enable-slow or --enable-slow-only waspassed in on the command line. Intended for time-consuming test cases thatare only run under certain circ*mstances, like daily builds. These testcases are still compiled.

extra_args: <list of extra arguments>

Extra arguments to pass to Make when building or running thetest case.

extra_configs: <list of extra configurations>

Extra configuration options to be merged with a master prj.confwhen building or running the test case. For example:

common: tags: drivers adctests: test: depends_on: adc test_async: extra_configs: - CONFIG_ADC_ASYNC=y

See Also

Test Framework — Zephyr Project DocumentationZephyr for JIRA Tutorial: Test Management ToolJira and Zephyr: Track Your Team's Progress | AtlassianJira und Zephyr: Verfolgen des Teamfortschritts | Atlassian

Using namespacing, it is possible to apply a configuration only to somehardware. Currently both architectures and platforms are supported:

common: tags: drivers adctests: test: depends_on: adc test_async: extra_configs: - arch:x86:CONFIG_ADC_ASYNC=y - platform:qemu_x86:CONFIG_DEBUG=y

build_only: <True|False> (default False)

If true, twister will not try to run the test even if the test is runnableon the platform.

This keyword is reserved for tests that are used to test if some codeactually builds. A build_only test is not designed to be run in anyenvironment and should not be testing any functionality, it only verifiesthat the code builds.

This option is often used to test drivers and the fact that they are correctlyenabled in Zephyr and that the code builds, for example sensor drivers. Suchtest shall not be used to verify the functionality of the driver.

build_on_all: <True|False> (default False)

If true, attempt to build test on all available platforms. This is mostlyused in CI for increased coverage. Do not use this flag in new tests.

depends_on: <list of features>

A board or platform can announce what features it supports, this optionwill enable the test only those platforms that provide this feature.

levels: <list of levels>

Test levels this test should be part of. If a level is present, thistest will be selectable using the command line option --level <level name>

min_ram: <integer>

minimum amount of RAM in KB needed for this test to build and run. This iscompared with information provided by the board metadata.

min_flash: <integer>

minimum amount of ROM in KB needed for this test to build and run. This iscompared with information provided by the board metadata.

timeout: <number of seconds>

Length of time to run test before automatically killing it.Default to 60 seconds.

arch_allow: <list of arches, such as x86, arm, arc>

Set of architectures that this test case should only be run for.

arch_exclude: <list of arches, such as x86, arm, arc>

Set of architectures that this test case should not run on.

platform_allow: <list of platforms>

Set of platforms that this test case should only be run for. Do not usethis option to limit testing or building in CI due to time or resourceconstraints, this option should only be used if the test or sample canonly be run on the allowed platform and nothing else.

integration_platforms: <YML list of platforms/boards>

This option limits the scope to the listed platforms when twister isinvoked with the --integration option. Use this instead ofplatform_allow if the goal is to limit scope due to timing orresource constraints.

platform_exclude: <list of platforms>

Set of platforms that this test case should not run on.

extra_sections: <list of extra binary sections>

When computing sizes, twister will report errors if it findsextra, unexpected sections in the Zephyr binary unless they are namedhere. They will not be included in the size calculation.

sysbuild: <True|False> (default False)

Build the project using sysbuild infrastructure. Only the main project’sgenerated devicetree and Kconfig will be used for filtering tests.on device testing must use the hardware map, or west flash to loadthe images onto the target. The --erase option of west flash isnot supported with this option. Usage of unsupported options will resultin tests requiring sysbuild support being skipped.

harness: <string>

A harness keyword in the testcase.yaml file identifies a Twisterharness needed to run a test successfully. A harness is a feature ofTwister and implemented by Twister, some harnesses are defined asplaceholders and have no implementation yet.

A harness can be seen as the handler that needs to be implemented inTwister to be able to evaluate if a test passes criteria. For example, akeyboard harness is set on tests that require keyboard interaction to reachverdict on whether a test has passed or failed, however, Twister lack thisharness implementation at the moment.

Supported harnesses:

• ztest

• test

• console

• pytest

• gtest

• robot

Harnesses ztest, gtest and console are based on parsing of theoutput and matching certain phrases. ztest and gtest harnesses lookfor pass/fail/etc. frames defined in those frameworks. Use gtestharness if you’ve already got tests written in the gTest framework and donot wish to update them to zTest. The console harness tells Twister toparse a test’s text output for a regex defined in the test’s YAML file.The robot harness is used to execute Robot Framework test suitesin the Renode simulation framework.

Some widely used harnesses that are not supported yet:

• keyboard

• net

• bluetooth

Harness bsim is implemented in limited way - it helps only to copy thefinal executable (zephyr.exe) from build directory to BabbleSim’sbin directory (${BSIM_OUT_PATH}/bin). This action is useful to allowBabbleSim’s tests to directly run after. By default, the executable filename is (with dots and slashes replaced by underscores):bs_<platform_name>_<test_path>_<test_scenario_name>.This name can be overridden with the bsim_exe_name option inharness_config section.

platform_key: <list of platform attributes>

Often a test needs to only be built and run once to qualify as passing.Imagine a library of code that depends on the platform architecture wherepassing the test on a single platform for each arch is enough to qualify thetests and code as passing. The platform_key attribute enables doing justthat.

For example to key on (arch, simulation) to ensure a test is run onceper arch and simulation (as would be most common):

platform_key: - arch - simulation

Adding platform (board) attributes to include things such as soc name,soc family, and perhaps sets of IP blocks implementing each peripheralinterface would enable other interesting uses. For example, this could enablebuilding and running SPI tests once for each unique IP block.

harness_config: <harness configuration options>

Extra harness configuration options to be used to select a board and/orfor handling generic Console with regex matching. Config can announcewhat features it supports. This option will enable the test to run ononly those platforms that fulfill this external dependency.

The following options are currently supported:

type: <one_line|multi_line> (required)

Depends on the regex string to be matched

regex: <list of regular expressions> (required)

Strings with regular expressions to match with the test’s outputto confirm the test runs as expected.

ordered: <True|False> (default False)

Check the regular expression strings in orderly or randomly fashion

record: <recording options> (optional)

regex: <regular expression> (required)

The regular expression with named subgroups to match data fieldsat the test’s output lines where the test provides some custom datafor further analysis. These records will be written into the builddirectory ‘recording.csv’ file as well as ‘recording’ propertyof the test suite object in ‘twister.json’.

For example, to extract three data fields ‘metric’, ‘cycles’, ‘nanoseconds’:

record: regex: "(?P<metric>.*):(?P<cycles>.*) cycles, (?P<nanoseconds>.*) ns"

fixture: <expression>

Specify a test case dependency on an external device(e.g., sensor),and identify setups that fulfill this dependency. It depends onspecific test setup and board selection logic to pick the particularboard(s) out of multiple boards that fulfill the dependency in anautomation setup based on fixture keyword. Some sample fixture namesare i2c_hts221, i2c_bme280, i2c_FRAM, ble_fw and gpio_loop.

Only one fixture can be defined per testcase and the fixture name has tobe unique across all tests in the test suite.

pytest_root: <list of pytest testpaths> (default pytest)

Specify a list of pytest directories, files or subtests that need to beexecuted when a test case begins to run. The default pytest directory ispytest. After the pytest run is finished, Twister will check ifthe test case passed or failed according to the pytest report.As an example, a list of valid pytest roots is presented below:

harness_config: pytest_root: - "pytest/test_shell_help.py" - "../shell/pytest/test_shell.py" - "/tmp/test_shell.py" - "~/tmp/test_shell.py" - "$ZEPHYR_BASE/samples/subsys/testsuite/pytest/shell/pytest/test_shell.py" - "pytest/test_shell_help.py::test_shell2_sample" # select pytest subtest - "pytest/test_shell_help.py::test_shell2_sample[param_a]" # select pytest parametrized subtest

pytest_args: <list of arguments> (default empty)

Specify a list of additional arguments to pass to pytest e.g.:pytest_args: [‘-k=test_method’, ‘--log-level=DEBUG’]. Note that--pytest-args can be passed multiple times to pass several argumentsto the pytest.

pytest_dut_scope: <function|class|module|package|session> (default function)

The scope for which dut and shell pytest fixtures are shared.If the scope is set to function, DUT is launched for every test casein python script. For session scope, DUT is launched only once.

robot_test_path: <robot file path> (default empty)

Specify a path to a file containing a Robot Framework test suite to be run.

bsim_exe_name: <string>

If provided, the executable filename when copying to BabbleSim’s bindirectory, will be bs_<platform_name>_<bsim_exe_name> instead of thedefault based on the test path and scenario name.

The following is an example yaml file with a few harness_config options.

sample: name: HTS221 Temperature and Humidity Monitorcommon: tags: sensor harness: console harness_config: type: multi_line ordered: false regex: - "Temperature:(.*)C" - "Relative Humidity:(.*)%" fixture: i2c_hts221tests: test: tags: sensors depends_on: i2c

The following is an example yaml file with pytest harness_config options,default pytest_root name “pytest” will be used if pytest_root not specified.please refer the examples in samples/subsys/testsuite/pytest/.

common: harness: pytesttests: pytest.example.directories: harness_config: pytest_root: - pytest_dir1 - $ENV_VAR/samples/test/pytest_dir2 pytest.example.files_and_subtests: harness_config: pytest_root: - pytest/test_file_1.py - test_file_2.py::test_A - test_file_2.py::test_B[param_a]

The following is an example yaml file with robot harness_config options.

tests: robot.example: harness: robot harness_config: robot_test_path: [robot file path]

filter: <expression>

Filter whether the testcase should be run by evaluating an expressionagainst an environment containing the following values:

{ ARCH : <architecture>, PLATFORM : <platform>, <all CONFIG_* key/value pairs in the test's generated defconfig>, *<env>: any environment variable available}

Twister will first evaluate the expression to find if a “limited” cmake call, i.e. using package_helper cmake script,can be done. Existence of “dt_*” entries indicates devicetree is needed.Existence of “CONFIG*” entries indicates kconfig is needed.If there are no other types of entries in the expression a filtration can be done without creating a complete build system.If there are entries of other types a full cmake is required.

The grammar for the expression language is as follows:

expression : expression 'and' expression | expression 'or' expression | 'not' expression | '(' expression ')' | symbol '==' constant | symbol '!=' constant | symbol '<' NUMBER | symbol '>' NUMBER | symbol '>=' NUMBER | symbol '<=' NUMBER | symbol 'in' list | symbol ':' STRING | symbol ;list : '[' list_contents ']';list_contents : constant (',' constant)*;constant : NUMBER | STRING;

For the case where expression ::= symbol, it evaluates to trueif the symbol is defined to a non-empty string.

Operator precedence, starting from lowest to highest:

See Also

Zephyr für JIRA-Tutorial: Testmanagement-Tool

• or (left associative)

• and (left associative)

• not (right associative)

• all comparison operators (non-associative)

arch_allow, arch_exclude, platform_allow, platform_excludeare all syntactic sugar for these expressions. For instance:

arch_exclude = x86 arc

Is the same as:

filter = not ARCH in ["x86", "arc"]

The : operator compiles the string argument as a regular expression,and then returns a true value only if the symbol’s value in the environmentmatches. For example, if CONFIG_SOC="stm32f107xc" then

filter = CONFIG_SOC : "stm.*"

Would match it.

required_snippets: <list of needed snippets>

Snippets are supported in twister for test cases thatrequire them. As with normal applications, twister supports using the basezephyr snippet directory and test application directory for findingsnippets. Listed snippets will filter supported tests for boards (snippetsmust be compatible with a board for the test to run on them, they are notoptional).

The following is an example yaml file with 2 required snippets.

tests: snippet.example: required_snippets: - cdc-acm-console - user-snippet-example

The set of test cases that actually run depends on directives in the testcasefiled and options passed in on the command line. If there is any confusion,running with -v or examining the discard report(twister_discard.csv) can help show why particular test cases wereskipped.

Metrics (such as pass/fail state and binary size) for the last coderelease are stored in scripts/release/twister_last_release.csv.To update this, pass the --all --release options.

To load arguments from a file, add + before the file name, e.g.,+file_name. File content must be one or more valid arguments separated byline break instead of white spaces.

Most everyday users will run with no arguments.

Managing tests timeouts

There are several parameters which control tests timeouts on various levels:

• timeout option in each test case. See here for moredetails.

• timeout_multiplier option in board configuration. Seehere for more details.

• --timeout-multiplier twister option which can be used to adjust timeouts in exact twister run.It can be useful in case of simulation platform as simulation time may depend on the hostspeed & load or we may select different simulation method (i.e. cycle accurate but slowerone), etc…

Overall test case timeout is a multiplication of these three parameters.

Running in Integration Mode

This mode is used in continuous integration (CI) and other automatedenvironments used to give developers fast feedback on changes. The mode canbe activated using the --integration option of twister and narrows downthe scope of builds and tests if applicable to platforms defined under theintegration keyword in the testcase definition file (testcase.yaml andsample.yaml).

Running tests on custom emulator

Apart from the already supported QEMU and other simulated environments, Twistersupports running any out-of-tree custom emulator defined in the board’s board.cmake.To use this type of simulation, add the following properties tocustom_board/custom_board.yaml:

simulation: customsimulation_exec: <name_of_emu_binary>

This tells Twister that the board is using a custom emulator called <name_of_emu_binary>,make sure this binary exists in the PATH.

Then, in custom_board/board.cmake, set the supported emulation platforms to custom:

set(SUPPORTED_EMU_PLATFORMS custom)

Finally, implement the run_custom target in custom_board/board.cmake.It should look something like this:

add_custom_target(run_custom COMMAND <name_of_emu_binary to invoke during 'run'> <any args to be passed to the command, i.e. ${BOARD}, ${APPLICATION_BINARY_DIR}/zephyr/zephyr.elf> WORKING_DIRECTORY ${APPLICATION_BINARY_DIR} DEPENDS ${logical_target_for_zephyr_elf} USES_TERMINAL )

Running Tests on Hardware

Beside being able to run tests in QEMU and other simulated environments,twister supports running most of the tests on real devices and producesreports for each run with detailed FAIL/PASS results.

scripts/twister --device-testing --device-serial /dev/ttyACM0 \--device-serial-baud 115200 -p frdm_k64f -T tests/kernel

scripts/twister --device-testing --device-serial-pty "script.py" \-p intel_adsp/cavs25 -T tests/kernel

Executing tests on multiple devices

To build and execute tests on multiple devices connected to the host PC, ahardware map needs to be created with all connected devices and theirdetails such as the serial device, baud and their IDs if available.Run the following command to produce the hardware map:

./scripts/twister --generate-hardware-map map.yml

The generated hardware map file (map.yml) will have the list of connecteddevices, for example:

- connected: true id: OSHW000032254e4500128002ab98002784d1000097969900 platform: unknown product: DAPLink CMSIS-DAP runner: pyocd serial: /dev/cu.usbmodem146114202- connected: true id: 000683759358 platform: unknown product: J-Link runner: unknown serial: /dev/cu.usbmodem0006837593581

Any options marked as unknown need to be changed and set with the correctvalues, in the above example the platform names, the products and the runners needto be replaced with the correct values corresponding to the connected hardware.In this example we are using a reel_board and an nrf52840dk/nrf52840:

- connected: true id: OSHW000032254e4500128002ab98002784d1000097969900 platform: reel_board product: DAPLink CMSIS-DAP runner: pyocd serial: /dev/cu.usbmodem146114202 baud: 9600- connected: true id: 000683759358 platform: nrf52840dk/nrf52840 product: J-Link runner: nrfjprog serial: /dev/cu.usbmodem0006837593581 baud: 9600

The baud entry is only needed if not running at 115200.

If the map file already exists, then new entries are added and existing entrieswill be updated. This way you can use one single master hardware map and updateit for every run to get the correct serial devices and status of the devices.

With the hardware map ready, you can run any tests by pointing to the map

./scripts/twister --device-testing --hardware-map map.yml -T samples/hello_world/

The above command will result in twister building tests for the platformsdefined in the hardware map and subsequently flashing and running the testson those platforms.

Note

Currently only boards with support for pyocd, nrfjprog, jlink, openocd, or dediprogare supported with the hardware map features. Boards that require other runners to flash theZephyr binary are still work in progress.

Hardware map allows to set --device-flash-timeout and --device-flash-with-testcommand line options as flash-timeout and flash-with-test fields respectively.These hardware map values override command line options for the particular platform.

Serial PTY support using --device-serial-pty can also be used in thehardware map:

- connected: true id: None platform: intel_adsp/cavs25 product: None runner: intel_adsp serial_pty: path/to/script.py runner_params: - --remote-host=remote_host_ip_addr - --key=/path/to/key.pem

The runner_params field indicates the parameters you want to pass to thewest runner. For some boards the west runner needs some extra parameters towork. It is equivalent to following west and twister commands.

west flash --remote-host remote_host_ip_addr --key /path/to/key.pemtwister -p intel_adsp/cavs25 --device-testing --device-serial-pty script.py--west-flash="--remote-host=remote_host_ip_addr,--key=/path/to/key.pem"

Note

For serial PTY, the “–generate-hardware-map” option cannot scan it outand generate a correct hardware map automatically. You have to edit itmanually according to above example. This is because the serial portof the PTY is not fixed and being allocated in the system at runtime.

Fixtures

Some tests require additional setup or special wiring specific to the test.Running the tests without this setup or test fixture may fail. A testcase canspecify the fixture it needs which can then be matched with hardware capabilityof a board and the fixtures it supports via the command line or using the hardwaremap file.

Fixtures are defined in the hardware map file as a list:

- connected: true fixtures: - gpio_loopback id: 0240000026334e450015400f5e0e000b4eb1000097969900 platform: frdm_k64f product: DAPLink CMSIS-DAP runner: pyocd serial: /dev/ttyACM9

When running twister with --device-testing, the configured fixturein the hardware map file will be matched to testcases requesting the same fixturesand these tests will be executed on the boards that provide this fixture.

Fixtures can also be provided via twister command option --fixture, this optioncan be used multiple times and all given fixtures will be appended as a list. And thegiven fixtures will be assigned to all boards, this means that all boards set bycurrent twister command can run those testcases which request the same fixtures.

Notes

It may be useful to annotate board descriptions in the hardware map filewith additional information. Use the notes keyword to do this. Forexample:

- connected: false fixtures: - gpio_loopback id: 000683290670 notes: An nrf5340dk/nrf5340 is detected as an nrf52840dk/nrf52840 with no serial port, and three serial ports with an unknown platform. The board id of the serial ports is not the same as the board id of the development kit. If you regenerate this file you will need to update serial to reference the third port, and platform to nrf5340dk/nrf5340/cpuapp or another supported board target. platform: nrf52840dk/nrf52840 product: J-Link runner: jlink serial: null

Overriding Board Identifier

When (re-)generated the hardware map file will contain an id keywordthat serves as the argument to --board-id when flashing. In somecases the detected ID is not the correct one to use, for example whenusing an external J-Link probe. The probe_id keyword overrides theid keyword for this purpose. For example:

- connected: false id: 0229000005d9ebc600000000000000000000000097969905 platform: mimxrt1060_evk probe_id: 000609301751 product: DAPLink CMSIS-DAP runner: jlink serial: null

Quarantine

Twister allows user to provide configuration files defining a list of tests orplatforms to be put under quarantine. Such tests will be skipped and markedaccordingly in the output reports. This feature is especially useful whenrunning larger test suits, where a failure of one test can affect the executionof other tests (e.g. putting the physical board in a corrupted state).

To use the quarantine feature one has to add the argument--quarantine-list <PATH_TO_QUARANTINE_YAML> to a twister call.Multiple quarantine files can be used.The current status of tests on the quarantine list can also be verified by adding--quarantine-verify to the above argument. This will make twister skip all testswhich are not on the given list.

A quarantine yaml has to be a sequence of dictionaries. Each dictionary has to havescenarios and platforms entries listing combinations of scenarios and platformsto put under quarantine. In addition, an optional entry comment can be used, wheresome more details can be given (e.g. link to a reported issue). These comments will alsobe added to the output reports.

When quarantining a class of tests or many scenarios in a single testsuite orwhen dealing with multiple issues within a subsystem, it is possible to useregular expressions, for example, kernel.* would quarantineall kernel tests.

An example of entries in a quarantine yaml:

- scenarios: - sample.basic.helloworld comment: "Link to the issue: https://github.com/zephyrproject-rtos/zephyr/pull/33287"- scenarios: - kernel.common - kernel.common.(misra|tls) - kernel.common.nano64 platforms: - .*_cortex_.* - native_sim

To exclude a platform, use the following syntax:

- platforms: - qemu_x86 comment: "broken qemu"

Additionally you can quarantine entire architectures or a specific simulator for executing tests.

Test Configuration

A test configuration can be used to customize various aspects of twisterand the default enabled options and features. This allows tweaking the filteringcapabilities depending on the environment and makes it possible to adapt andimprove coverage when targeting different sets of platforms.

The test configuration also adds support for test levels and the ability toassign a specific test to one or more levels. Using command line options oftwister it is then possible to select a level and just execute the testsincluded in this level.

Additionally, the test configuration allows defining leveldependencies and additional inclusion of tests into a specific level ifthe test itself does not have this information already.

In the configuration file you can include complete components usingregular expressions and you can specify which test level to import fromthe same file, making management of levels easier.

To help with testing outside of upstream CI infrastructure, additionaloptions are available in the configuration file, which can be hostedlocally. As of now, those options are available:

• Ability to ignore default platforms as defined in board definitions(Those are mostly emulation platforms used to run tests in upstreamCI)

• Option to specify your own list of default platforms overriding whatupstream defines.

• Ability to override build_on_all options used in some testcases.This will treat tests or sample as any other just build for defaultplatforms you specify in the configuration file or on the command line.

• Ignore some logic in twister to expand platform coverage in cases wheredefault platforms are not in scope.

platforms: override_default_platforms: true increased_platform_scope: false default_platforms: - qemu_x86

levels: - name: my-test-level description: > my custom test level adds: - kernel.threads.* - kernel.timer.behavior - arch.interrupt - boards.*

platforms: override_default_platforms: true default_platforms: - frdm_k64flevels: - name: smoke description: > A plan to be used verifying basic zephyr features. - name: unit description: > A plan to be used verifying unit test. - name: integration description: > A plan to be used verifying integration. - name: acceptance description: > A plan to be used verifying acceptance. - name: system description: > A plan to be used verifying system. - name: regression description: > A plan to be used verifying regression.

scripts/twister --test-config=<path to>/test_config.yaml -T tests --level="smoke"

Running in Tests in Random Order

Enable ZTEST framework’s CONFIG_ZTEST_SHUFFLE config option torun your tests in random order. This can be beneficial for identifyingdependencies between test cases. For native_sim platforms, you can providethe seed to the random number generator by providing -seed=value as anargument to twister. See Shuffling Test Sequence for moredetails.

Robot Framework Tests

Zephyr supports Robot Framework as one of solutions for automated testing.

Robot files allow you to express interactive test scenarios in human-readable text format and execute them in simulation or against hardware.At this moment Zephyr integration supports running Robot tests in the Renode simulation framework.

To execute a Robot test suite with twister, run the following command:

$ ./scripts/twister --platform hifive1 --test samples/subsys/shell/shell_module/sample.shell.shell_module.robot

$ twister -p qemu_riscv32 -s tests/kernel/interrupt/arch.shared_interrupt