Properties, Environments and test matrix generation

understand what they mean and how they work

Posted by Alexander Todorov on Mon 24 January 2022

The upcoming Kiwi TCMS v11 contains new functionality around TestCase properties and TestRun environments which has an impact on how your final test execution matrix will look like. This article provides detailed information about these features but have in mind that they are still considered to be a technology-preview.

Properties

Consider a login functionality which accepts email address and password. Possible states for these fields are:

Depending on how the software under test is put together you can design multiple test cases. Fundamentally however these are the same test case and the above states are input parameters to it!

Definition: TestCase properties are different input values which do not fundamentally affect the result of a test case! A TestCase with properties will result into multiple test executions!

In other words you will be executing a parameterized test scenario multiple times with different input values! Inside Kiwi TCMS the actual parameter values during execution are recorded into the TestExecution model which will not change if you modify test case property values afterwards!

Definition: TestExecution properties record a snapshot of TestCase properties at the time when you intended to execute a particular test scenario!

Environments

A testing environment represents the specifics of where exactly you executed your test suite. Consider this example:

The default desktop environment of Fedora is GNOME, but if you prefer an alternative, you can download installation media which contains slightly differently defaults, e.g. KDE, Xfce, MATE and others, see https://spins.fedoraproject.org.

Regardless of which Fedora variant you choose the expected functionallity of the default desktop experience is the same. However this can only be guaranteed with exhaustive testing across all variants. Check-out the test matrix at https://fedoraproject.org/wiki/Test_Results:Fedora_36_Rawhide_20220118.n.0_Desktop?rd=Test_Results:Current_Desktop_Test#Non_release-blocking_desktops:_x86_.2F_x86_64

Definition: a TestRun environment is a set of properties which apply to the entire test suite at the time of execution. Usually you expect test results in different environments to be the same!

In Kiwi TCMS environments are represented as named containers of key/value pairs. The same key may have multiple values! They can be found under ADMIN -> Everything else -> Environments.

Because environments are meant to affect the entire test suite they are linked to the TestRun model. When creating a new test run you can select multiple Environment records.

Test matrix generation

The existing behavior in Kiwi TCMS is that when a test run is created there will be only one test execution for every test case which is added inside this test run.

In the Fedora example shown above some of the test cases also have their own parameters, e.g. the QA:Testcase_desktop_app_basic scenario.

Definition: TestRun environment key/values will be combined with TestCase property key/values to form the final test matrix! This opens up the possibility for combinatorial test execution generation.

Once properties and environment(s) are specified you will start seeing multiple test executions for the same test case inside newly created test runs. By default a full-combination test matrix will be created. The other option is to pairwise all key/value records.

Important: test execution generation works only when creating or cloning a test run that contains test cases. This feature still does not work for test cases added after a test run is created!

Environment(s) vs Tag(s)

Inside Kiwi TCMS you can use both environments and tags to annotate test runs. There are 3 important facts that hold true:

"Example from #1344"

If we look at this example from Issue #1344 we can make out the following keys:

Here Driver, API and Python clearly should affect your test matrix. Otherwise there isn't much point in having the different values recorded in the first place. That results in a 8x multiplication factor for every functionality that may be affected/related to these attributes, presumably the entire functionality of the product under test.

Java, Eclipse, Host OS and Target OS carry only informational value but they look like more values could be possible. If that's the case these attributes will also affect the overall test matrix.

Redistributable and Testing Type look like information-only attributes. They don't appear to have any relevance to the test matrix at all. The same information-only effect can be achieved both with environments and with tags.

Practical rules:

  1. Attributes which affect a single test case should be defined as TestCase properties
  2. Attributes which affect all test cases in a suite should be defined as TestRun environment(s)
  3. One big TestRun is likely the best from organizational and optimizational point of view

You may decide to have multiple smaller test runs, usually with 1 value per environment key, if you think that fits your workflow better. However you may be missing on some optimizations if you choose to do so.

Real life example

To illustrate how all of these new features work let's look at Partitioning custom software RAID test case from Fedora QA. It instructs the tester to install Linux and inside the partitioning screen create a Software RAID partition, format it with a filesystem and assign a mount point! It is expected that once installation is complete the machine will reboot, tester will be able to login as root and the created filesystem will be available!

Factors that could affect this functionality:

We may add the actual filesystem type, used to format the RAID block device, e.g. xfs, ext3, ext4, but that's not needed here! Here's how this test case looks in Kiwi TCMS:

"RAID test case"

Remember that Fedora comes with multiple variants for multiple CPU architectures! Of those we'll consider Server and Workstation, which are both available for the aarch64 and x86_64 CPU architectures. Here's how this can be represented inside Kiwi TCMS:

"Fedora variants represented as environment"



Next we need to organize test execution for an upcoming release by creating test run(s) and selecting environment and matrix generation type: "New test run with environment"

The possible outcomes are:

Finally this is how the resulting test run looks like. Notice the 3 boxes icon for each test execution, listing the actual properties which should be used during testing:

"TR with environment, TE with properties"

Environment properties are read-only here b/c they have been copied to all test execution records. It usually doesn't make sense to modify your environment mid-way during test execution. If that's needed or you've made a mistake it's probably easier to create a new test run.

Happy Testing!


If you like what we're doing and how Kiwi TCMS supports various communities please help us!