How to use Kiwi TCMS plugins, Pt. 1

to import results from automated tests

Posted by Alexander Todorov on Fri 22 February 2019 under community , plugins

In release notes for v6.5 we announced several plugins which will fetch test names and execution results from your automated test suite.

Plugins can be controlled via environment variables which will affect how test results are recorded in the Kiwi TCMS database! This blog post is an introduction how that works and what you can do with it! For this purpose I will be using the plugin-demo repository, which simulates real development work.

Full documentation and list of available plugins is available in chapter Automation Frameworks Plugins!

Always create new TestRun by default

The default behavior is always to create a new TestRun if controlling variables are not overridden! Product name, version and build will receive default values if tests are running in Travis CI or Jenkins. For example Travis Build #2 for commit d455fb4 creates TR-12 and TP-10!

Product=kiwitcms/plugin-demo
Version=d455fb42fb7c2aedadfd5f66de7d131109c03350
Build=2

Next we convert the README file from Markdown to reStructuredText which triggers Travis Build #3 for commit 418b80b. This build again creates a new TestRun and new TestPlan for it. Respectively TR-14 and TP-12!

Product=kiwitcms/plugin-demo
Version=418b80b3bbb65a799f695dc59d488c76f560fa2b
Build=3

Important: we can see that version is different which will affect how artifacts are organized in Kiwi TCMS, possibly affect how you will report status to stakeholders!

Override ENV variables for more control

Let's say the QA team has decided that all test results must be reported under the same TestPlan. This means version must be the same between various builds in Travis CI! To control this we export TCMS_PRODUCT_VERSION=master in CI before executing the TAP plugin! Checkout the commit on GitHub to see how it is done!

This triggers Travis Build #4 for commit e484e59. Because this is the first time where Version == master the plugin creates TP-14 and reports the results under TR-16.

Product=kiwitcms/plugin-demo
Version=master
Build=4

Right after that I realized we can make this configuration a bit more generic because our team is planning to support multiple versions of the product and development will be done in separate branches! Travis Build #5 for commit f1f2878 still ends up with Version == master because we are still working on the master branch! That is to say, the product is in active mode of development.

Results are reported in TR-18 which is again part of TP-14.

Product=kiwitcms/plugin-demo
Version=master
Build=5

Travis Build #6 for commit df6153b adds the new functionality README badges and reports test results in TR-20 which is again part of TP-14.

More ENV overrides

While giving status reports back to stakeholders and developers the information that we have in the TestRun is Build number! This follows the numbering scheme in Travis CI (or Jenkins job number) and isn't very useful.

Let's define TCMS_BUILD to be the first 7 characters of the commit hash! When QA tells devel that something isn't working and redirects them to the TestRun they can immediately use the Build information and git checkout the offending variant of the product for investigation.

This results in Travis Build #7 for commit bf75d0a, TR-22, TP-14.

Product=kiwitcms/plugin-demo
Version=master
Build=bf75d0a

Report results in pre-existing TestRun

There are many reasons you may want to do this:

To do so I've created TR-24 beforehand and configured TCMS_RUN_ID=24 in my CI environment. TR-24 also contains TC-57: Verify we can report results from several plugins into the same TR. this was created and added via the web interface.

Note: mixing additional test cases can be done either before or after automation results are reported with the plugin!

Important: when reporting results to an existing TestRun Kiwi TCMS plugins don't care in which TestPlan this TR is! In theory it is possible to report the results for Product=kiwitcms/plugin-demo into any TP/TR pair! However we don't want to do this crazy thing and instead I've created TR-24 inside the already existing TP-14!

Note: because I don't know what is the git commit beforehand I've configured TR-24 with Build=unspecified. If I wanted I could update this with the correct value once I know the commit hash for the related changes I am testing.

Important: the plugin-demo repository uses both kiwitcms-tap-plugin and kiwitcms-junit.xml-plugin at the same time! They differ in the way test names are compiled! Both are reported in TR-24. See TC-57 text for information how to distinguish between the two.

See Travis Build #8 for commit 85ad939, TR-24, TP-14.

Product=kiwitcms/plugin-demo
Version=master
Build=unspecified

Also check-out comments in TR executions to see when and who had executed the test case.

So far there have been some tests which were failing (although Travis reports PASS) so I decided to fix them. Travis Build #9 for commit a25b384 is still configured with TCMS_RUN_ID=24. This means results will be reported in TR-24, effectively overwriting previous results.

Check-out Change Log under each individual execution and you will see several time stamps when status was updated! In other words, as long as TCMS_RUN_ID is pointing to an existing TestRun this TR will keep the results of the last test suite execution!

Continue development, restore ENV configuration

Travis Build #10 for commit c4f1ae5 removes TCMS_RUN_ID! Results are reported in TR-25, TP-14.

Product=kiwitcms/plugin-demo
Version=master
Build=c4f1ae5

Branch out for an LTS version

Our team has decided to make the first LTS release of the product. We branch out into lts-v0 branch in git and cut the first NVR. This results in Travis Build #11 for commit 9f1ef71 TR-27, TP-16.

Product=kiwitcms/plugin-demo
Version=lts-v0
Build=9f1ef71

Because this is the first time we are running tests for this product version we end up with the newly created TP-16!

Note: see how version was populated with the correct value for the git branch! This is because my CI config uses TCMS_PRODUCT_VERSION=$TRAVIS_BRANCH and no further changes were required! I made this change back in Travis Build #5 anticipating what will come in the future!

The product is now live and customers have reported critical bugs for it: URLs for the badges in README are wrong! I fix those and add more tests of course, see: Travis Build #12 for commit 2d72754, TR-29 for TP-16.

Product=kiwitcms/plugin-demo
Version=lts-v0
Build=2d72754

TR-29 contains the new TC-58!

cherry-pick between branches

In the lts-v0 branch customers have identified a serious issue. It doesn't exist on master but the test is still valid so I cherry-pick it! In git you can backport or forwardport very easily. Regardless of the direction Kiwi TCMS plugins will record the test results for you.

See Travis Build #13 for commit 31ae5b3, TR-31 for TP-14.

We can see that TC-58, which was originally implemented on the lts-v0 branch is now present!

Summary

This is a very basic example of how you can organize collection and reporting for your automation test suite results with Kiwi TCMS. The links here only refer to artifacts created by kiwitcms-tap-plugin while in the DB we keep others as well.

There are feature requests for more ENV variables which will allow you to control TestPlan creation and child/parent relationship between test plans. See https://github.com/kiwitcms/tap-plugin/issues and vote with a :+1: reaction to help us plan for these features.

Kiwi TCMS automation framework plugins are nothing more than result parsers which talk back to a database. It is up to you to decide how to organize the collection of test results and how to report on them later, when the need arises.

Future installments in this post series will take a look at workflows with feature branches and pull requests and discuss possible organization scenarios. You are welcome to share your ideas in the comments below.

Happy testing!