Articles with tag plugins

How to use Kiwi TCMS plugins, Pt. 1

Posted by Alexander Todorov on Fri 22 February 2019

In release notes for v6.5 we announced several plugins which will fetch test names and execution results from your automated test suite.

Plugins can be controlled via environment variables which will affect how test results are recorded in the Kiwi TCMS database! This blog post is an introduction how that works and what you can do with it! For this purpose I will be using the plugin-demo repository, which simulates real development work.

Full documentation and list of available plugins is available in chapter Automation Frameworks Plugins!

Always create new TestRun by default

The default behavior is always to create a new TestRun if controlling variables are not overridden! Product name, version and build will receive default values if tests are running in Travis CI or Jenkins. For example Travis Build #2 for commit d455fb4 creates TR-12 and TP-10!

Product=kiwitcms/plugin-demo
Version=d455fb42fb7c2aedadfd5f66de7d131109c03350
Build=2

Next we convert the README file from Markdown to reStructuredText which triggers Travis Build #3 for commit 418b80b. This build again creates a new TestRun and new TestPlan for it. Respectively TR-14 and TP-12!

Product=kiwitcms/plugin-demo
Version=418b80b3bbb65a799f695dc59d488c76f560fa2b
Build=3

Important: we can see that version is different which will affect how artifacts are organized in Kiwi TCMS, possibly affect how you will report status to stakeholders!

Override ENV variables for more control

Let's say the QA team has decided that all test results must be reported under the same TestPlan. This means version must be the same between various builds in Travis CI! To control this we export TCMS_PRODUCT_VERSION=master in CI before executing the TAP plugin! Checkout the commit on GitHub to see how it is done!

This triggers Travis Build #4 for commit e484e59. Because this is the first time where Version == master the plugin creates TP-14 and reports the results under TR-16.

Product=kiwitcms/plugin-demo
Version=master
Build=4

Right after that I realized we can make this configuration a bit more generic because our team is planning to support multiple versions of the product and development will be done in separate branches! Travis Build #5 for commit f1f2878 still ends up with Version == master because we are still working on the master branch! That is to say, the product is in active mode of development.

Results are reported in TR-18 which is again part of TP-14.

Product=kiwitcms/plugin-demo
Version=master
Build=5

Travis Build #6 for commit df6153b adds the new functionality README badges and reports test results in TR-20 which is again part of TP-14.

More ENV overrides

While giving status reports back to stakeholders and developers the information that we have in the TestRun is Build number! This follows the numbering scheme in Travis CI (or Jenkins job number) and isn't very useful.

Let's define TCMS_BUILD to be the first 7 characters of the commit hash! When QA tells devel that something isn't working and redirects them to the TestRun they can immediately use the Build information and git checkout the offending variant of the product for investigation.

This results in Travis Build #7 for commit bf75d0a, TR-22, TP-14.

Product=kiwitcms/plugin-demo
Version=master
Build=bf75d0a

Report results in pre-existing TestRun

There are many reasons you may want to do this:

  • include both manual and automation tests for the same build;
  • mix results from multiple test suites for the same build, e.g. unit, functional, performance
  • mix results from multiple but similar platforms in the same build, e.g. cross-platform application for iOS and Android

To do so I've created TR-24 beforehand and configured TCMS_RUN_ID=24 in my CI environment. TR-24 also contains TC-57: Verify we can report results from several plugins into the same TR. this was created and added via the web interface.

Note: mixing additional test cases can be done either before or after automation results are reported with the plugin!

Important: when reporting results to an existing TestRun Kiwi TCMS plugins don't care in which TestPlan this TR is! In theory it is possible to report the results for Product=kiwitcms/plugin-demo into any TP/TR pair! However we don't want to do this crazy thing and instead I've created TR-24 inside the already existing TP-14!

Note: because I don't know what is the git commit beforehand I've configured TR-24 with Build=unspecified. If I wanted I could update this with the correct value once I know the commit hash for the related changes I am testing.

Important: the plugin-demo repository uses both kiwitcms-tap-plugin and kiwitcms-junit.xml-plugin at the same time! They differ in the way test names are compiled! Both are reported in TR-24. See TC-57 text for information how to distinguish between the two.

See Travis Build #8 for commit 85ad939, TR-24, TP-14.

Product=kiwitcms/plugin-demo
Version=master
Build=unspecified

Also check-out comments in TR executions to see when and who had executed the test case.

So far there have been some tests which were failing (although Travis reports PASS) so I decided to fix them. Travis Build #9 for commit a25b384 is still configured with TCMS_RUN_ID=24. This means results will be reported in TR-24, effectively overwriting previous results.

Check-out Change Log under each individual execution and you will see several time stamps when status was updated! In other words, as long as TCMS_RUN_ID is pointing to an existing TestRun this TR will keep the results of the last test suite execution!

Continue development, restore ENV configuration

Travis Build #10 for commit c4f1ae5 removes TCMS_RUN_ID! Results are reported in TR-25, TP-14.

Product=kiwitcms/plugin-demo
Version=master
Build=c4f1ae5

Branch out for an LTS version

Our team has decided to make the first LTS release of the product. We branch out into lts-v0 branch in git and cut the first NVR. This results in Travis Build #11 for commit 9f1ef71 TR-27, TP-16.

Product=kiwitcms/plugin-demo
Version=lts-v0
Build=9f1ef71

Because this is the first time we are running tests for this product version we end up with the newly created TP-16!

Note: see how version was populated with the correct value for the git branch! This is because my CI config uses TCMS_PRODUCT_VERSION=$TRAVIS_BRANCH and no further changes were required! I made this change back in Travis Build #5 anticipating what will come in the future!

The product is now live and customers have reported critical bugs for it: URLs for the badges in README are wrong! I fix those and add more tests of course, see: Travis Build #12 for commit 2d72754, TR-29 for TP-16.

Product=kiwitcms/plugin-demo
Version=lts-v0
Build=2d72754

TR-29 contains the new TC-58!

cherry-pick between branches

In the lts-v0 branch customers have identified a serious issue. It doesn't exist on master but the test is still valid so I cherry-pick it! In git you can backport or forwardport very easily. Regardless of the direction Kiwi TCMS plugins will record the test results for you.

See Travis Build #13 for commit 31ae5b3, TR-31 for TP-14.

We can see that TC-58, which was originally implemented on the lts-v0 branch is now present!

Summary

This is a very basic example of how you can organize collection and reporting for your automation test suite results with Kiwi TCMS. The links here only refer to artifacts created by kiwitcms-tap-plugin while in the DB we keep others as well.

There are feature requests for more ENV variables which will allow you to control TestPlan creation and child/parent relationship between test plans. See https://github.com/kiwitcms/tap-plugin/issues and vote with a :+1: reaction to help us plan for these features.

Kiwi TCMS automation framework plugins are nothing more than result parsers which talk back to a database. It is up to you to decide how to organize the collection of test results and how to report on them later, when the need arises.

Future installments in this post series will take a look at workflows with feature branches and pull requests and discuss possible organization scenarios. You are welcome to share your ideas in the comments below.

Happy testing!

Test runner plugin specification

Posted by Alexander Todorov on Mon 05 November 2018

Happy Monday testers! Kiwi TCMS needs your help! We are looking for developers who wish to create plugins for popular test runners that will record test results in Kiwi TCMS! Initially we are looking for plugins for Python's unittest, Django and JUnit!

What is a test runner?

When working with automated testing you have several components:

  • A test module (or test script), e.g. test_models.py which contains tests for your software;
  • A test framework, e.g. Python nose, which provides the structure for test classes and methods and assert methods to use;
  • A test runner, which knows how to discover your test scripts, load them, execute the testing scenarios inside of them and report the results.

Very often all of the components above live together inside the testing framework but don't need to. For example the standard unittest module in Python provides a test runner but there are also nose and py.test and Django provides its own test runner that knows how to work with the database.

Workflow organization

Once you agree to writing a plugin we are going to create a separate GitHub repository where you will be granted write privileges making you an independent contributor to the Kiwi TCMS project!

Design and architecture of the plugin is up to you, following the practices established by the testing framework in question. You will also have to create a test suite for your plugin based on the specification below.

You are expected to use public.tenant.kiwitcms.org while working on the plugin and afterwards. This is known as eating your own dog food!

For Python we provide the tcms-api module which already takes care of the communications layer. For other languages you will have to create this layer or depend on other open source libraries that provide a XML-RPC or JSON-RPC client!

You can use this gist for inspiration!

Behavior Specification

Please use the comments section to discuss unclear behavior and missing scenarios!

    SUMMARY: Will use configuration file if it exists
    GIVEN: the file ~/.tcms.conf exists
    WHEN: plugin initializes
    THEN: the plugin will log an info message, read the file and
    THEN: configure TCMS_API_URL, TCMS_USERNAME, TCMS_PASSWORD
          variables with the respective values


    SUMMARY: Will use ENVIRONMENT if configuration file doesn't exist
    GIVEN: the file ~/.tcms.conf does not exists
    WHEN: plugin initializes
    THEN: the plugin will read configuration from environment and configure
          the following variables/class members:
          TCMS_API_URL, TCMS_USERNAME and TCMS_PASSWORD


    SUMMARY: Will exit if TCMS_API_URL not configured
    GIVEN: TCMS_API_URL variable is empty
    WHEN: plugin initializes
    THEN: log a warning message and exit
    AND: depending on the test runner framework set exist status 1


    SUMMARY: Will exit if TCMS_USERNAME not configured
    GIVEN: TCMS_USERNAME is empty
    WHEN: plugin initializes
    THEN: log a warning message and exit
    AND: depending on the test runner framework set exist status 1


    SUMMARY: Will exit if TCMS_PASSWORD not configured
    GIVEN: TCMS_PASSWORD is empty
    WHEN: plugin initializes
    THEN: log a warning message and exit
    AND: depending on the test runner framework set exist status 1


    SUMMARY: Will re-use existing TestPlan if configured
    GIVEN: TCMS_RUN_ID environment variable is not empty
    WHEN: plugin initializes
    THEN:  this will be the Current_TestRun record to which the plugin is
           going to add test execution results
    AND: Current_TestPlan document in which the plugin will
           search for test cases becomes Current_TestRun.plan


    SUMMARY: Will create new TestPlan & TestRun if TCMS_RUN_ID not configured
    GIVEN: TCMS_RUN_ID environment variable is empty
    THEN: plugin will create a new TestPlan in Kiwi TCMS with attributes:
        name='Automated test plan for %(product)'
        product='%(product)'
        product_version='%(version)'
        type='Unit'
    WHERE: %(product) is a placeholder for TCMS_PRODUCT==TRAVIS_REPO_SLUG==JOB_NAME
           %(version) is a placeholder for TCMS_PRODUCT_VERSION==TRAVIS_COMMIT==TRAVIS_PULL_REQUEST_SHA==GIT_COMMIT
    THEN: plugin will crate a new TestRun in Kiwi TCMS with attributes:
        summary='Automated test run ....'
        plan=Current TestPlan
        build='%(build)'
        manager=TCMS_USERNAME
    WHERE: %(build) is a placeholder for TCMS_BUILD==TRAVIS_BUILD_NUMBER==BUILD_NUMBER
    Environment variables are specified in:
    https://docs.travis-ci.com/user/environment-variables#default-environment-variables
    https://wiki.jenkins.io/display/JENKINS/Building+a+software+project#Buildingasoftwareproject-belowJenkinsSetEnvironmentVariables

    SUMMARY: Will not create duplicate Product, Version & Build if they already exist
    GIVEN: TCMS_RUN_ID is not configured
    AND: %(product) exists
    AND: %(version) exists
    AND: %(build) exists
    WHEN: plugin tries to auto-create TestPlan and TestRun
    THEN: plugin will re-use %(product), %(version) and %(build) from the database
    AND: not try to auto-create them


    SUMMARY: Will auto-create Product, Version & Build if they don't exist
    GIVEN: TCMS_RUN_ID is not configured
    AND: %(product) doesn't exist
    AND: %(version) doesn't exist
    AND: %(build) doesn't exist
    WHEN: plugin tries to auto-create TestPlan and TestRun
    THEN: %(product), %(version) and %(build) be created automatically


    SUMMARY: Unit test names are added to TestPlan
    GIVEN: we have good plugin configuration
    WHEN: plugin loops over unit tests emitted by the test runner
    THEN: plugin will check Current_TestPlan for a TestCase with the same name
    AND: if test case doesn't exist in Current_TestPlan
    THEN: it will be added to Current_TestPlan
    hint: it is probably best to process all unit test results at the end!


    SUMMARY: Unit test names are added to TestRun
    GIVEN: we have good plugin configuration
    WHEN: plugin loops over unit tests emitted by the test runner
    THEN: plugin will check Current_TestRun for a TestCaseRun object which matches
          the current unit test name
    hint: (or Current_TestCase object from previous scenario, depending on implementation)
    AND: if such TestCaseRun doesn't exist in Current_TestRun
    THEN: it will be added to Current_TestRun
    hint: it is probably best to process all unit test results at the end!


    SUMMARY: Current_TestRun is updated with unit test results
    GIVEN: we have good plugin configuration
    WHEN: plugin loops over unit tests emitted by the test runner
    THEN: plugin will check Current_TestRun for a TestCaseRun object which matches
          the current unit test name
    hint: (or Current_TestCase object from previous scenario, depending on implementation)
    AND: if TestCaseRun object exists in Current_TestRun
    THEN: its status will be updated with the execution result coming from the test runner
    hint: it is probably best to process all unit test results at the end!

Happy testing!

Page 1 / 1