How to override bug details in Kiwi TCMS

Starting with version 7.0 Kiwi TCMS pages displaying URLs to bugs also contain an info icon which shows additional information as tooltip. These are designed to provide more contextual information about the bug. By default the tooltip shows the OpenGraph metadata for that URL. This article will explain how to override this in 2 different ways.

bug details shown

Option #1: using the caching layer

Additional bug information is cached on the application layer. The cache key is the bug URL! By default Kiwi TCMS uses local-memory caching which isn't accessible for external processes but can be reconfigured very easily. This example changes the CACHES setting to use a directory on the file system like so

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.filebased.FileBasedCache',
        'LOCATION': '/tmp/kiwi-cache',
        'TIMEOUT': 3600,
    }
}

Then you need to poll your 3rd party bug tracker (and/or other systems) and update the cache for each URL

from django.core.cache import cache
from tcms.core.contrib.linkreference.models import LinkReference

for reference in LinkReference.objects.filter(is_defect=True):
    # possibly filter objects coming only from your own bug tracker
    # in case there are multiple trackers in use

    # custom methods to grab more information. Must return strings
    title = fetch_title_from_bug_tracker(reference.url)
    description = fetch_description_from_bug_tracker(reference.url)

    # store the information in Kiwi TCMS cache
    cache.set(reference, {'title': title, 'description': description})

Then execute the Python script above regularly. For example use the following as your cron script

#!/bin/bash
export VIRTUAL_ENV=/venv
export PATH=/venv/bin:${PATH}
cat /path/to/cache_updater.py | /Kiwi/manage.py shell

bug details from customized cache

IMPORTANT

Option #2: extend bug tracker integration

Let's say you are already running a customized Docker image of Kiwi TCMS. Then you may opt-in to extend the existing bug tracker integration code which provides the information shown in the tooltip. In this example I've extended the KiwiTCMS bug tracker implementation but you can even provide your own from scratch

class ExtendedBugTracker(KiwiTCMS):
    def details(self, url):
        result = super().details(url)

        result['title'] = 'EXTENDED: ' + result['title']
        result['description'] += '<h1>IMPORTANT</h1>'

        return result

Then import the new ExtendedBugTracker class inside tcms/issuetracker/types.py like so

index 9ad90ac..2c76621 100644
--- a/tcms/issuetracker/types.py
+++ b/tcms/issuetracker/types.py
@@ -17,6 +17,9 @@ from django.conf import settings

 from tcms.issuetracker.base import IssueTrackerType
 from tcms.issuetracker.kiwitcms import KiwiTCMS  # noqa
+from tcms.issuetracker.kiwitcms import ExtendedBugTracker

and change the bug tracker type, via https://tcms.example.com/admin/testcases/bugsystem/, to ExtendedBugTracker.

bug details extended internally

IMPORTANT

Happy testing!

How to add API logging for Kiwi TCMS

In this blog post I will show more ways to customize Kiwi TCMS by adding logging capabilities to the API backend. Indeed this is a feature that our team deemed not required for upstream and was removed in PR #436.

Start by creating the following directory structure:

    api_logging/
        __init__.py
        handlers.py
        models.py

This is a small Django application that will log every call to the API backend. Each file looks like this:

    # models.py contains DB schema for your new table
    from django.db import models
    from django.conf import settings

    class ApiCallLog(models.Model):
        executed_at = models.DateTimeField(auto_now_add=True)
        user = models.ForeignKey(settings.AUTH_USER_MODEL, null=True, blank=True,
                                 on_delete=models.CASCADE)
        method = models.CharField(max_length=255)
        args = models.TextField(blank=True)

        def __str__(self):
            return "%s: %s" % (self.user, self.method)

Then

    # handlers.py overrides the RPC handlers coming from django-modernrpc
    from modernrpc import handlers

    from django.conf import settings
    from django.contrib.auth import get_user_model

    from .models import ApiCallLog

    def log_call(request, method_name, args):
        """ Log an RPC call to the database or stdout in DEBUG mode. """
        request_user = request.user
        if not request_user.is_authenticated:
            # create an anonymous user object for logging purposes
            request_user, _ = get_user_model().objects.get_or_create(
                username='Anonymous',
                is_active=False)

        if method_name is None:
            method_name = '--- method_name missing ---'

        if settings.DEBUG:
            print('API call:: user: {0}, method: {1}, args: {2}'.format(
                request_user,
                method_name,
                args))
        else:
            ApiCallLog.objects.create(
                user=request_user,
                method=method_name,
                args=str(args))

    class XMLRPCHandler(handlers.XMLRPCHandler):
        def process_request(self):
            encoding = self.request.encoding or 'utf-8'
            data = self.request.body.decode(encoding)
            params, method_name = self.loads(data)

            log_call(self.request, method_name, params)
            return super().process_request()

    class JSONRPCHandler(handlers.JSONRPCHandler):
        def process_single_request(self, payload):
            method_name = payload.get('method', None)
            params = payload.get('params')

            log_call(self.request, method_name, params)
            return super().process_single_request(payload)

NOTE: You will have to execute ./manage.py makemigrations api_logging to create the initial migration for Django. This could be easier if you place the above directory into existing Django application or craft the migration file by hand!

The last thing you want to do is create a local_settings.py file which will override Kiwi TCMS defaults:

    # local_settings.py
    from django.conf import settings

    settings.INSTALLED_APPS += [
        'api_logging',
    ]

    MODERNRPC_HANDLERS = ['api_logging.handlers.XMLRPCHandler',
                          'api_logging.handlers.JSONRPCHandler']

Then place everything in Dockerfile like so:

    FROM kiwitcms/kiwi

    COPY ./api_logging/ /venv/lib64/python3.6/site-packages/api_logging/
    COPY local_settings.py /venv/lib64/python3.6/site-packages/tcms/settings/

Kiwi TCMS will import your local_settings.py and enable the logging application. Now build your customized Docker image and use it for deployment!

Happy testing!

How to override templates for Kiwi TCMS

This is the first publication in our customization series. It will show you how to override any template used by Kiwi TCMS. As an example we will override the email template that is used when registering new account. By default the email text looks like this:

    Welcome {{ user }},
    thank you for signing up for an {{ site_domain }} account!

    To activate your account, click this link:
    {{ confirm_url }}

https://public.tenant.kiwitcms.org runs a custom Docker image based on kiwitcms/kiwi. For this image the confirmation email looks like this

    Welcome {{ user }},
    thank you for signing up for our Kiwi TCMS demo!

    To activate your account, click this link:
    {{ confirm_url }}

    GDPR no longer allows us to automatically subscribe you to
    our newsletter. If you wish to keep in touch and receive emails
    with news and updates around Kiwi TCMS please subscribe at:
    https://kiwitcms.us17.list-manage.com/subscribe/post?u=9b57a21155a3b7c655ae8f922&id=c970a37581

    --
    Happy testing!
    The Kiwi TCMS team
    http://kiwitcms.org

The file that we want to override is tcms/templates/email/confirm_registration.txt.

Create a local directory (git repository) which will hold customization configuration and create a file named templates.d/email/confirm_registration.txt with your text!

Next you want to make this file available inside your docker image so your Dockerfile should look like this:

    FROM kiwitcms/kiwi

    COPY ./templates.d/ /venv/lib64/python3.6/site-packages/tcms/overridden_templates/
    COPY local_settings.py /venv/lib64/python3.6/site-packages/tcms/settings/

where local_settings.py contains

    import os
    from django.conf import settings

    settings.TEMPLATES[0]['DIRS'].insert(0, os.path.join(settings.TCMS_ROOT_PATH, 'overridden_templates'))

The following code states instruct Django to look into overridden_templates first and use any templates it finds there; also make my files available in that specific location inside the docker image.

This approach can be used for all templates that you wish to override. Take into account that file names must match (Django searches templates by directory path). Now build your customized Docker image and use that for deployment!

Happy testing!

How to backup Docker volumes for Kiwi TCMS

When you start Kiwi TCMS by running docker-compose up (see here) it will automatically create 2 volumes: kiwi_db_data and kiwi_uploads. This blog post will outline how to backup these docker volumes.

Backing up the database

Kiwi TCMS is a Django application and the manage.py command provides an easy way to dump and load the database contents. To export all contents on your docker host execute:

docker exec -it kiwi_web /Kiwi/manage.py dumpdata --all --indent 2 > database.json

This will create a file named database.json in the current directory, outside of the running container!

You can restore the database contents by using the following commands:

# delete data from all tables
docker exec -it kiwi_web /bin/bash -c '/Kiwi/manage.py sqlflush | /Kiwi/manage.py dbshell'
# then reload the existing data
cat database.json | docker exec -i kiwi_web /Kiwi/manage.py loaddata --format json -

NOTE: depending on your scenario you may want to remove the existing volume (docker-compose down && docker volume rm kiwi_db_data) and re-create the DB schema (/Kiwi/manage.py migrate) before restoring the contents!

WARNING: the above steps are applicable to Kiwi TCMS 5.1 or above. On earlier versions manage.py will fail due to various issues.

Backing up multi-tenant database

The kiwitcms-tenant add-on depends on the PostgreSQL database. It will create multiple DB schemas, one per tenant. To backup all tenants use the following command:

docker exec -i kiwi_db /bin/bash -c 'pg_dump --dbname=kiwi -F c' > backup.bak

This will create a file in the PostgreSQL custom database dump format which contains all data and schema definitions. That is a binary file which can be read with the pg_restore command.

To [drop and] restore the entire multi-tenant database:

docker volume rm kiwi_db_data
cat backup.bak | docker exec -i kiwi_db /bin/bash -c 'pg_restore --dbname=kiwi -v'

To [drop and] restore an individual tenant:

docker exec -it kiwi_web /Kiwi/manage.py dbshell

kiwi=> DROP SCHEMA $tenant_name CASCADE;
....
DROP SCHEMA
kiwi=> CREATE SCHEMA $tenant_name;
CREATE SCHEMA
kiwi=>Ctrl+D

cat backup.bak | docker exec -i kiwi_db /bin/bash -c 'pg_restore --dbname=kiwi -v --schema $tenant_name'

WARNING: sqlflush | dbshell will not work when you have multiple DB schemas so you must use the PostgreSQL database shell to manipulate the contents of the database!

Backing up file uploads

Uploaded files can easily be backed up with:

docker exec -it kiwi_web /bin/tar -cP /Kiwi/uploads > uploads.tar

and then restored:

cat uploads.tar | docker exec -i kiwi_web /bin/tar -x

You may also try the rsync command but be aware that it is not installed by default!

The same approach may be used to backup /var/lib/mysql/ from the kiwi_db container.

Backing up multi-tenant uploads

By default multi-tenant file uploads are stored under /Kiwi/uploads/tenant/$tenant_name. You can archive all contents with the same procedure above. If you wish to restore files per tenant you will have to upload the $tenant_name directory into the docker volume.

Alternatives

By default both docker volumes created for Kiwi TCMS use the local driver and are available under /var/lib/docker/volumes/<volume_name> on the host running your containers. You can try backing them up from there as well.

Another alternative is to use the docker-lvm-plugin and create these volumes as LVM2 block devices. Then use lvcreate -s command to create a snapshot volume. For more information see chapter 2.3.5. Snapshot Volumes from the LVM Administrator Guide for Red Hat Enterprise Linux 7.

Happy testing!

Project news only, GDPR compliant. Subscribe here!