I am happy to announce that our team is steadily growing! As we work through our roadmap, status update here, and on-board new team members I start to feel the need for a bit more structure and organization behind the scenes. I also wish for consistent contributions to the project (commit early, commit often) so I can better estimate the resources that we have!

I am also actively discussing Kiwi TCMS with lots of people at various conferences and generate many ideas for the future. The latest SEETEST in Belgrade was particularly fruitful. Some of these ideas are pulling into different directions and I need help to keep them under control!

Development-wise sometimes I lose track of what's going on and who's doing what between working on Kiwi TCMS, preparing for conferences and venues to promote the project, doing code review of other team members, trying not to forget to check-in on progress (especially by interns), recruiting fresh blood and thinking about the overall future of the project. Our user base is growing and there are days where I feel like everything is happening at once or that something needs to be implemented ASAP (which is usually true anyway)!

Meet Rayna Stankova in the role of our team coach! Reny is a director for Women Who Code Sofia, senior QA engineer at VMware, mentor with CoderDojo Bulgaria and a long-time friend of mine. Although she is an experienced QA in her own right she will be contributing to the people side of Kiwi TCMS and less so technically!

Her working areas will be planning and organization:

• help us (re)define the project vision and goals
• work with us on roadmaps and action plans so we can meet the project goals faster
• help us (self) organize so that we are more efficient, including checking progress and blockers (aka enforcer) and meet the aforementioned consistency point
• serve as our professional coach, motivator and somebody who will take care of team health (yes I really suck at motivating others)

and generally serving as another very experienced member of the team!

We did a quick brainstorming yesterday and started to produce results (#smileyface)! We do have a team docs space to share information (non-public for now, will open it gradually as we grow) and came up with the idea to use Kiwi TCMS as a check-list for our on-boarding/internship process!

I don't know how it will play out but I do expect from the team to self-improve, be inspired, become more focused and more productive! All of this also applies to myself, even more so!

Existing team members progress

Last year we started with 2 existing team members (Tony and myself) and 3 new interns (Ivo, Kaloyan and Tseko) who built this website!

Tony is the #4 contributor to Kiwi TCMS in terms of number of commits and is on track to surpass one of the original authors (before Kiwi TCMS was forked)! He's been working mostly on internal refactoring and resolving the thousands of pylint errors that we had (down to around 500 I think). This summer Tony and I visited the OSCAL conference in Tirana and hosted an info booth for the project.

Ivo is the #5 contributor in terms of numbers of commits. He did learn very quickly and is working on getting rid of the remaining pylint errors. His ability to adapt and learn is quite impressive actually. Last month he co-hosted a git workshop at HackConf, a 1000+ people IT event in Sofia.

Kaloyan did most of the work on our website initially (IIRC). Now he is studying in the Netherlands and not active on the project. We are working to reboot his on-boarding and I'm hoping he will find the time to contribute to Kiwi TCMS regularly.

From the starting team only Tseko decided to move on to other ventures after he contributed to the website.

Internship program

At Kiwi TCMS we have a set of training programs that teach all the necessary technical skills before we let anyone actively work on the project, let alone become a team member.

Our new interns are Denitsa Uzunova and Desislava Koleva. Both of them are coming from Vratsa Software Community and were mentors at the recently held CodeWeek hackathon in their home city! I wish them fast learning and good luck!

Happy testing!

In this blog post I will show more ways to customize Kiwi TCMS by adding logging capabilities to the API backend. Indeed this is a feature that our team deemed not required for upstream and was removed in PR #436.

Start by creating the following directory structure:

    api_logging/
        __init__.py
        handlers.py
        models.py

This is a small Django application that will log every call to the API backend. Each file looks like this:

    # models.py contains DB schema for your new table
    from django.db import models
    from django.conf import settings

    class ApiCallLog(models.Model):
        executed_at = models.DateTimeField(auto_now_add=True)
        user = models.ForeignKey(settings.AUTH_USER_MODEL, null=True, blank=True,
                                 on_delete=models.CASCADE)
        method = models.CharField(max_length=255)
        args = models.TextField(blank=True)

        def __str__(self):
            return "%s: %s" % (self.user, self.method)

Then

    # handlers.py overrides the RPC handlers coming from django-modernrpc
    from modernrpc import handlers

    from django.conf import settings
    from django.contrib.auth import get_user_model

    from .models import ApiCallLog

    def log_call(request, method_name, args):
        """ Log an RPC call to the database or stdout in DEBUG mode. """
        request_user = request.user
        if not request_user.is_authenticated:
            # create an anonymous user object for logging purposes
            request_user, _ = get_user_model().objects.get_or_create(
                username='Anonymous',
                is_active=False)

        if method_name is None:
            method_name = '--- method_name missing ---'

        if settings.DEBUG:
            print('API call:: user: {0}, method: {1}, args: {2}'.format(
                request_user,
                method_name,
                args))
        else:
            ApiCallLog.objects.create(
                user=request_user,
                method=method_name,
                args=str(args))

    class XMLRPCHandler(handlers.XMLRPCHandler):
        def process_request(self):
            encoding = self.request.encoding or 'utf-8'
            data = self.request.body.decode(encoding)
            params, method_name = self.loads(data)

            log_call(self.request, method_name, params)
            return super().process_request()

    class JSONRPCHandler(handlers.JSONRPCHandler):
        def process_single_request(self, payload):
            method_name = payload.get('method', None)
            params = payload.get('params')

            log_call(self.request, method_name, params)
            return super().process_single_request(payload)

NOTE: You will have to execute ./manage.py makemigrations api_logging to create the initial migration for Django. This could be easier if you place the above directory into existing Django application or craft the migration file by hand!

The last thing you want to do is create a local_settings.py file which will override Kiwi TCMS defaults:

    # local_settings.py
    from django.conf import settings

    settings.INSTALLED_APPS += [
        'api_logging',
    ]

    MODERNRPC_HANDLERS = ['api_logging.handlers.XMLRPCHandler',
                          'api_logging.handlers.JSONRPCHandler']

Then place everything in Dockerfile like so:

    FROM kiwitcms/kiwi

    COPY ./api_logging/ /venv/lib64/python3.6/site-packages/api_logging/
    COPY local_settings.py /venv/lib64/python3.6/site-packages/tcms/settings/

Kiwi TCMS will import your local_settings.py and enable the logging application. Now build your customized Docker image and use it for deployment!

Happy testing!

This is the first publication in our customization series. It will show you how to override any template used by Kiwi TCMS. As an example we will override the email template that is used when registering new account. By default the email text looks like this:

    Welcome {{ user }},
    thank you for signing up for an {{ site_domain }} account!

    To activate your account, click this link:
    {{ confirm_url }}

https://demo.kiwitcms.org runs a custom Docker image based on kiwitcms/kiwi. For this image the confirmation email looks like this

    Welcome {{ user }},
    thank you for signing up for our Kiwi TCMS demo!

    To activate your account, click this link:
    {{ confirm_url }}

    GDPR no longer allows us to automatically subscribe you to
    our newsletter. If you wish to keep in touch and receive emails
    with news and updates around Kiwi TCMS please subscribe at:
    https://kiwitcms.us17.list-manage.com/subscribe/post?u=9b57a21155a3b7c655ae8f922&id=c970a37581

    --
    Happy testing!
    The Kiwi TCMS team
    http://kiwitcms.org

The file that we want to override is tcms/templates/email/confirm_registration.txt.

Create a local directory (git repository) which will hold customization configuration and create a file named templates.d/email/confirm_registration.txt with your text!

Next you want to make this file available inside your docker image so your Dockerfile should look like this:

    FROM kiwitcms/kiwi

    COPY ./templates.d/ /venv/lib64/python3.6/site-packages/tcms/overridden_templates/
    COPY local_settings.py /venv/lib64/python3.6/site-packages/tcms/settings/

where local_settings.py contains

    import os
    from django.conf import settings

    settings.TEMPLATES[0]['DIRS'].insert(0, os.path.join(settings.TCMS_ROOT_PATH, 'overridden_templates'))

The following code states instruct Django to look into overridden_templates first and use any templates it finds there; also make my files available in that specific location inside the docker image.

This approach can be used for all templates that you wish to override. Take into account that file names must match (Django searches templates by directory path). Now build your customized Docker image and use that for deployment!

Happy testing!

When you start Kiwi TCMS by running docker-compose up (see here) it will automatically create 2 volumes: kiwi_db_data and kiwi_uploads. This blog post will outline how to backup these docker volumes.

Backing up the database

Kiwi TCMS is a Django application and the manage.py command provides an easy way to dump and load the database contents. To export all contents on your docker host execute:

docker exec -it kiwi_web /Kiwi/manage.py dumpdata --all --indent 2 > database.json

This will create a file named database.json in the current directory, outside of the running container!

You can restore the database contents by using the following commands:

# delete data from all tables
docker exec -it kiwi_web /bin/bash -c '/Kiwi/manage.py sqlflush | /Kiwi/manage.py dbshell'
# then reload the existing data
cat database.json | docker exec -i kiwi_web /Kiwi/manage.py loaddata --format json -

NOTE: depending on your scenario you may want to remove the existing volume (docker-compose down && docker volume rm kiwi_db_data) and re-create the DB schema (/Kiwi/manage.py migrate) before restoring the contents!

WARNING: the above steps are applicable to Kiwi TCMS 5.1 or above. On earlier versions manage.py will fail due to various issues.

Backing up file uploads

Uploaded files can easily be backed up with:

docker exec -it kiwi_web /bin/tar -cP /var/kiwi/uploads > uploads.tar

and then restored:

cat uploads.tar | docker exec -i kiwi_web /bin/tar -x

You may also try the rsync command but be aware that it is not installed by default!

The same approach may be used to backup /var/lib/mysql/ from the kiwi_db container.

Alternatives

By default both docker volumes created for Kiwi TCMS use the local driver and are available under /var/lib/docker/volumes/<volume_name> on the host running your containers. You can try backing them up from there as well.

Another alternative is to use the docker-lvm-plugin and create these volumes as LVM2 block devices. Then use lvcreate -s command to create a snapshot volume. For more information see chapter 2.3.5. Snapshot Volumes from the LVM Administrator Guide for Red Hat Enterprise Linux 7.

Happy testing!

Hello everyone, in this article I will outline the progress that the Kiwi TCMS team has made towards achieving the goals on our roadmap.

Make code easier to maintain

Status: moderate progress

Initially CodeClimate reported a "D" rating with a 1 year estimated effort. Now it is still on "D" rating with a 7 months estimated effort to bring the project back in shape. Code smells have dropped from 600+ to 418, duplications have been reduced from 600+ to 359! At the same time technical debt ratio has been decreased from 32,5% to 21,6% and little over 10000 lines of code have been removed from the source code. Checkout the stats for more info!

Use pylint and pylint-django

Status: good progress

Both pylint and pylint-django have been integrated into our CI workflow. There are even a few custom built plugins that we use. The number of issues reported is down to around 900 from 4000+ initially. The cleanup has been lead by Anton Sankov with help from Ivaylo Ivanov and myself.

Render HTML, return JSON

Status: no progress

Several views were probably modified to return pure JSON in the meantime but we've not done any targeted work to resolve this issue.

Submit forms, post JSON, GET clean URLs

Status: no progress

Same as above, not much has been done in this area.

API layer

Status: complete

After Kiwi TCMS v4.0 the server side API has been reorganized and updated to follow the model/method names used internally.

After the recent version 5.0 the client side API library has been stripped to its most basic form so that you can work directly with the responses from the server.

There is no more duplication and ambiguity in names because there isn't a lot of code left!

Documentation

Status: moderate progress, dropped

All RPC methods have been documented! The rest of the internals will be documented as we go along.

No vendored JavaScript libraries

Status: moderate progress

Several JavaScript libraries have been removed but we still carry around jQuery and Handlebars.js. No work has been done to convert Kiwi TCMS to use the jQuery version provided with Django.

Less HTML templates with better organization

Status: minimal progress

There are still over 100 HTML templates in Kiwi TCMS. Some of the HTML templates have been merged together, some email templates have been refactored and marked as translatable but the majority of them have not been updated for a long time.

Modern interface with Patternfly

Status: no progress

JavaScript updates and front-end testing

Status: small progress

A number of JavaScript functions have been refactored and removed during the past few releases but there are still thousands of lines of code left to deal with.

Community efforts

Status: moderate progress

We are seeing a steady stream of new users registered on https://demo.kiwitcms.org and there are several active contributors (issues, translations).

Kiwi TCMS was represented at OSCAL Tirana, DjangoCon Heidelberg and PyCon Prague! We're planning to attend HackConf and OpenFest in Sofia by the end of the year.

Happy testing!