Distributed Test Reporting using ELK stack!

Last updated: December 2020

Image for post
Image for post

The code used throughout this article is available on the open-source project “Test Automation Bootstrap”, hosted on GitHub.

  1. The purpose of test reporting
  2. The challenge
  3. The solution
  4. The implementation

Motivation

So, I answered an open question posted on the Ministry of Testing’s LinkedIn page. It got a couple of likes and people commenting — feels good mate! — but that got me thinking 🤔

Image for post
Image for post
Image for post
Image for post

Based on the answers I read from my fellow test automation engineer colleagues, it’s noticeable that the majority seems truly divided regarding test reporting and its purpose. The majority hinted that test reporting would be used to serve some sort of requirement or request — usually, managers request release or regression reports.

Test Reporting

However, and in my humble opinion, test reports should serve the whole team and not a single individual nor a single purpose.

Test reporting should provide customizable and tailored feedback of the current state of the art of your product — preferably in real-time — so that anyone on your team (managers included) can take action and act accordingly, even though if that simply means making the message get across.

Value comes in many shapes and forms. And, for less technical team members would be nicer to have some charts indicating incidents, the names and amount of successful tests, and maybe in which browser are tests failing? — if there are any — , and on the other hand for the more technical members, it would be optimal if they could have the ability to debug the issue and to check up on the logs to understand and solve the issue.

Thus, how can we provide value for both technical and non-technical individuals?

Visualization!

Well, if you’re a designer you probably knew the answer 😅

Yep, this is a rather visualization issue than anything else. If you provide the right view for the right individual, the value is there, and it’s there for the whole team. Thus, making test reporting a common and valuable asset, again, for the whole team.

Distributed Test Reporting using ELK stack!

ELK stack is an acronym for three open-source projects that come together to provide search and analysis mechanisms (Elasticsearch), data ingestion and pipeline processing (Logstash), and data visualization (Kibana).

Image for post
Image for post

So, as the code presented is fully open-source, we’re going into more detail on the single pieces that we need to understand to have a solid ground on the proposed solution. That way you’ll have all you need to further scale, tailor, and customize it according to your needs.

Image for post
Image for post
The proposed solution from the base source code

a. Maven logback and logstash dependencies

There are two dependencies that our project needs to be able to communicate with our stack’s pipeline entrypoint (logstash). Thus, we need to add them to our pom file.

b. TestNG listener

A common way is to log the test results since we’re going to base our reports on those results. Therefore, we implement a listener that will be responsible for letting us know the current result of every single test.

⚠️ Note that the logs have a very specific structure. For example:

logger().error("Pipeline: {} Browser: {} Status: {} Test: {} Suite: {} Trace: {}", ...)

You can have your own structure, but whatever it is, it will be needed for further actions in step f). Thus, make it a thorough choice.

c. Logback configuration file

Now, let’s configure logback so that those precious logs get to their right place. Below is a configuration that will send out the logs to the console with a pretty layout 💅, as well as to logstash, via tcp (default is localhost:5000).

That simple, we have now our testing framework fully ready 🚀

d. ELK packaging structure

Onto the ELK stack. Below is the base packaging structure for its implementation using Docker and Compose:

├── docker-compose.yml
├── elasticsearch
│ ├── Dockerfile
│ ├── config
│ │ └── elasticsearch.yaml
├── kibana
│ ├── Dockerfile
│ ├── config
│ │ └── kibana.yml
└── logstash
├── Dockerfile
├── config
│ ├── logstash.yml
│ └── pipelines.yml
└── pipeline
├── main.conf
└── patterns
├── browser
└── status

Friendly reminder, the code used throughout this article is available on the open-source project “Test Automation Bootstrap”, hosted on GitHub.

e. Logstash Grok plugin

Grok is a great way to parse unstructured log data into something structured and queryable”. Thus, we’ll need the grok logstash plugin to parse and to turn those logs that we implemented in step b) into queryable data.

Grok works by combining text patterns into something that matches your logs. Therefore, we’ll filter out those logs accordingly, and drop out the remaining ones that don’t match our criteria.

Regarding the patterns, grok sits on top of regular expressions, so any regular expressions are valid. The regular expression library is Oniguruma, and you can see the full supported regexp syntax on the Oniguruma site.

Yet, there are two custom patterns, one for the browser and another one for the status.

All you have to do now is to get the stack up and running and execute your testing framework, using the listener we implemented in step b). Afterward, verify that your logs are being indexed and are discoverable in Kibana (by default on https://localhost:5601).

Image for post
Image for post

Done (for now) 🎉

Now you can create dashboards to be used as your test reporting centralized platform, accessible by everyone. The options are endless. But remember, tailor your dashboard (or dashboards) according to your team’s needs. Make them usable for both technical and non-technical individuals.

Inspired by https://www.elastic.co/pt/blog/improving-quality-assurance-automation-at-ramsey-solutions-with-the-elastic-stack.

Kudos

Support the article, leave a clap 👏 ❤️ , and let us know your thoughts in the comments.

For more articles like this one, you can follow me on medium and Twitter. We can also have a chat on LinkedIn or share some code on GitHub 🤓

Written by

Test Automation Engineer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store