The dark side of UI test automation - Part 1

My first steps in UI test automation were bumpy and frustrating. Today, almost nine years into automated quality assurance, thinking about my beginning, makes me grin. I consider myself lucky since I had good mentors to provide the knowledge and motivation to help me strive to be a better engineer. In my journey, I read many books and articles about the ‘best practices’ around building test automation, software development, and development in general. However, I have seldom read about the problems with UI test automation. Where you could go wrong, and what should be done or not be done at a team or organization level.

This two-part blog series is not a ‘Top 5’ or ‘Top 10’ compilation of things to avoid or to do, but an honest assessment of my experience regarding the real problems with UI test automation.

My career as a test automation engineer started because of bad communication. Before I started working in IT, I was a chef and had a pretty basic education – an apprenticeship – where I learned the most basic stuff to set foot into the world of IT. My internship, which came along with the apprenticeship, started well as I developed a tool for data transfer from one device to another.  

Hard work paid off, and I got a job there. My knowledge of C# helped me get a new project, which was to write automated tests using Ranorex (a Record-Play tool), which also allows writing test cases as (C#) code. So, my new project was to write automated tests. Alone. With no senior developer or even another junior developer.

I had no knowledge of test automation, its benefits or challenges. I didn’t know anything about concepts like Data Driven Testing (DDT) or Keyword Driven Testing. I didn’t understand the importance of structuring the code modular, so it may be reusable and had never heard about the test pyramid or an Xpath selector. To put it plain and simple, I was a total noob.

Also, I had no idea how to take a manual regression test and transform it into test sets for automated tests. I didn’t even know that there is something like proper test design.

My first task in test automation was to automate regressions tests, which were executed manually, classic right?

Problem 1: Missing architecture – Missing coding skills – Missing everything

The situation:

Since this was my first job in IT and my education was elementary, I had no idea how to structure or architect code. I didn’t know anything about the concepts of clean code or SOLID. As one would expect, my first test automation solution was a really bad case of spaghetti code.

• One method was one test case, which sounds alright, but wasn’t, since all the logic and communication with the SUT (system under test) has been handled in this method
• There were no keywords or data-driven test cases
• Selectors were managed because of Ranorex – but with more knowledge about selectors, Xpaths, and the handling of selectors, it could be improved
• The source code I wrote had endless duplications

Despite the bad design, the automation worked as planned, and I was satisfied.

The solution didn’t even last three nightly runs. Simple changes to the UI of the SUT broke around 15 tests, and thanks to my bad code design (if you could call it design), I now had to address each broken test case to fix it. These were not the only changes, and new changes broke the same test again. Soon, my only focus was to fix test cases as there was no time left to implement new ones.

What I have also seen more often than not is, that someone sets up a framework for test automation. In best case someone with experience and understanding for clean code and architecture, only to put it back into a drawer, because there is no immediate need or time for test automation. Sometime later, an intern must pick it up and start working with it because suddenly the pain to manual regression test is too big and may only be handled by automation. So, the intern is in the same situation as I was at the beginning of my career. The test automation will not be reliable or efficient, will get a bad reputation, and will not be used any further.

How to handle this situation:

In my opinion, a test automation engineer has a hybrid role. On the one hand, he/she is a QA engineer and, on the other hand, is a developer.

This means that a test automation engineer should understand test data management and test design, as well as be able to program and implement a maintainable framework, and subsequently extend such a framework. There is nothing to be said against training an intern or a junior, to a test automation engineer. I welcome this approach, but one should not forget that (as with any junior) there is a need to work intensively on personal development. An experienced developer or test automation engineer must, therefore:

• Conduct regular code reviews
• Perform pair-programming sessions
• Be available as a mentor (in the best case, provide a training program)
• Set realistic, measurable goals and track them with regular reviews

For someone who starts with test automation, this means that they should do self-study and read relevant literature and blogs. The possibility to participate in trainings, conferences, and meetups that cover topics like software development, testing and test automation should be provided. These opportunities should be openly communicated; on the one hand, it helps to focus on work, and on the other hand, motivation stays high since the development is on a personal level as well. To track this, there should be at least an annual performance review where these things, as well as career path, can be openly be discussed.

Problem 2: Test automation as a pet project

The situation:

Over the years, I have often observed that a test automation framework should be set up and expanded by a developer. The idea of having a developer speaks for the approach, but the work of a test automation engineer cannot be done besides the daily development tasks.

Test automation becomes more and more extensive over time. The typical daily tasks are:

• Evaluation of failed tests
• Replication of the detected errors
  • Creating defects or bugs
  • Or by simply adjusting selectors
• Adaptation of test cases due to changes in UI or use case process
• Implementation of new tests
• Test data management (prepare, anonymize, or synthesize data)
• Test environment management
• Service virtualization (occasionally)

These are all things required to maintain a robust test automation framework that delivers actual reliable results with value. But, as mentioned above, a 'test automation engineer' is a hybrid role. He/she should develop a framework and work very closely with quality assurance. Typical tasks are:

• Analyzing the acceptance criteria of new features
• Using the result of the analysis to design test cases
• Using the existing manual regression test for test automation tests
  1. Therefore, the tests need to be adapted or redesigned, so they are robust and deliver useful results
• Documenting and in the best case mapping test with the respective acceptance criteria or user story

Based on their education and experience, developers and QA engineers have different perspectives on software quality. This fact can affect the test case design. I'm not saying that developers can't test or design test cases. Eventually, they need to write a lot of unit and integration tests on a daily basis, but the focus required for a unit test is completely different from the focus required for UI tests. Therefore, it is important to value the different views in a team and use them to our advantage.

How to handle this situation:

Test automation does not work on the side!

If a developer should develop UI test automation, it must be ensured that UI test automation can be developed full-time. If there is spare time, it is possible to work on development tasks, but the primary focus must be on test automation.

In my opinion, a test automation engineer must have a good understanding of QM/QA in the software lifecycle. With trainings like the ISTQB Certified Tester Foundation Level and ISQI-Certified Agile Tester, it is possible to acquire knowledge and become a successful test automation engineer. Additionally, reviews of implemented test cases with a focus on test design by experienced QA engineers are very valuable and help to minimize bad test design. This, by the way, is also true for a developer’s unit tests. It might be a good practice to talk through a unit test with QA engineers to get more meaningful coverage.

Conclusion

A lot more can go wrong with test automation. My main point is that test automation is software development and we should view it likewise. It should be treated like the project it will test. When everybody is on board and knows the importance of test automation, it is only then that test automation can bring value into the team.

In part 2, we will continue to talk about how to handle problems with the environment with a short take on test automation tools. Stay tuned!