Top 5 pitfalls of Test Automation & how to avoid them!
How to avoid the Top 5 pitfalls of Test Automation Why automate your tests at all? For starters, it’s about 100 times more costlier to fix a defect after the product has gone live, rather than detecting & fixing it during the testing cycle itself! There are some other drivers as well.
Improved customer experience & reduced delivery timelines: With automated tests, you save time, here’s a graphic to showcase the time differential required to test each software functionality, across each release between manual vs automated testing.
Avoid post production defects: You got the software out in a hurry but what happens if your clients discover bugs in the software? You deal with irate customers, take a big hit on the customer satisfaction index and go back to the drawing board. With automated tests you spot & fix bugs early in the development lifecycle itself.
All said & done, test automation tools are supposed to scale up your test practice; however even after investing so much of time, energy & effort into it, why do companies still suffer from low Return on Investment (ROI) from these tools?
As you are aware, the general formula for calculating ROI (which is also applicable for test automation tools) is as below:
ROI for test automation tools = Benefit / Investment = Gain – Costs ( both Direct & Indirect Costs*) / Investment
Listing out a few Direct & Indirect costs as below for reference:
➢ Tool acquisition & evaluation costs
➢ License costs
➢ Tool Training of core automation team
➢ Initial configuration of the tool
➢ Application specific costs – including creation of a test automation framework, test data, creation, maintenance & execution of manual and automated test cases
➢ Risk of undetected failures before the application hits production
➢ Costs attached to these undetected failures
Note: Only if the benefits/ gains outweigh the costs & invested amounts do you reap a positive ROI. So what do you think are some of the pitfalls which can prohibit you from enjoying a positive ROI from your Test automation tool?
Let’s take a look at the Top 5 here:
1. Test automation tools are left to execute everything from test planning to execution!
The major concern of test automation managers is that the test automation tool doesn’t really investigate bugs; the software just runs a series of yes/ no questions to determine the results.
Michael Bolton aka the Magician Tester mentions this whole process of specific action/ inspect result as checking not testing.
All of which means that software bugs can quietly slip away unnoticed!
For instance, when you are testing new software you receive information from different sources; be it the product specs from the client, user stories, the programmers who wrote the code, and information about similar function in rival products. Also, any new information isn’t considered or even observed by the test automation tool.
In short, if you leave the entire testing strategy & execution process to the test automation tool itself, you are surely headed for disaster.
There is a critical need for human intervention here; you need to analyze data, decide which features are doable, which problems to fix,then design & redesign tests, get the tool to execute them until you have the final piece of bug free software.
2. Test automation tools can automate all tests!
Under the guise of reducing time to run tests to check out more & more scenarios or even to address product changes, you end up running parallel tests, sometimes on different servers, to speed things up.
The net result? You have so many tests running that the feedback loop slows down; results begin to arrive the next day instead of the next hour with an army of testers spending day after day running highly detailed tests and programmers running suites of automated checks on different layers of the product. All of which will result in huge time lags in shipping out the final product.
A better option would be for you to classify the tests and decide which piece of software can be tested manually and which needs to be automated.
Take a look at the chart below to get started.
In fact, a test automation tool should get you started on the journey of delivering bug free software in a matter of few clicks, instead of requiring extensive physical infrastructure set up as a pre requisite for it to begin functioning.
Cloud based test automation tools are ideal in this regard, ; in a short span of time, you can run Unit, Functional, Regression, and Performance Tests seamlessly on cloud without having to invest in obsolete hardware. If the test automation tool gives you an option to import the manual test cases into the test automation tool itself it’s an icing on the cake, as both your test & development teams can have a bird’s eye view of the whole test project from start to finish.
3. Test automation tools require extensive coding!
Sooner or later in the test project, there will be a need for recalculation and someone needs to write a code or an algorithm to check if the software passes or fails this on new requirement.
So your testing teams also need to know how to code; especially if you don’t have enough programmers on board, which is again a tall order.
What you need is a test automation tool that does away with the entire need to code itself. Go for a tool with an inbuilt Scriptless automation feature means your testers enter keywords in English; the tool interfaces with the framework & executes the rest.
4. Test automation tools throw up False Positives!
False positives or tests that fail when there is really no product problem are one of the biggest diseases of test automation suites.
A tester can spend endless ours reading failure logs, rerun the offending script and spend loads of time trying to analyze the failure.
Was the test failure caused by timing issues as the tool was trying to click on buttons or enter text when the page hadn’t fully loaded? Or did the product itself change so much that the script itself failed? Or did something happen to the test environment itself causing the tests to fail?
What you need is a tool which makes debugging of automated test runs easy; a tool which can help categorize failures and make test analysis easier.
5. Test automation tools mean you can replace testers!
So now that you have the test automation tool in place, you hive off the testers into support, development or product management teams or lay them off altogether.Which means that you land up with 2 teams of programmers; one writing code for the software product itself and the other writing code to test it.
Ultimately you land up with a team which focuses on confirming that the product works rather than finding out ways in which it might fail!
The sign of a robust test automation tool is that it serves as an aid to a tester, to make the whole testing process more streamlined; however it can never prove to be replacement of the tester itself! The ultimate aim of integrating a test automation tool into your test practice should be towards having a more seamless execution of test planning & execution strategy.
What has been your experience with test automation tools? Share your feedback & comments here.