The Automation Misconception: Should you really automate your testing?

“Let’s automate!”  is a newly popular rallying cry from technology companies. 

“Looking for a QA manager who will transform our manual process into a 95% automated process!” teases a job listing for a role at a technology company.

Many companies see automation as a “quick fix” for testing.  Not so fast.

These statements imply manual and automated testing are equivalent, if not interchangeable.  The reality is that automation during the testing phase is simply one tool at a tester’s disposal. 

But  automation isn’t a silver bullet.  Automation only makes sense if it either:

1. Saves you time and money, or
2. Results in a higher quality product that justifies the additional cost

In other words, automation doesn’t make sense if  the time that it takes to automate and analyze automated regression tests takes longer than it would have taken to do the same tests manually. Usually, the simpler a regression test is and the more times it needs to be run, the more cost efficient it becomes to automate that regression test. On long term projects, there is in fact a substantial savings of time and money that can be had through automating and reusing regression tests.

Automation Case Study

One recent gamified mobile app we built involved upwards of 300 badges awarded for various user actions.   Since our QA environment allowed us to spoof awarding of badges we quickly identified an opportunity for automated testing.  Our script triggered each badge’s confirmation page, verified the page’s copy, shared the badge to supported social sites, and took a screen shot of the resulting social posts.

Using the automated script, each of the 300 badges took about 25 seconds to test for a total of about two hours. Critically, the tester was able to continue working during much of those two hours since the automated test does not need close monitoring. It took another two hours to check the 300 resulting screenshots looking for bugs.

Compare that to the manual process. The tester would have taken upwards of 2 minutes each to trigger the badge completion, verify the page copy, and verify social posts.  The entire manual test would have taken ten hours. Using automation testing we saved time and ensured a higher-quality product.

Automated testing is a powerful tool in the right circumstances.  But it has to be carefully considered and weighed against other variables such as the length of the project, cost of manual testing, and maturity of the testing tools and scripts your team has available.