Abstract

There is this myth in most of the IT organizations and teams that Test Automation is some sort of a magical pill that can take care of improving quality of the software. It is often considered as a gateway in improving the SDLC processes to achieve quality product. However, one faces the real challenges at the time of implementation of Test Automation.

It feels overwhelming at first place to decide scope and timing of automation as well as selection of tools for the same. Even if one clears these initial hurdles, there is always somethings that would have been ignored at first place, that could eventually lead to an inefficient test automation setup. Despite following Automation Best Practice, investing time and efforts result does not yield the expected outcome.

The KEY problem is one is NOT adopting the right process at the right time under the right circumstances which result in poor ROI and high cost.

This whitepaper looks at potential issues and proposes ways forward to overcome the same as well as suggest some best practices that when followed could help in achieving maximum returns from a test automation strategy. The main objective of this white paper is to assess, how can one make test automation more cost-effective and improve success percentage practically.

Essence of Test Automation

With the ever-growing demand for reduction in time to market, there is a significant pressure on the IT industry to adopt innovative approaches and technologies. In order to remain competitive in the market, most of these IT organizations face challenges in bringing out new products/features that is stable, reliable and of very high quality.

Initially manual testing was considered as the easiest, cost effective and less risky approach in ensuring high quality deliverable. However, manual testing takes a longer time to complete thereby delaying the product release. Hence, it becomes inevitable for the organizations to migrate from manual to automated testing. This has the potential for timely release of the product with desired quality and in a cost-effective way.

Test automation has thus become more and more popular in recent years and has been a constantly increasing activity in the existing software industry. Test automation is often seen with many benefits e.g. to reduce the costs of testing, increase test coverage and effectiveness, shorten testing cycles etc. Since test automation is initially costlier than manual testing, it should be thoroughly studied before being adopted. However, success in test automation can only be achieved if certain best practices are followed diligently before and during the implementation phase of test automation suite. 

Pre-requisites for Test Automation

Before jumping into implementing a test automation plan, one should understand that it is a long process and the desired results are not immediate. One should be pragmatic in the approach in planning an automation strategy based on the business needs.

Before migrating from manual to automation, one should pass through the following checkpoints.

Adhering to these checkpoints, is the one of the most practical and easy approach in strategizing test automation and will help in making a rational decision, there by saving quite a lot of effort, time and cost.

All these checkpoints are sequential in nature. Hence, one cannot skip one checkpoint and jump to another.

Why to Automate?

It is very much essential to determine the objectives, benefits and ROI before taking a decision on Test Automation. Before starting any activity in this area, one should ask, “why am I automating?”

One should look at:

1. Objective and Benefits to move to Automation (purpose and benefits of migrating from manual to automation must be very clear)
2. Calculate ROI for the current setup considering the following factors:

         a. Validity of application

         b. Frequency of testing

         c. No. of planned releases in the near future (at least next two years)

         d. Time for implementation of automation project

         e. Automation tool cost

         f. Automation resource cost

         g. Cost of current manual testing

Overall objective of this exercise should be to determine how effective the test automation will be for the project.

Research and experience indicate that it takes almost 10-12 releases to recover ROI on test automation. Also test scripts keep changing all the time based on the change in the requirement or design. Thereby, lifetime of a test script is limited.

The main goal of this checkpoint is to ascertain;

• Reduction in cost of testing
• Reduction in release time
• Quality of deliverable
• Effectiveness of test automation as value addition

If all the above goals are achievable and measurable, then it makes sense to move to the next checkpoint i.e. When?

When to Automate?

Once it is decided to go for test automation, the timing of implementing the same becomes very important. One should not attempt to do test automation if the application is not stable and most of the basic regression tests are failing when executed manually. A reasonable amount of stability in the application is expected before implementing test automation. Hence the minimal criteria for the right timing for test automation should be as follows:

1. All the regression tests must be consistently passing with 100% results.
2. All the test cases should be clearly defined with preconditions, test data and expected result. All these test cases must have been reviewed and finalized.
3. The entire application’s manual testing should be completed for at least two releases with at least 70% pass percentage as test result of the overall application.

If all the above things are in place, then it makes sense to move to the next checkpoint i.e. Which?

Which test case to Automate?

It is important to know that all the test cases cannot be automated. Some of them will still have to be tested manually. Hence it becomes critical to identify the test cases for automation based on priority and ease of implementation.

Typically following test cases are ideal candidates for automation:

1. Regression Test Cases – Those test cases that needs to be run with every new build.
2. Data Driven Test Cases – Those test cases that use multiple values for the same actions.
3. Test cases that have passed manual validation.
4. Test cases that have more data verification points.

Overall, those test cases that are more repetitive in nature are the best for test automation.

Following test cases are not fit for automation:

1. Usability Test – How easy is the application to use?
2. Test cases that have failed in manual testing
3. Test cases without expected results
4. Test cases that are executed only once
5. Adhoc or Exploratory test cases

Once the test cases for automation are clearly identified, then it makes sense to move to the next checkpoint i.e. How?

How to Automate?

Selection of right tool/framework for test automation is very critical for a successful test automation implementation. It is very essential to select the right tool that best suits for overall test automation.

Following points should be taken into consideration while selecting a right tool/framework for test automation:

1. Technology Compatibility – Language used for application development (C#, Java, JavaScript etc.)

2. All the automation functionality/feature available – GUI, Event Handling, Database, Value verification etc.

3. Ease of learning, adapting and training in the team

4. Ease of usability

5. Scalability, Maintainability and Support 

6. Availability of competent resource

7. Licensing Cost

If based on the above-mentioned factors, a tool/framework is selected for test automation, then it makes sense to move to the next checkpoint i.e. Who?

Who should Automate?

Test automation is not very trivial in terms of selecting a tool and starting with it. Test automation involves writing test scripts for the identified test cases and involves significant amount of coding. Hence it is imperative to have skilled programming resources who are passionate about testing and test automation. Typically, one would need to identify the right resource based on the type of testing being done in test automation such as Unit Testing, GUI testing, Web Services/API testing or database testing.

The test automation team should include:

1. Automation Testers who are good programmers. They are at par with developers.

2. Resources who have knowledge and experience of manual testing.

3. Resources who are passionate about automation. 

It is very much essential for automation tester to design and develop reusable test programs, review and test the test scripts to enable quality test automation process.

If the test automation team is now set in place, then it makes sense to move to the final checkpoint i.e. Where?

Where to Automate?

Test Automation environment is very much similar to the overall QA environment as mentioned below:

1. Test automation should be performed on the QA environment and not on the dev environment.

2. Test automation should be done on the same build/release version on which manual test is going on.

3. Test automation can be on a local setup or VM or cloud. But the same environment should be replicated for other testing.

Benefits of Successful Test Automation Strategy

Test Automation is a long-term strategy that requires initial investment in terms of effort, time and cost. However, a successful implementation of test automation strategy can bring the following benefits across the table:

1. Reduced time and cost for testing.

2. Increased testing quality level.

3. Shortened release cycle.

4. Reliable results.

5. Increased test coverage.

6. Avoiding mundane and cumbersome tasks.

7. Repeatability (running same test cases again and again).

8. Ensure process in place for future success.

Causes for Test Automation Failure

Even though the intent is good, many a times implementation of a test automation strategy fails. Following are the some of the primary causes for failure:

1. Lack of time, competent resources and budget.

2. Lack of process.

3. Wrong selection of automation tool.

4. Application under test being unstable.

5. Lack of programming resources – Purely dependent on GUI automation using record and playback.

6. Automating those test case that failed during manual testing.

7. Poor test coverage.

8. Tight schedule.

9. Lack of resources with testing background.

10. Automation test script not reviewed and maintained.

11. Too many changes in the existing features in each release.

12. Improper test environment

13. Inconsistent and unreliable behavior of the application and framework

14. Too much of automation – Not delivering on time

15. Regression tests failing

Conclusion

Based on our knowledge and experience, one can achieve success in implementing an effective test automation strategy by following the above-mentioned checkpoints diligently.

Test automation is a strategic investment and should be done by adopting the right approach at the right time. The main objective is to develop and follow best practices that highlight the various factors to be considered in order to make a wise and rational decision for test automation.