Testing in production refers to the act of frequently performing tests in a live environment, after production deployment. It is not a substitute for any other type of testing, and the significance of the QA team still remains intact.
Production testing has both its advantages and disadvantages, which may explain why some testers approach it with hesitancy, or have done so in the past. Let’s have a look at these pros and cons, below.
Pros
Real World Accuracy
Testing production systems are more geared towards reality. On the other hand, testing in non-production environments like development, quality assurance, and staging systems, does have value but it does not mirror the real world. This is because they are running on different codes, server operating systems, and patch levels. Often, testing in the two environments will yield starkly different results, which are particularly noticeable in penetration testing, as configurations vary at the application and server levels.
Verifying Functionality
Smoke and sanity tests are mostly performed in production environments. Smoke testing checks if the product’s main functionalities are working correctly. Sanity testing verifies if bugs have been fixed and the functionality of the product is as expected. This enhances the likelihood of improved service.
User Experience
Performing usability testing in a production environment can give businesses an understanding of the user experience. A/B testers offer two versions of a product to users and find statistics to determine which version performs better. This A/B testing method is ideal to test for new features, whose market value is unknown.
Disaster Prevention
No matter how much testing is done in development, it is inevitable that bugs will reach the production stage. If the DevOps team is prepared to handle defects at the production stage, the recovery time will be faster.
Hence, testing in production helps in averting disastrous situations—by taking testing a step ahead of basic prevention techniques, to monitoring methods that detect and fix bugs continuously. The application gains resilience and can recover from unexpected chaos without loss of functionality or data.
Complete test coverage
As conditions of a production environment cannot be fully replicated, complete test coverage is not possible before product deployment. Due to data and traffic limitations, many edge case tests in staging environments will be missed out. Testing in production will give full test coverage via continuous monitoring. With production data, every possible scenario can be detected. Also, developers often make changes in applications and deploy to production, skipping the Quality Assurance verification. These changes create bugs in the application. Testing in production with software testing tools removes the risks caused by these changes.
Cons
Irresponsibility in testing
Testing in production improves the overall testing process but it is not meant to lessen the tester’s responsibility for testing in other environments. As production testers share the responsibility of testing, QA testers may develop an unmotivated approach and not utilize the saved time towards other types of testing. Testing in production will only be effective if team members collaborate fully and not neglect other types of testing.
Test timings
If a load test is run on the production environment during business hours, it increases the chances for poor user experiences. Experts recommend running tests in production environments during times that do not impact a lot of users; for example, during maintenance hours. Testers get a smaller time slot to perform the tests. As user experience is tied with profit, a tester not adhering to the recommended timings can lower the business revenue.
Chaos Monkey
Chaos Monkey is a code with which chaos is created on purpose in a production environment. It is a technique to verify if production is working as expected. However, a code that injects failures deliberately can cause problems in an environment like damaging a web server. Chaos monkey should be used carefully when developing a resilient application to ensure application tiers are not killed.
Verdict
If conducted properly in a way that enhances efficiency, testing in production is a fruitful opportunity. Its benefits bring the product team closer to the end-users. However, the scope of testing in production has not been fully tapped into yet. Thus, exploring the potential of testing in production will become clearer gradually, as it is carried out more.
When executed effectively, testing in production offers invaluable insights into real-world user behavior and system performance. By directly engaging with end-users, teams can identify potential issues and opportunities for improvement in a live environment. However, despite its potential benefits, the adoption of testing in production remains underutilized in many organizations. As teams continue to explore and embrace this approach, its full potential to enhance product quality and user experience will become increasingly evident.
