What Testing in Production Can and Can’t Do

Shift right isn’t quite shift left. There’s a lot of value in testing early in the software development life cycle, also known as shift left testing. There’s also value in testing in production, or shift right testing, albeit with some clear drawbacks compared to a more proactive approach.

Testing in production was once considered largely forbidden, often due to concerns about altering customer data analytics with bogus information. This practice was common for about 10-15 years. But the ongoing market pressure to rapidly evolve applications now places an emphasis on validating features and functionality throughout the entire life cycle.

Production testing adds business value, but there are risks and costs alongside the benefits. In this blog, we’ll offer some tips for performing valuable user testing without adversely affecting the live environment, including the types of testing to consider and the benefits of shift right.

Why consider testing in production?

Testing in production complements QA functional testing, which is typically performed on development and staging servers. By testing in production, development teams gain valuable feedback from real-world testers that can improve application functionality and market response — including some insights that can only be gained from production testing, such as real payment testing feedback.

Professional testers understand the importance of testing pre-production on servers set up to simulate production. The problem is that test and staging servers rarely include all active connections, third-party software, data and settings to truly mirror a production environment. 

Application development organizations should consider leveraging production testing to provide safeguards from unforeseen bugs associated with these potential blind spots. Production testing also offers unique insights from real-world user conditions with real data and wildly variable traffic patterns, device configurations and connectivity levels. 

Types of production testing

Different use cases call for different approaches with production testing. Application development teams gain value from testing in production in scenarios, such as: 

• Hotfix verification
• AB, blue/green or canary testing
• Third-party integration validation
• Validation testing
• Visual testing
• Payment processing validation

Each type of testing in production serves a specific purpose, all with the shared goal of improving the real user experience. When testers verify hotfixes in production, they look for any issues or problems with the specific implemented fix as well as functionality in the same area. AB, blue/green and canary testing typically occur when development teams implement feature flags in the code to control the release of features to production.

All types of production testing, including third-party, visual, usability, accessibility and payment processing, are types of validation testing. The purpose of validation testing is to reduce the chances of defects occurring in the production system. Validation testing confirms a product is ready for release and identifies design issues for future code improvements. Validating an application satisfies users’ needs and expectations is crucial for both new applications entering the market and existing applications adding or enhancing features.

Many application development organizations use crowdtesting teams for production testing. Crowdtesting providers like Applause provide the expertise and resources needed to thoroughly test an application using a wide variety of platforms, OSes, versions, devices and connectivity speeds. Crowdtesting experts provide real-world testing and feedback in rapid fashion. Applause crowdtesting teams often help identify critical production issues — though their expertise is available throughout the software development life cycle — and confirm how an application performs in its intended market. 

Production testing requires planning 

Testing in production requires collaborative planning between development teams, operations and IT. Why? Production testing requires the creation of test data, often a time-consuming and difficult process. Compliance, data specialists, testers or developers might work together to create test data strategies and processes to manually copy data from another system or generate it using automated tools. Creating accurate production test data also means setting configuration options and naming data so developers and testers can recognize it. Data naming must accurately represent real user scenarios and be reusable. 

Despite the temptation to use real user data, this should be avoided unless the team can ensure personally identifiable information (PII) is protected through masking, obfuscation or other methods, which can take time, specialized expertise and tooling. Exposing sensitive data may result in possible security breaches that cost an organization financially and mar its reputation. Keep in mind that using real data might also expose the business to violations of data privacy laws.

This is another area where Applause can help. Applause can assist organizations in sourcing diverse user data for AI training and digital quality validation through a secure, fully managed ecosystem. Through its global community of over 1 million testers, Applause can connect customers to high-quality, human-sourced data — including audio, video and text.

Production testing benefits 

Production testing offers benefits for improving the user experience and quality of applications based on real user feedback. Testing in production provides real-world and real-time validation testing for applications under varied user conditions. 

Product teams can receive feedback to improve designs specifically to enhance the application by receiving feedback from real users. Recurring testing of production systems enables development teams to address issues and to improve the overall system resilience and reliability, providing users with a higher quality experience. 

Developers and testers who adopt a shift left approach might also benefit from implementing test automation as part of their strategy for testing in production. Automated testing in production can expand the benefits of automated regression testing, but production testing must include additional consideration for the creation and modification of test data for the scripts. Many automated scripts need edits or modification to support environment-specific requirements and risk mitigation.

Advantages of crowdtesting 

Consider the possible advantages of partnering with a crowdtesting provider like Applause to complement internal testing. Once internal testing is complete, crowdtesting provides additional test coverage for multiple types of devices, versions, platforms, connectivity and other real-world conditions that aren’t present in internal testing. Crowdtesting teams from Applause provide global testing coverage and varied testing expertise. Real testers perform validation testing, following a test strategy the application team and Applause define together. Applause manages the sourcing, testing and translation of feedback into actionable tasks for improvement.

Testing in production provides real-world access to test in the same environment as real users. As long as the team manages risks and plans for test data creation and testing, production testing has benefits that can lead to product insights. Enhance the customer experience and application quality by implementing a strategy for production testing.

Get started today with a crowdtesting pilot or contact Applause to inquire how they can help you achieve your digital quality goals.