Your Modern Application Testing Strategy: Big Bang vs. Parallel vs. Prioritized

What drives you crazy at work? Is it your co-worker having every single call on speakerphone and later accusing you of eavesdropping? Cutting her fingernails during a conference call? Eating a smelly tuna sandwich at his desk — every single day without a fail?

But if you are an operational packaging manager your pet peeve might be people that don’t take application testing seriously enough and later complain if there is a problem. If you are nodding your head, I feel your pain — please read on and share your thoughts and experiences with me.  

Realistic Application Testing Is Now More Important Than Ever

With the introduction of Windows-as-a-Service, Microsoft has been touting the new operating system’s improved app compatibility: “Windows 10 has been the most compatible version of Windows to-date, with over 90% compatibility against thousands of popular apps.”

And in fact, Microsoft achieves a much higher out-of-the-box compatibility rate because it utilizes its massive telemetry data collection as well as its Windows Insider program and “flighted builds” to collect as much insight as possible and fix the biggest leaks before the new version is released into the Semi-Annual Channel. However, most of this data is collected from consumers and small businesses and isn’t 1:1 translatable to an enterprise environment. 

I have heard on multiple occasions that Microsoft has told customers that enterprises should retest between 5% and 10% of their estate, but this

1. Assumes an extremely tightly-managed application inventory and lifecycle of each individual app and
2. Is in most cases unrealistic as many larger organizations have custom-developed software or niche products that do not fall into the “thousands of popular apps” category. 

The risk associated with not sufficiently retesting your applications for the next Windows 10 build release, not identifying which of your apps are compatible and which are not, and therefore not retesting them cannot be understated!

Enterprises Tackle Their Application Testing In One Of Three Ways

So, let’s for argument’s sake say that you are managing your applications in a central repository, you are categorizing them by potential impact for causing business disruption should something go wrong, and you are retesting a large chunk or the majority of your applications before rolling out a new Windows Servicing or Office 365 update.

Now, can you do that without getting overwhelmed? There are three main ways I have seen enterprises tackle this:

1. Big Bang: Pause in their migration stream and get at least XX% (mostly its 80-90%) of the applications in a business unit or location tested before proceeding.
2. Parallel: Test your applications for one business unit, corporate division, or location just in time to be ready for the next migration step, before heading to the next one. 
3. Prioritize: By prioritizing your applications, you can discover low hanging fruit. This enables you to ready and test widely-used, high- and medium-impact applications first and accelerate your migration significantly.

Let’s look at each in more detail:

Big Bang Testing In A Business-As-Usual World

Some organizations prefer to get (nearly) all their applications ready and tested before moving on to the next step. This traditional big bang approach gives them a sense of control and accomplishment as they can check off another milestone in their project plan.

This is especially true if organizations are combining a larger upgrade with a hardware refresh. This way, IT can have a clear run with a vendor to upgrade and they don’t have to worry about the end users that have already signed off on their apps.

However, the problem here is that you might never get done! Even if your goal is to get 80% done before moving on to the next step, for a large global enterprise, this could easily mean 5,000 apps or more! With 99% certainty, you will not be finished in time to migrate before the next upgrade cycle rolls around the corner! It’s simply not possible anymore to plow through such a workload using a stop-and-go approach. 

Parallel or Just-In-Time Application Testing

While almost all organizations were managing their application testing using the big bang approach, some now recognize that this old way of doing things won’t cut it in an environment that has to undergo massive changes at least twice a year. 

Not quite ready yet to go full into Business-as-Usual, IT teams using this approach run their application testing in parallel, which means they are trying to get the applications ready for one department or division just in time to be migrated. Once that set is completed, the packaging and testing team is quickly moving onto the next set of apps for another department.

This approach has several implications. In the example of an investment bank, we could test all applications for human resources first before turning our attention to corporate services and investment banking. Now, once those three divisions are completed, other business units can already benefit since there will always be some overlap in app usage. The key here is reaching critical mass quickly and leveraging it correctly. But how do you know where you could have the biggest initial impact? That is where prioritizing comes in.

Prioritizing Applications For Testing To Streamline Business-As-Usual

Managing your application testing workload through prioritization is the most advanced of the three approaches — and without a doubt the most accelerated.

While you will need to initially invest in an IT Transformation Management tool, like Juriba Dashworks, that will give you a central command and control over your estate, you will be able to run through scenarios to find those low-hanging fruit. For example, by finding applications that are used by 80-100% of your user base, you can determine which departments or locations should be migrated first as they require the least manual intervention but help you get big numbers on the board fast.

Share Your Thoughts: How Are You Getting Your Users Over The Line?

We would love to hear your thoughts. Are you going to use a big bang approach and get your apps ready first and then migrate? Or are you going to get your apps ready just in time and migrate just in time? Are you planning on using an IT Transformation management tool and automating your application packaging and testing process? Please let us know in the comments below!