Image Source: fukushimaupdate.com
Let me paint a picture for you. You pay meticulous detail to your test plan. You have automated test plans to cover common and fringe use cases. You build performance testing into your continuous integration process like you should. You triple check everything by testing the mobile or web application on multiple browsers and devices.
But all of that could be for naught if you forget this one thing: Geography.
The geographic distribution of your users is a factor commonly overlooked in performance testing. However, ignoring it can sink your entire test plan. In this post, we will be explaining why geography matters in performance testing and what you should be doing in your test plan to account for it.
Why Geography Matters
Ultimately, the purpose of testing is to ensure a great user experience that’s bug-free and reliable. But the last mile is the most important part of the user experience. People around the world live on different networks and different internet service providers. Each one has varying qualities of service, bandwidth capabilities etc. Below is a map indicating the average level of access people have to the Web by country. (Blue is high access, grey is none at all.) And of course you’ll find stark differences even within individual countries.
Visit the Web index to see the Web’s growth, utility and impact on people and nations.
Simulating load from a single source doesn’t exercise your server infrastructure in the same way as real users distributed across regions and continents. Bottlenecks caused by network issues, local spikes in traffic, and caching infrastructure in the network can all play a role in your user experience and the performance characteristics of your server infrastructure.
But – how many test plans actually take into account the internet quality of their users when doing testing?
Geo-Realistic Testing
Your tests are considered “geo-realistic” if you are accurately simulating the geographical distribution of your user population while conducting performance and load testing. What ends up happening is the load placed on your servers mimics that of users accessing the application from multiple locations. Which is what you want, because in reality your users are accessing your servers from multiple locations.
The best place to start is by looking at historical traffic patterns. Cuddle up with Google Analytics (or your favorite alternative) and find out which countries are generating traffic to your website. Examine both the countries themselves and any timing patterns. Do people all visit at the same time no matter where they are, because they are tuned in to a single event? Or do individual regions generate their own local spikes? These are the patterns you want to emulate.
Then, talk with your marketing team to determine if there are likely to be any changes to those patterns. For example, if your marketing team is making a big push into England in the next year, then you would want to plan to have a larger proportion of your testing traffic coming from Dear Old Blighty.
Tips for Making Your Test Geo-Realistic
- Evaluate user behavior to determine where users are
Once your application is released, take some time to figure out where in the world your users are coming from. Once you see where they are adjust your load testing accurately.
- Account for both web and mobile
Don’t forget to create a geo-realistic load test for both your web application and mobile application. If you forget one, it could spell a major disaster. For example, the World Cup generated millions of views on mobile and web devices causing one major sports network to crash, read more about it here.
- Use automation as part of your load testing strategy
Develop scripts and patterns for simulating various distributions of geographic users as a part of your automated testing strategy. Plug in to data centers around the world to spin traffic up and down just like you see in the real world.
- Select a load testing product with geo-realism built in
You can generate a geo-realistic load on your own without a product however, building your own can be very expensive. We recommend using a cloud generation tool it’s cheaper and allows you to scale the load appropriately. Check out our load testing tool here.
- Monitor your app to adjust your testing strategy
Don’t forget to check the stats of your application frequently. Monitor to see if there are any new behavior patterns or spikes in load generation that would indicate a need to adjust your load test according to geographical location of users.
Don’t Underestimate Geography
Geography for mobile or web applications is important. Think about it, there are over 2 billion internet users, and of those 2 billion 70% are accessing the internet everyday on their smartphones or computers. You don’t want to have your application suffer because you forgot to include geography into your load testing strategy. Remember, above all, it’s the end user experience that matters most.