If you’re like the customer I spoke to yesterday and it takes you 24-48 hours to setup and build your load test environment each time, then you’re definitely doing something wrong.
Unlike functional testing, where all that’s needed for a test is a desk and a desktop, building a load test environment is a bit more challenging – regardless of the load testing tools being used. So here are a few tips that might help.
1 – Use your production environment
Yes, this a bit awkward as the first tip, and it is rarely feasible. But still, your production environment is the ideal environment for testing too. No need for replications, and it always includes those unnoticed background processes and environment variables that affect performance and are often missed.
A few of our customers (including a bank and a stock exchange), use their live production environment to run load tests during night hours and/or weekends. So it’s definitely do-able, and would be my first choice whenever possible.
When using a production environment for testing, make sure to:
- Backup production data so that you can quickly revert to the real environment once testing is done
- Address the issue of Either isolate/disable 3rd party applications (such as payment systems). For example, when testing an ecommerce site, include Visa or PayPal payment activities. For these, you’d need to collaborate with 3rd party companies and delete all test data. Alternatively, you can use stubs to simulate such activity. If this is also not possible, then run transactions only up to the point of the 3rd party activity you want to avoid such as a transfer to an account.
- Use fictitious entities (such as user accounts) to simulate the activity you want to test.
2 – Make an exact replica of your production environment
Extrapolating load test results is a very risky business. If you run the test on a 1GB machine with 2 cores, what will results look like on a 2GB machine with 4 cores? It’s practically impossible to know.
Make sure, therefore, to replicate your production environment in every aspect – machine profiles, configuration, DB, network architecture, load balancers, firewalls, etc. One method is to create complete images of production machines, to be duplicated in your test environment.
3 – Replicate a customer environment
Let’s assume your product is an on-premises software – for instance, an information system for universities, managing students, registration, administration, etc. It would be quite difficult for your R&D or QA teams to simulate a real-life environment for testing purposes. The most effective way would be to duplicate a customer’s production environment. Try working with selected customers with the assurance that you’ll protect or remove critical data, and by providing discounts or free support. Essentially you’re offering to test your customer’s real environment and take over some of their test efforts, no?
4 – Build a real environment from scratch
In some cases, you’ll have no production environment to copy and will have to build a load test environment from scratch. Possibly, you’re in early stages of developing your software and don’t have a production environment or enough customers yet. Or you may also be restricted by security issues and have no access to the real production environment.
In such a case, make sure to:
- Take into account different types of potential deployments that you might want to test – small, medium, large, etc. With each, address the number of users configured, how many records in the DB, etc.
- Create configurations as similar as possible to the typical production – security, hardware, software, network, other apps deployed, load balancing, and so on.
- Use automated scripts to populate the DB representing the real environment.
- Incorporate less apparent, ongoing procedures. Quite often, performance degradation occurs due to periodical tasks such as a database backup or a weekly BI report generated for management. Problems are not detected since these periodical tasks were never configured in the test environment.
- Add 3rd party integrations used in the live production system such as payment systems, reporting or support, etc. See the earlier comment as to how to handle these during testing.
5 – Establish a procedure to quickly setup your test environment
You want to be able to set up your load test environment as quickly as possible, so that you can re-used it for new tests. Once you have your environment working and configured, create a snapshot or image so that you can re-create it in minutes. Also, there are many configuration management and IT automation tools you can use to quickly setup a server environment such as Puppet, Chef, Saltstack, Ansible, Docker, or Vagrant.
6 – Isolate the test environment
Computing resources are always scarce and test machines are often the solution. When the CFO or the new product team are looking for a machine to run a process or trial, they’ll often land on some test machine – and don’t be disappointed if you’re not notified. You want to make sure that no one but your virtual clients can reach your tested application. Also important is that your test network is isolated so it is not affected by other company activities, of which you’re not aware.