There are endless articles about types of website load testing, but most remain at the theoretical level. It is well understood that ‘capacity testing’ is aimed to test the maximum capacity of your system – but what does this mean exactly? When would you run such a test and what type of results can you expect to see?
This post takes a look at four key types of load testing, describes when to use each, and provides examples of the type of results you may expect to see.
1 – Load Testing
As expected – this is the bread and butter of every load tester, where you test how a system behaves with a large number of users and what is the response time received for pages under different scenarios.
For example, in the graph below, you’re running a load of 20 users to see that the page time does not exceed 3.5 seconds.
2 – Capacity Testing
Capacity testing (sometimes called scalability testing) helps you identify the maximum capacity of users the system can support, while not exceeding a max page time you defined.
The previous example showed that the system easily served 20 users with a page time of 3.5. Now you want to find out the capacity of your system – or at which point it will not be able to keep the 3.5 seconds response time? Will it be at 21 users, 30, 40, or 50 users?
Your higher-level objective is to identify the ‘safety zone’ of your system. To what extent can you ‘stretch’ it, without hurting end user experience?
In the results graph that follows you’ll notice that for a page time of 3.5 seconds, the system supports 28 users. But when the load size reaches 29, page time starts exceeding the threshold of 3.5 seconds.
3 – Stress Testing
The objective of a stress test is to find how a system behaves in extreme conditions. You purposely try to break your system, using any set of extreme conditions – whether doubling the number of users, using a database server with much less memory, or a server with a weaker CPU.
What you’re trying to find out is how will the system behave under stress and what will be the user experience? Will the system start throwing out errors? Will response time double? Or will the entire system get stuck and crash?
To continue with the previous example, instead of stopping at around 30 users (when the page time exceeds your goal), you’d continue to increase the user load on the system. Your stress test discovers that up to a load size of 41 users, the system functions, despite the increase in page time. But when the load size reaches 42 users, the system start to deteriorate, with page time reaching 15-17 seconds.
4 – Soak Testing
Quite often, the ‘standard’ load testing, which is run for a short limited time will not succeed uncovering all problems. A production system typically runs for days, weeks and months. For example, an eCommerce application must be available 24×7 and a stock exchange application will run continuously on work days. The duration of a soak test should have some correlation to the operational mode of the system under test.
Soak testing aims to uncover performance problems that surface only during a long duration of time. During soak testing, you’re asking questions such as:
- Is there a constant degradation in response time when the system is run over time?
- Are there any degradations in system resources that are not apparent during short runs, but will surface when a test is run for a longer time? For example, memory consumptions, free disk space, machines handles, etc.
- Is there any periodical process that may affect the performance of the system, but which can only be detected during a prolonged system run? For example, a backup process that runs once a day, exporting of data into a 3rd party system, etc.
When running soak testing, you’re looking for trends and changes in system behavior. In the following example, notice how the load size is constant for a while, but still, there is a memory leak (=green line). At first this memory leak may not affect the system, but after a while the system may crash. (The example below is shorter than a typical soak test, which would last hours, days and even weeks.)
Another use case, seen below, is a constant load size, but then, after 50 minutes of running the test, there’s an increase in memory (red line), which hurts throughput. This is due to a periodical, background process that affects the system when it starts running. (Again, in a real soak test, this would take much more time).
If there’s one lesson from these four types of load testing, it is this — too often performance engineers don’t give enough thought as to what are the objectives of each test run what are their expectations from the system under test. Before you invest in building complex, time-consuming scenarios, make sure you have clear goals for your tests.
-
Jason