Here is a leading question, if you could chose anywhere for new code or a complete application NOT to break where would you chose that not to happen – Development, Staging-QA or Production? I know the answer is obvious – Production! Yet I have spent seven years total travelling the world helping medium and large entities recover from web applications that were breaking in Production, either partially or totally.
In well over 80% of cases no load testing of any kind was carried out before the code or application was deployed. This is directly equivalent to taking a concept car that has being around a test track a couple of times and then driving it in the Indy 500 believing that it can be driven to win the race. It’s far more likely to disintegrate and at worst plough into the spectators causing mayhem.
Where I did find load-testing was taking place it was often after an environment had already become unresponsive and typically the load testing agent, the tool responsible for generating the load via virtual users (vUsers), was running from the wrong place. Typically load tests would be running from outside all network equipment at the same place where real users reside. The problem with starting load testing there is that if the network or networking equipment was the cause of the application slowdown it is very difficult to pinpoint. I can tell you there were many times that networks and/or ancillary equipment caused application slow-downs.
So where should load-testing begin and what kids of load testing are there? Before we can answer that we have another more basic question to answer; are there other kinds of testing? And certainly there are.
- Unit Testing – This takes place as code is developed and involves testing new or changed code until it is free from errors. Typically the developer is responsible for their own code and should not request that the code be moved-deployed until it is bug and error free.
- Integration Testing – Once the Unit Testing shows that the code is bug-error free the code can be moved to Integration Testing where it is typically tested in the context of the whole application to make sure any changes or new code do not break other things. Once the code passes these tests it is typically moved to QA/Staging.
- QA Staging – This is typically where load testing takes place. The main objective of load-testing code is to make sure new or changed code will perform adequately and with stability whilst the application is handling multiple concurrent users. The number of users required in any application scalability planning.
The steps above relate to the development and deployment of changed and new code. There is another major use of load-testing techniques which itself falls into two major sub sections. The first would be to identify bottlenecks in poorly performing applications. The second would be to determine the capacity of a complete application or large part of an application to take a significant user load increase. This second reason could be as a result of a marketing campaign, product launch etc.
Load Testing itself is an overall blanket term and in our work with clients we follow this methodology to cover one or both of the needs shown in the preceding paragraphs.
Firstly we create/record a series of load testing scripts based on client needs. These typically fall into two categories, first to create scripts that accurately mimic normal user behavior or secondly to apply load to a particular part of an application.
If we are looking to identify existing bottlenecks we use the marvelous JRun-ColdFusion logs and in particular we always add metrics logging to the JVM so we can see how threading and memory behaves. If we suspect possible Garbage Collection (GC) issues we will add verbose GC logging which we disable after analysis. We also use ancillary tools such as SeeFusion or FusionReactor and of course if this is a ColdFusion 8 server the Server Monitoring tools.
Lastly, we employ three types of load tests. Load tests with increasing-variable numbers of virtual users (vUsers) over set periods of time. Secondly, stress testing where we attempt to break the application with maximum concureent vUsers. Thirdly, endurance testing where we run vUser load for extended periods, typically overnight.
By employing all of the strategies in this article you will ensure that no code gets out into production only to fail when put there. Here at Alagad we have a maxim, “Break the code before it Breaks You!”