Performance test


DevOps is a popular trend in IT and have been gathering steam. The challenges with DevOps for large companies are plentiful. One of the key roadblocks comes from the Vendor Management and Vendor Contracting groups.  Nothing to do with the actual task at hand. Large companies often outsource development to one or more large System Integrators.  They develop the code or assemble the solution and hand it off to another company doing the operations support. The contracts these companies write are usually very specific on who is responsible for what.  If you want one Integrator to support your performance testing, that must be in the contract and part for the price or bid.

How do we get the two or more SI’s to work towards a DevOps model? The customer must adjust their contracting terms to accommodate this and provide a path.

Starting the DevOps culture in a Large fortune 500 company is no small task. You can define and start a pilot within your Performance testing team. A meaningful performance test requires multiple disciplines to execute and analyze results and for root cause analysis.
The Performance engineer drives the process, the application architect can see the application under load or stress to see how their design is working, the operations team can see the installation and configuration process, can help with the automation and deployment scripts or approach.  All can benefit from the APM monitoring that is required for critical transactions. I think it might a smaller challenge to get some SI buying for this type of approach.  Then perhaps more onto a Production application.



Purpose of a stress test

The purpose of the stress test is to design and execute extreme testing scenarios that will cause the application to violate its Service Level Objectives. This is accomplished by increasing the workload on the application for both online and background transactions. The workload for the stress test is well above the normal workload day. A key outcome of the stress test; it allows you to find out how much headroom there is in the system before the user experience is severely impacted or the background processes slow down.

  • For online transactions, the response time will start to increase under the workload as more users are added and as the transaction arrival rate increases.
  • For background transactions, the component throughput will decrease as the workload increases. For example, starting at 10 Orders per second, it will slow down to three Orders per second.

Ultimately the workload is increased until the application breaks.


Stress test entry criteria

A large application stress test requires a large amount of preparation in order to get ready to run the tests and maximize the value from it. To successfully execute a Stress test, the starting performance of the application is critical to know. Is the application performing well under a normal workload? Otherwise, what is the value derived from the test if the application has not achieved the service level objectives before you start the stress test? Here is a sample checklist you can use to help determine if your application is ready for a stress test.

Item No. Criteria Comments/details
1 Passed performance test scenarios and achieved the performance goals
2 Critical business transactions have met the service level objectives   (list critical transactions)
3 Meeting current online transaction service level objectives
4 Meeting the background or batch service level objectives
5 Meeting the real-time messaging requirements, at or better than the   desired throughput
6 Application is at the expected Release Level or version for   production
7 At normal load, it is fitting within the Batch window
8 The Stress testing system and application configuration is like   production configuration
9 Data in the database is at the expected production size.