Monthly Archives: May 2013

Market Data

Calculating system momentum – A market basket of transactions and an index

Can we use momentum as a derived value or index to alert us to impending problems with the application or system? Well, the transaction response time is really a byproduct of the workload on the system resources. So, may be a better way to look at it is; does the workload have momentum? Is the workload increasing or decreasing? Borrowing from Physics, momentum is equal to mass times velocity.

We could use transaction complexity to represent mass, we all know that some transactions are heavier than others. However, using response time as velocity really does not work. Instead I could use the transaction arrival rate to represent velocity. Then I could say that the transaction or system momentum is increasing as the arrival rate increases, taking into account the weight of the transactions.

What I am looking for is a communication vehicle to let non-technical people know how the health of the system is.
Momentum is equal to the transaction weight times the arrival rate of the transactions.

I need to pick a rating or scale for my transactions; 1,5,10. Then there is an overall transaction arrival rate and an individual transaction arrival rate. I need the individual transactions in order for the momentum index to have a chance of being relevant.

M = (T1 * T1 TPS) + (T2 * T2 TPS) + (….) or index?

This would be a very custom index for each application. It represents a market basket of transactions. Much list an EFT represents a basket of stocks.
Also, what I want to determine is how quickly the momentum is changing up or down. If I can get the real-time transaction arrival rates, then I can use the momentum to get an early warning of trouble in the system. Another term, might be a volatility index for the application. Can I get the alert in the front-end of the application early and the correlate with all the system resource monitors.

For this I need to borrow from the Financial markets High Frequency Traders. They have tools and techniques that track large amounts of market date in real-time and try to jump in-front of the market momentum. In need to jump in-front of my system momentum.
The faster I can determine that the arrival rate of the heavy transactions is increasing, then I might be able to jump in-front of that and prevent an application or system outage. I need to calculate the rate of change in real time of the arrival rates. I need a to see that at a clock tick at time zero, the arrival rate is 10 TPS and the transaction response time is 300 ms. Then I need another sample at the next clock tick to calculate the TPS is now 11, and the response time is 305 ms. Perfect for using HFT techniques.


Model test case execution

So, what is the difference between a Software Benchmark and a software performance test?
The key difference is the size of the bet the business has placed on the outcome of the project.

There are hardware benchmarks, database benchmarks, and of course the Transaction Processing Council (TPC), with a long list of standard benchmark tests for companies. The Software Benchmark I am referring to in this article it about the custom application benchmark; designed for your particular business or industry. The best way for a business to make a critical bet the company decision is to define, design and execute a Benchmark for its unique workload and transaction volumes.

Who uses a benchmark

The Software Benchmark is needed to determine if the business will use the new application or technology for business advantage. The workload must be well defined, the database must be at production sizes, and the system resource consumption must be clearly monitored. When the benchmark is executed and the question is answered; all the details, the facts, the database demographics, the workload, must be clearly understood by the business decision makers. The Benchmark team must get the test right, in the allocated timeline, if the benchmark takes too long, then you have your answer. The results of the Benchmark typically undergo tremendous scrutiny. A third party is often required to provide the needed visibility and help pass the scrutiny.
There are two categories of companies that undertake a benchmark;
1) Software vendors – the companies that make the software application
2) Consumers of the software – the companies that will use the application for business value

The software performance test is usually for a project or program already underway, the application and technology are already decided. Performance testing is used to make sure the Releases will meet the Service Level Expectations. The Workload for each test may focus on specific parts of the workload and skip others for a given Release cycle. The performance test plan may include;
1) Component testing
2) Duration testing
3) Stability testing
4) Failover and failback testing
5) Also, a round of tuning may be added

Benchmarks for software vendors

The business has decided to move the product in to the large client segment of the market. As such, they need to demonstrate that their application will be able to scale to what their market considers to be large, 10 Million accounts for instance. The application must perform well at this level. The key business transactions must still respond in less than two seconds. The overnight processing must still be able to be completed in the defined window, say 4 hours to complete the billing process.

The bet: Business and revenue growth in the large segment of the market. Repositioning of the company in the marketplace in relation to competitors.

The consumers

There are a couple of scenarios for this category. One is the business is already considered large in the marketplace, they already have 10 Million accounts or more. However the systems in place are older with restricted functionality that is not easily changed. The business needs to add new features to stay ahead of the competition. The second case is a growth plan, where the business believes to increase its market share, it must grow significantly. The business may currently serve 1 Million accounts, but now have a three year business plan calling for a growth plan to 50 Million accounts. They need a technology platform that can scale with them.
The bet for an already large business: Maintain the current business and add new features quickly on a new platform. Stay a market leader, if the new platform fails, you are no longer the market leader.

The bet for a growth business: Easily acquire new accounts and gain market share, or stumble.

Implementation approach

The software benchmark is an event for the business. It is highly visible to Sr. Management, if you are the software vendor, it might be highly visible to your sales pipeline. There may be significant deals waiting on the outcome.
There are typically three to four phases required. Even before that, the organization must review the resources required, people, time, equipment, and budget. The focus on a benchmark can distract already busy people. The developers of the code may not have the time or the skills to design and execute a formal Benchmark. The same for the QA team. A Benchmark may require the use of an external testing lab in order to get the proper configuration. The Benchmark project must be treated as a distinct project.

Critical areas

Business goals and objectives

Clearly state the purpose, to demonstrate the system can safely support the workload of 10 Million accounts. To demonstrate predictable scalability of the application as the workload increases.

Workload profile

In order to gain value from the benchmark, you must have an accurate workload. There will be several user profiles for the online component, with the detailed steps they take through the application. For instance, the casual users, the new users, the power users. There must be a representative batch component as well. As the online purchase transactions will drive the invoice creation process in the batch schedule. There could be a month end process and a quarter end process.

Database size and demographics

The data distribution must be well understood and clearly defined. If you have 10 Million accounts, some are active and some are closed. There may be a residential profile and business profile, with different numbers of details under each. For instance, an insurance policy can have one driver or four drivers, plus the cars. For a web shopping cart application, there can be four years of historical purchases. Your database needs to simulate this.

Performance testing process

The performance testing process must be completely visible and flawless. Generally, you do not have time to rerun a Benchmark. The testing scenarios, test execution, metrics and monitoring must be complete. Many tests may be executed in order to get you ready for the official set of benchmark tests. You need to have a very good test results archive system, because you may not have to completely evaluate the test results, until the end.

Results analysis and executive summary

How do you know you ran a successful test? All the detailed results must be compiled and summarized into an executive view. The virtual web transactions and the batch processing must have detailed results. The Virtual users will record the response time of each request. The you must provide the response time distributions, 50th, 75th, 95th. Plus you must include the transaction per second load. Under 10 TPS, the 95th percentile was 2.5 seconds.
The batch processes must include the rate of processing. The rate for the entire Process, there were 5 million invoices processed in three hours. Also the critical path programs must be clearly identified.

External Vendor lab

Often times, in order to hit the 10 Million account target, the Benchmark requires the use of an external lab. The Benchmark may require a large server or large number of servicer, the database server may need to be large and the data storage may need the newest vendor equipment. The marketing department may have signed a deal with the vendor to use their equipment for a reduced price to use the equipment and be part of a press release.