Software Performance Model


A Farrari 458 Italia can move two people at speeds in excess of 200 MPH, the Superbus (pictured above) moves 23 people at speeds of 155 MPH, the newest Maglev train in Japan tops out at 312 MPH, with 14 cars each carrying 68 passengers (952 people). These are examples of bulk or batch movement of people.

Processing data, moving data, from one application to another. This occurs in a number of ways either as a singleton or a large set (batch), instantly, near-real time, delayed by minutes, in batches throughout the business day or in a large batch cycle at the close of the day. Each case has specific performance and throughput goals. Software performance engineering includes methods for batch processing.

Lets talk about batches…

Our complex business and IT systems of today still do a large amount of batch processing, moving data out of one system and into many more systems, with transformation and enhancements along the way. These can be internal systems in the Enterprise or external third party providers or even customers. I have noticed over the past few years that batch performance design has become (by accident) an iterative approach.
The original design goals were simply specified for the entire batch process, it must be completed by 06:00 EST. Often times, the batch critical path is not clear. There has been little thought to the throughput of each batch job (transaction per second). The iteration starts during the QA testing process when it is discovered that the current batch process will take 24 hours to process the days’ work. Someone finally did the math. The team did not know if it was designing a Farrari, a Superbus or a Maglev.
For the critical path batch processes you must define the required throughput. What are you moving? Invoices, orders, customers, transactions. How many of them do you have and what is the peak volume? What is your scalability approach and how do you achieve increases in throughput? Do you need to scale-up or scale-out?
You need to design the batch process to fit comfortably in the batch window and with room to grow. This year 500,000 invoices, next year is 750,000 invoices. How do you process the increase and stay within the window?

Software performance engineering Sample Questions

1 What are you processing?
2 How many steps (programs) are in the batch process?
3 What is the overall throughput?
4 What is the throughput of the individual programs?
5 What is the growth rate?
6 What is the peaking factor?
7 How will the programs scale with the load?
8 Have you identified the critical path throughput goals?
9 Are all the developers aware of the throughput goals?
10 How will you test and validate the throughput of each program?

Know your design goals

The barchettas were made in the 1940’s and 1950’s. An Italian style topless 2-seater sports car, designed and build for one purpose, to go fast. It was built for racing, weight and wind resistance were kept to a minimum, any unnecessary equipment was removed, no decoration. Doors were optional. Ferrari created one of the earlier models and other followed with the same style. The design team was focused on speed, they knew there were performance and response time goals for the car.


Model test case execution

So, what is the difference between a Software Benchmark and a software performance test?
The key difference is the size of the bet the business has placed on the outcome of the project.

There are hardware benchmarks, database benchmarks, and of course the Transaction Processing Council (TPC), with a long list of standard benchmark tests for companies. The Software Benchmark I am referring to in this article it about the custom application benchmark; designed for your particular business or industry. The best way for a business to make a critical bet the company decision is to define, design and execute a Benchmark for its unique workload and transaction volumes.

Who uses a benchmark

The Software Benchmark is needed to determine if the business will use the new application or technology for business advantage. The workload must be well defined, the database must be at production sizes, and the system resource consumption must be clearly monitored. When the benchmark is executed and the question is answered; all the details, the facts, the database demographics, the workload, must be clearly understood by the business decision makers. The Benchmark team must get the test right, in the allocated timeline, if the benchmark takes too long, then you have your answer. The results of the Benchmark typically undergo tremendous scrutiny. A third party is often required to provide the needed visibility and help pass the scrutiny.
There are two categories of companies that undertake a benchmark;
1) Software vendors – the companies that make the software application
2) Consumers of the software – the companies that will use the application for business value

The software performance test is usually for a project or program already underway, the application and technology are already decided. Performance testing is used to make sure the Releases will meet the Service Level Expectations. The Workload for each test may focus on specific parts of the workload and skip others for a given Release cycle. The performance test plan may include;
1) Component testing
2) Duration testing
3) Stability testing
4) Failover and failback testing
5) Also, a round of tuning may be added

Benchmarks for software vendors

The business has decided to move the product in to the large client segment of the market. As such, they need to demonstrate that their application will be able to scale to what their market considers to be large, 10 Million accounts for instance. The application must perform well at this level. The key business transactions must still respond in less than two seconds. The overnight processing must still be able to be completed in the defined window, say 4 hours to complete the billing process.

The bet: Business and revenue growth in the large segment of the market. Repositioning of the company in the marketplace in relation to competitors.

The consumers

There are a couple of scenarios for this category. One is the business is already considered large in the marketplace, they already have 10 Million accounts or more. However the systems in place are older with restricted functionality that is not easily changed. The business needs to add new features to stay ahead of the competition. The second case is a growth plan, where the business believes to increase its market share, it must grow significantly. The business may currently serve 1 Million accounts, but now have a three year business plan calling for a growth plan to 50 Million accounts. They need a technology platform that can scale with them.
The bet for an already large business: Maintain the current business and add new features quickly on a new platform. Stay a market leader, if the new platform fails, you are no longer the market leader.

The bet for a growth business: Easily acquire new accounts and gain market share, or stumble.

Implementation approach

The software benchmark is an event for the business. It is highly visible to Sr. Management, if you are the software vendor, it might be highly visible to your sales pipeline. There may be significant deals waiting on the outcome.
There are typically three to four phases required. Even before that, the organization must review the resources required, people, time, equipment, and budget. The focus on a benchmark can distract already busy people. The developers of the code may not have the time or the skills to design and execute a formal Benchmark. The same for the QA team. A Benchmark may require the use of an external testing lab in order to get the proper configuration. The Benchmark project must be treated as a distinct project.

Critical areas

Business goals and objectives

Clearly state the purpose, to demonstrate the system can safely support the workload of 10 Million accounts. To demonstrate predictable scalability of the application as the workload increases.

Workload profile

In order to gain value from the benchmark, you must have an accurate workload. There will be several user profiles for the online component, with the detailed steps they take through the application. For instance, the casual users, the new users, the power users. There must be a representative batch component as well. As the online purchase transactions will drive the invoice creation process in the batch schedule. There could be a month end process and a quarter end process.

Database size and demographics

The data distribution must be well understood and clearly defined. If you have 10 Million accounts, some are active and some are closed. There may be a residential profile and business profile, with different numbers of details under each. For instance, an insurance policy can have one driver or four drivers, plus the cars. For a web shopping cart application, there can be four years of historical purchases. Your database needs to simulate this.

Performance testing process

The performance testing process must be completely visible and flawless. Generally, you do not have time to rerun a Benchmark. The testing scenarios, test execution, metrics and monitoring must be complete. Many tests may be executed in order to get you ready for the official set of benchmark tests. You need to have a very good test results archive system, because you may not have to completely evaluate the test results, until the end.

Results analysis and executive summary

How do you know you ran a successful test? All the detailed results must be compiled and summarized into an executive view. The virtual web transactions and the batch processing must have detailed results. The Virtual users will record the response time of each request. The you must provide the response time distributions, 50th, 75th, 95th. Plus you must include the transaction per second load. Under 10 TPS, the 95th percentile was 2.5 seconds.
The batch processes must include the rate of processing. The rate for the entire Process, there were 5 million invoices processed in three hours. Also the critical path programs must be clearly identified.

External Vendor lab

Often times, in order to hit the 10 Million account target, the Benchmark requires the use of an external lab. The Benchmark may require a large server or large number of servicer, the database server may need to be large and the data storage may need the newest vendor equipment. The marketing department may have signed a deal with the vendor to use their equipment for a reduced price to use the equipment and be part of a press release.


While sitting on a slow triple chair lift, what do you think about?  Well, a ski resort is a great way to demonstrate software performance and capacity concepts.

You need a transaction rating scheme for your key Enterprise applications and systems.  Its winter in New England and I like to ski. While sitting on a triple chair lift, slowing being carried up the mountain, I thought it would be great if Enterprise applications had a rating system. Each business rated the business transactions as easy, intermediate, advanced and expert, in terms of not only business criticality, but technology complexity.  How many different systems are involved in the transactions? The ski industry has a trail rating system; green circles, blue squares and black diamonds. If the trail is really difficult, it can be rated double or even triple black diamond.

When making the trail ratings, the factors used to make that rating include the trail gradient and slope, the width of the trail, the trail conditions and whether the trail is groomed or not. The trail itself can be a nicely groomed path, a mogul trail (this has many bumps) you must be experienced to enjoy these, or (glades) wooded trails where you ski around trees while going downhill.  There are also terrain parks with half pipes and ski jumps.

The trail rating system is relative for each mountain.  A blue trail on one mountain might be a black diamond on another or a green on another.  So, there is somewhat of a range for the mountain resort owners. I think the key idea is that the rating is mountain specific, that is relative for each mountain. This can be applied to our Enterprise IT systems. Each system can be considered a mountain, each business transaction can be rated as a trail. If you take some of the mountains in the northeast;  Pats Peak (20 trails) , Waterville (52) , Bretton Woods (102), Killington (140), and Sunday River (132); you can see the mountain statistics sheet.

The  mountain fact sheet:

Quick Stats (snapshot of your system)

  • Mountain Name: Different from the resort; Mount Tecumseh
  • Total Acres: 500
  • Skiable acres: 220
  • Longest Run: 3 miles
  • Summit Elevation: 4,004
  • Vertical Drop: 2,020

Vertical Descent by peak (each peak could be an application)

  • Total Vertical: 2,340
  • White Cap: 1,630
  • Locke Mountain: 1,460
  • Bear Mountain: 1,400
  • Jordan Bowl: 1,490

Trails (these are your transactions)

  • Total Trails: 52
  • Novice: 20%
  • Intermediate: 50%
  • Advanced: 20%
  • Expert: 10%
  • Glades: 5
  • Moguls: 6

Snow Making (the engines of the business)

  • Terrain Coverage: 616 Acres
  • Water Capacity: 9,000 gallons per minute
  • Air Capacity: 60,000 cubic feet per minute @ 150 psi
  • Snow making capacity: 4 acre feet per hour
  • Snow making arsenal: 1,900 guns
  • Miles of pipe in system: 73
  • Mile of hose in system: 30

Lifts (hardware and software)

  • Total lifts: 16, including 5 high speed
  • Skier Capacity per hour: 30,000
  • High speed gondola: 1
  • High Speed Quads: 4
  • Fixed Quads: 4
  • Triples: 3
  • Doubles: 2
  • Surface Lifts: 5

Let’s take this approach with you key systems and rate the transactions.  

  • What are your Peaks (key applications)?
  • What is the ratio of the trail ratings (green to double diamonds)?
  • How many lifts and what is the capacity?

The next post is going to review the business and technical transactions with some examples ratings.

What type of application are you building?

The performance goals must be negotiated between the business stakeholders and the application Architect. To have a productive and two-way negotiation, you must understand the type of application and the amount of data manipulated for each transaction. For instance, generating a large report covering sales for that last two quarters, will take longer than adding a product to an order. And you must have supporting data, from current production systems. You must have measurements from the current system on the key business transactions.

  • Online web transactions, ecommerce, browser based and mobile based. These transactions are typically synchronous with a customer or business partner waiting for the responses.
  • Message based transactions: computer to computer transactions, what is the distance between the systems and what is the latency of the network. Typically, they are part of a larger workflow.
  • Business intelligence and reporting with on-demand reports
  • Data capture or collection from sensors
  • Calculation engine: Pricing, medical claim adjudication, insurance policy rating
  • Are there specific penalties regarding stability and availability, and performance?
  • Are there stringent time based deadlines that must be enforced?


  1. Have your historical data ready for the business discussion, be able to tell them how much it costs (real dollars) to achieve each level of performance; sub-second vs. two second response times.
  2. Understand how the application is categorized and how that influences the performance goals. The industry you are in will make a big difference in the performance goals.

Critical business transactions

First, you and the business must understand the user response time satisfaction index; there are four levels of the index, satisfied, tolerating, frustrated and abandonment. Some transactions will be fine with 3 second response time, other will need 1 second.

Then, what are the critical business transactions?

  • Define the response time for a satisfied user experience for those key business transactions.
  • Define the response time for a frustrated user experience for those key business transactions.

For message-based systems; what are the throughput requirements?  How many prices must be calculated per second, how many claims must be adjudicated per second?

Here are some basic questions that every architect/developer/ performance engineer must be able to answer for the application:

  • Who uses your application?
  • How many people use the application?
  • What do they use it for?
  • Why do they use it?
  • When do they use your application?
  • From where do they access your application?

Define the conditions that these performance goals need to operate in.  This is the workload discussion.  What is the average number of users, and what is the peak number of concurrent users?

If you have 500,000 registered users, how many access the application?

What is the business’ tolerance for failure? At what point does a degraded transaction response time become unacceptable and hurtful to the business?  A transaction, such as order status, may have a response time of two seconds under normal load.  How does is slow down under heavy load? It may go to 4/6/8 seconds.  It may just not respond.  The architect must decide when to turn users away due to load. You want to make sure that users in the application continue to process under load and not allow new users into to the system until they can be processed safely.

You must have a scalability plan defined for peak load conditions.

Take-away: requirements phase

1)      List all the key transactions and be specific with the transactions and the satisfied response times.  For instance, the Order Status transaction is satisfied at 2 seconds for the 90th percentile under normal load. Under peak load, tolerated response time is 3 seconds for the 90th percentile.

2)      Have a clearly defined scalability plan for flash events and for normal business growth for online transactions and calculation engines.  For instance, the adjudication engine under normal load must process 20 claims per second. At month end peak, it must process 40 claims per second.

3)      Communicate the performance goals clearly to the next phase of your SDLC. The architect team must be fully aware of these performance and scalability requirements. The goals will be validated in the performance testing process.

4)   Each component of the transaction must have a performance target. The client component, browser or app based, needs response time targets.  Once the HTTP request returns to the browser, the browser must render the screen in 500 milliseconds. The Architect is repsonsible to communicate the performance budget to each developer for each component.

In October this year the SEC sponsored a technology roundtable to discuss how to promote stability in today’s markets. This is after a year with headline grabbing stock market failures;

NASDAQ market

Disrupted trading in the markets

In March the BATS Exchange IPO was halted and the BATS exchange stopped trading for hours, because the IPO price of BATS went from $16 to zero in minutes. In May the Facebook IPO on the NASDAQ had issues due to extreme volume coupled with the effects of high frequency trading. The rate of order change and cancellations prevented the exchange from establishing a price for the IPO.  The orders changed that fast, sub five milliseconds. Then Knight Capital suffered from a software issue.  They installed new software to work with the NYSE’s new Retail Liquidity provider program. The software sent buy orders out  from the market open and continued unnoticed for 30 minutes. Knight Capital was not aware of the impact it was having to the marketplace.

In between all this, you can find many examples of micro-flash crashes of single stocks. These can be seen by visiting the web site of Nanex (

The SEC Roundtable review

The morning session lasted 2 ½ hours. The roundtable was kicked-off with by the SEC Chairman Mary Schapiro. The success of the market is tied to the technology and when it fails, the consequences are extreme. These events continue to erode confidence in the market.  The industry needs to address the high volume of cancellations.  Also, there are more basic technology 101 issues occurred during the two IPO cases and the Knight Capital case. We need to balance the need for rapid innovation and competition with the proper and diligent testing methods.

My key take-a-ways from the morning session of the roundtable;

  • There is a need for firms to have a better understanding of their impact to the overall market. This would involve using Drop-Copies, where the exchanges send real-time trading records back to the broker/dealers, so they would understand their order flow. The broker/dealer could then run real-time reports to check their orders.
  • Improved testing strategies within the firms and an elevation of the Software Quality and performance profession. While QA people are independent from the development teams, they must be integrated into the development teams. The QA role in the firms must change to attract the best and the brightest to the role, including functional testing, as well as performance testing.
  • Testing in production. This is always a controversial topic in any industry. The firms and exchanges would agree on a number of testing Symbols for use when testing new features. This would require significant cooperation across the marketplace.
  • A focus on internal software testing for stability, performance, and scalability. Introduce earlier involvement in the SDLC with software quality resources. The benefit that an outside organization can bring on the processes and test cases could be helpful.  However, the roundtable participants discussed how difficult it is to being new people into the teams, due to the complex and technical nature of their systems.
  • Orderflow kill switches for the firms and exchanges. The exchanges would provide this capability and allow each firm to set its own parameters or limits that would trigger the kill switch. This would allow brokers to manage specific order types, control the size or value of the order, limit prices, or restrict the stock they trade in.  This came out of a working group that was established after the new meltdown Knight Capital.

The issues this year occurred in the exchanges and the broker/dealers, no one was immune from disruption. The key items that caused disruption were related to new or changed software and large volumes of orders.  An Enterprise wide software performance engineering strategy will help mitigate these software issues, both for the brokerages and exchanges.  The Market is facing a significant challenge with these issues.  The need to innovate and introduce better features before the competition does,  in a very complex and interconnected marketplace, with the need for rigor and increased testing. In addition production monitoring (really marketplace monitoring) is a critical component.  The devil is in the details and the compensation models.

The Software Performance Engineering Body of Knowledge

The overview

The Software Performance Engineering Body of Knowledge is a collection of key knowledge areas, supported by underlaying competencies for delivering in those knowledge areas, and includes a set of techniques for each knowledge area. The BoK also addresses the career path of the performance engineer for formalizing the profession.  Most people who describe themselves as a performance engineer arrive in a familiar pattern.  They are technical problem solvers, who througout their career ended up on the problem projects. Asked to help on production systems that are having user experience problems (too slow), instability problems (stops working), and missing the batch window.

The goal of the BoK is to clarify roles and responsibilties. Many people interchange performance engineer and performance testing. They are not aware of the differences in skillsets. Some of the roles or titles found;

  • Performance architect
  • Performance engineer
  • Capacity planner
  • Performance analyst
  • Automation technician
  • Troubleshooter or problem solver

Within many organizations the actual title of Performance Engineer does not exist in the Human Resources (HR) talent management process.  They try to fit them into developer, architect, or quality assurance roles.The career of the performance engineer is not managed.  People are left on thier own to figure out how to improve or move up the ladder.  Finding and growing software performance engineers is a challange for every organization. By not having a formal role description in the company, each division or group within the company determine there own description. One group calls them performace testers, another Sr. developers, or performance engineer. The it takes to develop one can vary greatly, with the right experience and guidance, you can be a very good performance engineer in ten years. This begins the discusson on your skillset pyramid, growing the base (technical competency) and  the height (communication, management, leader) to acheive a Sr. Performance Engineer role.

Software Performance modeling – Part I
This is a complex topic. Where do you start? A complete approach must start with the current production business volumes. I will review the steps involved in the modeling effort. I will also cover the types of models, and please do not use a linear model for a complex system.

Know your workload profile
If you have a web site that is frequented by customers and they are transacting business, you must know how many customer and how often they visit the web site. What are they doing on your web site? What are the key business transactions they are using? Beyond the login, are they reviewing orders, searching for new products? If you have an online retail web site you must be fully aware of the workload profile your customer are putting on your systems. If you are providing healthcare benefits or prescription benefits to a large number of clients and members, you must understand the business volumes for each prior year, during the open season when the members and beneficiaries visited the member portal.
The business and IT must together estimate the future workload profile. Will there be a steady increase in volume each quarter, say 20% growth? Is there a new client coming onboard with one million new members?  How will this business growth impact the existing systems?

Know your system utilization
Each workload profile consumes resources of the system, and when I mention system, I mean a comples Enterprise system. When a user or member logins into the member portal, or brokerage account, they execute business transactions. If a member reviews their beneficiary, or the retail trader reviews their account positions, those business transactions turn into system transactions; the proverbial rubber hitting the road.
The system transactions span across all the components; web servers, application servers, middleware, CICS gateways, databases.  These transactions consume memory, CPU, network, and disks. The business workload can be measured in transactions per seconds (minutes).
The system must be monitored, the end user experience must be monitored. There are a number of tools in the market to help collect this information. These tool are in the Application Performance Monitoring market or APM.  These tools are critical to help measure resource utilization in complex systems. The information they collect are critical to the forecasting and capacity planning disciplines.
The next post will be a review of a simple business model mapping the system transactions.