Where does a Software Performance Engineering team fit into the Enterprise?
Shared services and business units
Many Performance engineering teams end up somewhere, rather than placed somewhere. There tends not to be much in the way of preplanning or discussion with the Sr. IT management team. In my experience teams have been in shared services organizations such as;
- Quality Assurance,
- Testing Center,
where they have to support many different applications and provide difference services across the Enterprise. Often times, the business units do not know how to engage with a shared services team and the PE team does not know how to engage the business.
When placed in a Business Unit, this typically means the business is aware the value that a PE team brings. The team has a more Sr. leader who is technical, part project manager, and can interact with the business to ensure the team is aligned. The PE team will also be involved in more technical and product evaluations. Each organization influences the goals and purpose of the PE team. There must be an Enterprise Performance engineer path.
Care and feeding of the core PE team
The high value PE team usually has multi-disciplined people, people with a wider technical skillset that most other team within IT. A low value performance testing team typically has a narrow set of skillsets, as their mission is to execute performance tests, where they do not understand the system under test very well. A critical part of managing a PE and a PT team is to make sure there is a well defined career path that helps shape the more junior person and helps to extend the technical and non-technical skillset of the more senior people. They should also growth their understanding of the business.
The EA team is involved in many of the Enterprises large scale projects and they usually have a handle on the risks of the large projects. The EA team is involved early in the SDLC, they are technical team that understands the business. When a PE team is placed here, it can have Enterprise wide visibility into the key projects, this allows the team manager to insert the team into the high risk projects and to see the planned projects.
The Director of PE must be a direct report to the leader of the EA team. The goals of the typical EA team are very much in alignment with the goals of the PE team, the EA mission is to translate business requirements to a technical solution for business value. The PE mission is to help manage risk; the risk the applications will not meet the user experience, the risk the applications will not achieve the business goals of scalability, and reliability.
The risk here for the PE organization is the team does not stay involved for the development and testing process. The PE career path here can be an issue, as the way to promotion is to be an EA. The EA are involved early and once the design is approved, they are not involved in the detailed development. The PE team must be involved in the full SDLC.
Most QA organizations are focused on the functional requirements of the application. They are involved later in the SDLC, however, there is a trend to involve them in the requirements phase to define the tests cases at that time. When a PE team is placed here, it tends to be very testing focused. The project development teams involve the PE team when getting ready to run performance tests. Not during the design and development phases.
The QA team is very focused on the business needs, however they tend not to be technical. There very few software developers in the QA team. A high value performance engineering team is technical and business aware. This lack of technical focus within the group will impact the PE team and the PE career path. The QA career path is different from the PE path. QA can be a manual process in many organization and lack the drive to automate testing. The PE team needs automation to be successful, as technology changes.
There is a risk here is not enough focus on the technical skillset, too much focus on testing and not enough on design and development, where does the career path lie?
The goal of Operations is to keep the production applications running smoothly and performing well, where all is well and the customers are happy. The Ops team is the end of the line for the application and they often feel the pain of poorly performing applications, where they have to make it perform. Often times the development teams do not make the application will performance well, unfortunaltey, response time, scalability, utilization of system resources is the problem of Ops. They were often not involved at all during design and development. This is a technical team, often times with a more limited understanding of the business needs.
The operations team put in place monitoring and measurement processes and tools for the applications. The performance team placed in this group usually has a capacity planning and troubleshooting focus. This is because when the application is moved into production, historically it may not have performed well, and the Ops team had to fix it. They monitor the resource consumption of the production system, they understand the workload of the applications and can determine when it changes.
The performance team in this group will be involved in predicting the expected capacity of the new or modified applications, they evaluate the workload, review performance test results and determine if additional computing resources are required. They will have access to the production workload and gain a good understanding on how the application is used. This information is critical to designing a high value performance test plan.
The risk for the PE team in this group is a reactive focus, and late involvement in the SDLC where they are not aware early enough on the changes to the applications. They may not be involved in the performance testing of the application and have limited access to the results of the testing. The career path is at risk here as well.
Large business units set their priorities and can have large critical development programs that span years. The IT and Business leadership team have bought into the value that PE brings and how it mitigates risks for the business. These large programs will also have many different Releases, each with added business functionality and complexity. The development team is also part of the business unit. The business unit may set up the PE team as a shared resource within the BU, to be leveraged across the critical business applications.
When a PE team is placed within the business unit it is viewed as a critical success factor in each major release. The development process in this situation often has PE activities embedded in the SDLC; non-functional requirements are defined, design reviews, code reviews and unit testing for performance, performance testing, and production monitoring and measuring. This is a more integrated team. The PE team is very in tune with the business goals and priorities. The PE team leads the PT team as well, there are well defined performance test scenarios and the expected outcome is well known.
The risk in this cases is minimal, however the career path will be maintained outside the program. There will still be the need for a Enterprise Performance Engineering leader who can manage training and career paths.
Scattered Performance team
In this case, a performance engineering team does not exist. There are key people scattered across the organization who can take on the role of PE when needed. They can be in all the organizations mentioned previously. Often times they are brought together (summoned) when there is a critical performance problem. This structure may be good enough for the business, there may be few and far between production issues related to performance and scalability.
A performance testing center.
The business has pushed for a low cost performance testing team that is typically in a different geography or off-shore. The development team or the QA team will define the test scenarios and test cases to be executed. In this case, you have people who are not performance engineers defining performance test cases for a testing team that does not have a performance engineer. The remote performance testing team often has very little understanding of the application under test. The PT team produces basic performance reports from the testing tool.
In some cases, the PT team cannot execute a test due to a technical issue that they cannot solve. They end up waiting for the next day for the development team to solve it. The value in a PT comes from the ability to understand the application, overcome many technical issues and provide insightful test results. The testing center model requires more involvement from the development team, in some cases so much involvement, the development ends up running the test.