To tackle these challenges so that the SOA application is not DOA, IT needs two capabilities at their disposal. First, IT has to be able to monitor application performance as experienced by the real user because that is the only place where the performance of all of the constituent services is felt. Furthermore, if a service-level violation is detected, they have to be able to quickly trace the offending transaction and pinpoint the cause of the performance problem. Capitalizing on these capabilities in a systematic way allows IT to:

• Improve service levels by discovering performance bottlenecks and shortening the time to problem resolution.
• Lower the costs (and frustrations) of operation management by eliminating unnecessary triage meetings and fruitless problem recreation attempts.

How do existing end-user monitoring techniques fare in delivering the capabilities IT needs to avoid application DOA? Not well. Let’s take a quick survey of existing monitoring techniques as it applies to SOA performance management:

• Sniffers or other packet capture appliances can estimate the round-trip response time of a packet algorithmically, but lack the ability to measure the response time of transactions whose path does not pass through the point on the network where the appliance is installed. Take mashup applications as an example – critical data objects are supplied by third-party applications, potentially bypassing any sniffers installed in front of the Web servers. In the data center, the sniffers are also blind to Web Services calls among servers or third-party services that form the basis of an SOA application.
• Server monitoring tools can only report on the transaction response time of the infrastructural silo they are monitoring. For example, popular J2EE application server monitoring tools measure only the response time on transactions that involve the application server. Transactions served directly by the Web or third-party servers, which never touch the application tier, cannot be managed by the J2EE monitoring tool.
• Traditional Website performance monitoring services can detect whether an SOA application is available or not. It cannot report on performance as experienced by real users or provide actionable information that pinpoints the cause of problems to guide corrective effort.
• Pure-play SOA management products can help IT model the interdependencies among various services and provide limited transaction path information, but are often blind to the health of the infrastructure that supports the orchestration. More important, they have no visibility into the ultimate performance as experienced by the end user.

In terms of providing “real” actionable information for managing SOA performance, these legacy tools are deficient not only in the type of performance data they collect, but also in where the data is collected. It is critical to define application performance as a response time as perceived by the end user instead of server, network, J2EE, database, or other silo-oriented metrics. There is no argument against the fact that the experience of the end user is the only thing that matters. Moreover, for mashup applications where the Web page is served by multiple servers or third-party data centers, or when the application is delivered using content delivery networks, the application might not even come together until the content arrives at the browser. As a result, the only valid measurement of good or bad SOA application performance is the one measured directly at the real user’s browser.

To deliver real user monitoring and transaction tracing capabilities to avoid SOA going DOA, IT needs three integrated functions in their SOA performance management tools:

Detect: “You cannot manage what you cannot measure.” Having a quantitative way to determine whether the SOA application meets service level requirements is the first step in SOA management. In other words: “Is the right application response (data, page, action, etc.) delivered to the right user in the right amount of time?” There are numerous QA techniques to ensure that the right application response is delivered. Furthermore, most organizations have the necessary security to assure that the right person is receiving the information. But assuring that the information is delivered at the right time to the end user through the complex Web-based SOA infrastructure is another matter. Having the ability to non-intrusively monitor application performance as experienced by real users is an absolute necessity because it is the (1) only way to accurately detect problems experienced by real users of SOA applications for service-level assurance and reporting, and (2) it provides a key driver for making process or application response time improvements. The starting point of such monitoring is the end-user’s browser, where the application truly “comes together.” It is at the browser that IT can take into account “last mile” circumstances and identify whether an incident has occurred that will affect user satisfaction. Data collected by legacy tools that focus on monitoring a particular technology silo – like network routers, Apache Web servers, WebSphere application servers, or .NET Frameworks – cannot be extrapolated to determine what the actual end users of complex SOA applications are experiencing in the browser.

Isolate: Once the application performance as experienced by the end user is known, it has to be correlated with the performance profile of all the infrastructure and application components involved in the delivery of the SOA-based response. Since composite applications (1) are made up of services that are “black boxes” whose performance cannot be controlled or tuned by those orchestrating the application, (2) run on physical or virtual infrastructure components that are not entirely under the control of IT operations, and (3) may have different parts of a transaction served by different data centers or servers including third-party service providers, it is important that the performance of each transaction is reported and correlated across all infrastructure tiers, third-party data centers, and application components. Performance correlation can be achieved by painstaking log file analysis and heuristics to match up IP addresses and request times across various tiers, but this methodology is error-prone and difficult even if access to all of the logging information is available, and impossible if the transaction touches a tier outside the data center where log files are unattainable. Another simpler mechanism is to tag each transaction originating at the end-users’ browser non-intrusively and dynamically trace it through the entire infrastructure, logging appropriate performance data at each tier. Such an end-to-end view of performance based on the real user’s experience offers the bird’s eye view needed to pinpoint incidents, errors, bugs, or bottlenecks that impact end-user response time.

Optimize: A holistic browser-to-database view of transaction performance provides actionable information so that ad hoc or trial-and-error approaches are no longer needed to identify and respond to performance problems. Without actionable information, IT incident response teams will likely spend more time debating the cause and attempting to re-create the problem than implementing a fix and restoring the business function. By analyzing correlated transaction performance information over time, IT can identify leading indicators of performance concerns so they can proactively resolve them before an incident impacts user satisfaction or business productivity. Furthermore, the information also helps identify areas for performance improvement in the infrastructure, services, and application.

Having these three functions integrated into a single SOA performance management tool gives IT an early response system to detect and react to end-user performance problems before they impact thousands of users or lead to costly site outages. Information on business impact or performance bottlenecks should be fed back to the operations staff for infrastructure or process improvement and to the developers for application optimization.

Yes, SOA can greatly enhance business agility and lower cost of application development. However, without a real user-oriented approach to manage SOA deployment and production management systematically, it is highly likely that the SOA application will be DOA.