[By Andreas Grabner, Dynatrace]

What happens if you lock a handful of seasoned performance engineers into a Scottish castle and let them discuss the future of performance engineering? Magic happens!

In all seriousness. Thanks to Neotys for hosting the first Performance Advisory Council – simply called PAC. I believe I got invited because I have been part of the global performance engineering community for the last 15+ years and my friends at Neotys probably thought I have my own thoughts to share on how performance engineering is evolving.

In the two days during PAC it became clear to me that the larger community understands that “Winter is coming”. And while I mean this literally (I saw snow last week) I also mean that a significant shift and disruption is happening in our profession. Performance Engineering in 2020 will not be the same as it is now – and it is because of several “buzzwords” floating around when talking DevOps Transformation which will also impact performance engineering. Wilson Mar gave a great overview at PAC where he touched on several new disciplines during his presentation:

• Automation: Not only test execution but also root cause analysis
• Shift-Left: Provide automated performance feedback earlier in the pipeline
• Shift-Right: Influence and leverage production monitoring
• Self-Service: Don’t become the bottleneck! Provide your expertise as self-service
• End User Monitoring: That’s the bottom line! Happy Users make Happy Business!
• Cloud-Scale: Static monitoring and planning is not IT works in 2017!
• Artificial Intelligence: Leverage machine learning and big data for better and faster performance advice!

A key capability and skill that was discussed by almost every presenter was Application Performance Management (APM). There are several aspects where APM supports future performance engineers:

• Code Level Root Cause Analysis and Tuning while running Performance Tests
• Leverage APM Production Data for creating more realistic test workloads
• Testing your production APM installation as part of your load tests
• Automate Performance Metrics into your CI/CD Pipeline
• Provide Performance Engineering Services based on live APM production data

Working for Dynatrace, a vendor in the APM space, I got to talk about how Dynatrace evolved our monitoring capabilities to support the “Future Performance Engineering Discipline”. While most of you might be aware of what APM used to be when we talk about our AppMon product (or tools like AppDynamics, NewRelic, Wily …) many have not yet been exposed to our new AI-powered Dynatrace Fullstack Monitoring platform powered by Dynatrace OneAgent. Wow – that’s a lot of buzzwords in the last sentence alone

To cut through the buzzword cloud I decided to share the internals of what Dynatrace built, what the whole AI (Artificial Intelligence) is really all about and why we believe our new approach of capturing, analyzing and providing access to performance data is the best approach to support the future requirements.

Dynatrace AI Demystified

I started my presentation with the statement: “I will explain what we built and how we analyze the data we have. It is up to you in the end to decide whether you want to call it AI or not. We strongly believe though that we built something different, something that will help you succeed with the new types of applications we are dealing with!”.

I structured my talk into 3 sections:

1. OneAgent: Why and what we built
2. Insights into the AI
3. AI in Action

Let’s dig into it …

#1 – OneAgent: Why and What we Built

With our AppMon product we learned that it is important to have visibility to 100% of all transactions within individual applications. What we missed were 100% visibility into the Full Stack, a good SaaS and On-premise offering as well as an easy fully automated way to deploy our agents regardless OS and Application Stack!

Our OneAgent solved all these problems: One Agent to monitor them all

1. A single installer for Windows, Linux, AIX and Solaris
2. Automatically monitors your host, all processes, services and log files
3. Automatically monitors all network connections and therefore understands all dependencies
4. Automatically injects into your Application Stack Runtimes to provide code-level visibility
5. Automatically provides end user monitoring for all your web and mobile applications

The following shows SmartScape, the visualization of all detected and monitored entities (data centers, hosts, processes, services and applications) and all its horizontal and vertical dependencies:

SmartScape visualizes all dependencies between all automatically monitored entities based on OneAgent data capturing

End-to-End Tracing and Code-Level Visibility are capabilities that most modern APM tools provide. In the case of Dynatrace we call it PurePath – the “pure” end-to-end execution path for a single transaction through your application stack:

End-to-End Code-Level Tracing is a mandatory capability in modern APM solutions. Dynatrace calls it PurePath!

Another key capability of any APM and Monitoring Tool is capturing time series data, automatic baselining and anomaly detection. At Dynatrace we do this on infrastructure as well as application and service level metrics, e.g: Response Time of any REST Endpoint exposed by a hosted service – or – CPU on a docker container:

Automatic Timeseries monitoring, baselining and anomaly detection is another key APM capability

There is much more that OneAgent captures, e.g: Deployment and Configuration Changes, Log Messages or Metrics from your Cloud (AWS, Azure, Google, VMWare…), Container (Kubernetes, Docker, …) or PaaS (OpenShift, CloudFoundry, …).

The problem with all this data is that we do not solve the fundamental problem: Is there a problem in our system, where are the hotspots, who is impacted and what can we do to fix the problem or make the application go faster.

Our solution to this problem is what we call Dynatrace AI. And here is what the AI really does:

#2 – Insights into the AI

The AI is first and foremost powered by very accurate Fullstack data captured by OneAgent. Additionally, you can feed more data to Dynatrace through a REST API or our Plugin technology.

The next key element is smart anomaly detection on each metric we see. For certain metrics, such as real user monitoring metrics, we apply hypercube baselining algorithms to detect whether there is an anomaly for certain Geos, Browsers, Features, OS, Bandwidths, …

Dynatrace AI leverages Hybercube baselining to detect metric anomalies across multiple dimensions

An anomaly detected based on a baseline violation will generate an event. But events are also generated for things such as process crashed, disk full, connection dropped, database unavailable, GC Time High, High Network Latency, … all these events are automatically analyzed based on the relationship the entities have that experience these events. Remember Smartscape? That’s how Dynatrace knows all the dependencies between processes, hosts, services and applications. This allows us to look at individual events, correlate based on dependencies, group them and then rank them based on impact and expert knowledge. This approach allows us to factor in causation as we do not simply rely on correlation of events or timeseries data. That’s one magic ingredient of the Dynatrace AI:

Dynatrace sees many events, understands the dependencies and is then able to identify real problems based on causation and not just plain correlation

In my presentation, I went through additional details on how we do each individual step that is shown in the illustration above. One aspect I want to highlight is that we do capture every interaction of every user with your applications – and we capture this end-to-end thanks to PurePath.

Having all this information allows us to also do multidimensional-transactional correlation to highlight problems that really impact your end users. The following illustration should make that clearer (if you miss the animations go ahead and watch the recorded session):

Every PurePath Matters: 100% capturing allows us to do real multidimensional transactional correlation

When Dynatrace detects a problem it is now able to tell us who is actually impacted (end users or service calls), what is the root cause (full disk, slow code, bade deployment …) and how did the problem actually evolve over time (looking at all these events). In case the AI detects a problem we open up a so-called Problem Ticket that contains all this relevant information:

Dynatrace AI automates problem detection by highlighting impact, root cause and problem evolution

I hope this gave you a bit more insight into what our Dynatrace AI is doing. Whether you want to call it AI or something else is up to you. We believe it is the right way forward when it comes to supporting performance engineers in a world where our applications are updated more frequently on a more dynamic infrastructure!

#3 – AI in Action

While the AI creates problem tickets that we visualize in the problem view we also give full access to the Problem Details via the Problem REST API. Dynatrace can also trigger external tools, e.g: PagerDuty, ServiceNow, Jenkins, Ansible, AWS Lambda … to execute incident management workflows or auto-remediation actions. The great thing about this is that you have all the data available that the AI pre- analyzed for you. Following is a quick animation showing the Problem Evolution screen which “replays” all analyzed events. Remember – all this information is also accessible via the REST API which allows you to leverage this data as problems happen to e.g: push them back to your engineers, change your load balancers, adapt your workload configuration or spin up additional resources to handle a shortage in resources.

Dynatrace AI gives full access to all observed and correlated problems. Either through the UI or REST Interface.

I hope this gave you some insights into what we built, which problems we address and why you as a performance engineer should embrace this new technology. If you happen to be a Neotys user you should also check out the YouTube Tutorial on “AI-Supported Performance Testing”. Henrik and I showed how to integrate Neoload with Dynatrace and how we can leverage the AI to do better, smarter, DevOpsy performance engineering.

Become a Performance Engineer for the Future

I want to thank Neotys for bringing us together. I hope that all existing and future performance engineers out there understand that performance engineering is no longer about wasting time creating and maintaining load testing scripts. It is not about creating lengthy load testing reports at the end of a release cycle that nobody looks at. It is about providing performance engineering as a self-service from dev all the way to ops embracing automation, integration into dev tools as well as not being afraid to look into your production environment.

All the best,

Andreas Grabner

Andreas Grabner has been a developer, tester, architect, and product evangelist for the past 18 years for CRM, eGovernment, testing and monitoring vendors. In his current role, he helps companies injecting metrics into the end-to-end delivery pipeline to make better decisions on feature and code changes and with that also closing the feedback loops between Ops, Biz and AppDev. Andreas has been speaking at meetups, user groups, and international conferences to share experiences and stories around architecture, agile transformation, automated testing, CI/CD and DevOps. In his spare time, you can find him on the salsa dance floors.

A Year for the Record Book

As the world prepares to turn the page on 2017, most will do so reflecting on a year full of reminders from Mother Nature of her strength and prowess. With natural disaster events all too common lately, we celebrate the charity, brotherhood, and perseverance on display across the globe. Rebuilding efforts have been a focal point, and for damage insurers like Promutuel Assurance, they are just a part of the process.

Controlling the Damage

Locally, the Insurance Bureau of Canada reports that two significant storm and flooding events in Ontario and Québec earlier in the year resulted in >$223 million in insured damage (source, Catastrophe Indices, and Quantification Inc). This type of volume can have a profound impact on the provider community, and for companies like Promutuel Insurance, a long-standing NeoLoad user, these events have a way of thrusting core values like cooperation and solidarity to the forefront.

Ranked 4th among damage insurers in Québec, Promutuel Insurance, alone, has over 160 years of history dedicated to protecting the local population’s assets in the true embodiment of “there when needed” service and support.

Application Performance Testing & Reliability a Must

When Promutuel Assurance’s IT team needs to evaluate “system” success, it recognizes the importance of the agent/customer relationship’s impact on the bottom line. It’s no wonder performance (availability, speed, and robustness), particularly for applications facilitating this agent/customer dialogue, is a primary objective. This is why load testing using NeoLoad has been well-established habit within the IT department for mission-critical applications for several years.

Guidewire Technology & NeoLoad Source of Growth

As part of the company-wide migration to Guidewire technology (a framework built on the use of Single-Page Applications (SPAs) Promutuel Assurance recently incorporated NeoLoad into the process. Referenced daily by over 400 agents, this system represents a super-competitive, high-growth/profitability market compared to other policy types.

With Guidewire and NeoLoad in their performance testing process, Promutuel Assurance has reduced business proposal processing time by several seconds; driving nearly 30% sales growth.

For the full story of how Promutuel Assurance is using Guidewire and NeoLoad, read the case study. To begin testing with NeoLoad today, click here.

Like any good customer-focused company, Neotys strives to provide a valuable service to its users. As part of our continued focus on delivering and supporting the most up-to-date and appropriate load testing and performance testing technology solutions, we are excited about WebSocket protocol. The following offers a great perspective on the technology, it’s beginning, and how one company has successfully integrated WebSocket testing into its larger load testing process. We thank INXPO for their support, partnership, and, particularly, Corey Miller, Web Developer, for his participation and co-authorship.

The Client-Server Handshake Starts with WebSocket

Introduced as computer communication protocol nearly ten years ago, WebSocket was intended to usher in native browser methods for pushing content from the server to client. Classically, browser operations are initiated by the client – the browser downloads a new web page only when the user clicks on a new link, for example.

However, some web applications work best when the server is capable of initiating contact with the client directly. Chat rooms and real-time games are two examples, but now nearly every modern app uses some form of push notification or real-time messaging to improve its user experience. Sadly, the HTTP protocol was never designed to accommodate such server-driven initiation, so developers created their own workarounds, including polling techniques and keeping expensive connections open for long periods of time. These workarounds often result in hard-to-scale apps and services or unreliable performance in production environments.

WebSocket: A Standard Approach to Two-way Communication

A WebSocket connection starts off just like a simple HTTP connection, but instead of exchanging a single, bidirectional payload, the connection upgrades itself with the WebSocket protocol to maintain an indefinite connection. This allows both server and client to hand-off payloads back and forth at any time. It is analogous to a phone call, during which either party can converse freely without hesitation or delay.

WebSocket provides a few advantages over HTTP:

• Bilateral communication. Enables messages to originate from either the client or the server.
• Bandwidth efficiency. HTTP requires header information for every message, thus increasing request size/response time. WebSocket, on the other hand, require header information for the connection handshake only. The subsequent message is 100% payload-based, which can significantly reduce total bandwidth used.
• Establishing a connection takes time. By preserving the connection, WebSocket payloads can travel to and from without repeated handshakes.

WebSocket can be a valuable time and speed champion, but what is it like to load test with it? Fortunately, we have firsthand accounts like that of INXPO, who can answer the question, and are happy to share.

WebSocket Promotes Competitiveness for INXPO

INXPO is a leading enterprise-serving video platform company. Years ago, after recognizing the impending death of Flash, INXPO chose WebSocket to backbone their next-generation video conferencing platform, INXPO Studio.

In a perfect world, virtual audiences would hear a speaker as if they were in the same room. In reality, both audience and speaker must deal with network latency (time between speaker delivery and recipient hearing). Synchronizing all webcast elements in order to present each viewer with a live experience requires multi-event timing complexity. WebSocket simplified this process significantly by allowing the server to actively push messages to clients rather than waiting for the client to poll for updates.

Corey Miller was part of the team who built the NeoLoad scripts used to load test INXPO Studio. He recently presented some of his experiences and best-practice considerations for WebSocket testing at one of our Ramp Up events, and I am excited to share his perspective. The following are Corey’s findings in his own words:

INXPO’s Load Testing with WebSocket

With the first version of our new webcasting product in testing, the team’s initial goal was to simply test whether the product worked. If we simulate a few hundred users via a NeoLoad script, then we wouldn’t need to tie up man-hours by having lots of employees logging on and off at the same time just so that we could test basic functionality. That done, it only took a few modifications to transform a user testing script into a load testing script to query the limits of our webcasting platform. Thus, we transformed from a company that load tested with typical HTTP scripts to one that could synchronize WebSocket communications within our scripts. Here are just a few of the many lessons we learned along the way:

1. Understanding your communications is the first step: Modeling a WebSocket application in NeoLoad requires a different approach vs. modeling simple HTTP requests. For example, after NeoLoad processes a recording, the WebSocket connection request hosts all push request messages (the order received), but provides no indication relative receipt time, whether synchronous or asynchronous and/or triggered. It is for us to synchronize WebSocket traffic with other, standard HTTP traffic.

It was critical for us to copy all WebSocket incoming/outgoing payloads into a separate tracking document (we used Excel) so we could review and analyze. If the WebSocket traffic is in clear text, this is fairly straightforward. However, if packets are condensed or encrypted, it can be very challenging.

Because we developed both our product platform and load testing scripts in-house, accessing the product developers for clarification as to what various packets meant was easy, which saved a lot of script build time.

2. Build the script around the WebSocket Channel: The central component of our WebSocket script is the WebSocket Channel transaction. The rest of the script essentially amounts to HTTP request logging, the passing of information from the server to client in order for them to produce the WebSocket request.

Configuring the WebSocket Channel transaction was surprisingly simple using NeoLoad. Inside the Request headers tab of the Advanced Parameters window, we formatted the channel to mirror what we had on our system. Of note, some of the parameters, such as Sec-WebSocket-Key, are automatically configured by NeoLoad during path execution.

3. Understand how to route packets to the actions that can handle them: Attached to each WebSocket Channel action are a set of Push Message requests. If you have not used these requests before, think of them as a combination of NeoLoad’s Switch and Fork actions.

When a WebSocket packet comes down from the server, NeoLoad uses the conditions defined within the various push messages to determine which transaction will handle the message assignment. It then creates a new thread to traverse the tree contained within that push message in the same manner as it would a fork action. If the end of the tree is reached, the thread simply expires. The end result – a WebSocket Channel action combined with a set of Push Messages (acts like operator – receives the message, determines the right recipient and delivers the message (contents of Push Request)).

4. Modeling WebSocket requires the use of threading: Up to this point, I haven’t touched upon the main thread which still needs to be addressed. Once the main thread creates the WebSocket connection via a WebSocket Channel transaction, it simply proceeds to the next transaction, exactly as it would if it encountered a Switch transaction in which none of the cases matched. If allowed to proceed to the End container, the user path will simply end and the virtual user will cease to operate regardless of any actions it may be undertaking. It is critical to construct a mechanism to prevent the main thread from completing until you want it to.

There are several viable approaches. Our preferred method is to use a While action with a Delay or Wait Until action inside so that the loop iterates every ten seconds. We can easily add actions within the loop to debug or simulate user behavior during the eventual webcast. A good example of this use case would be to have users send chats out at a particular rate. To accomplish this, we need only add a chat transaction with a “percent chance” of execution within our loop.

5. The payoff is finding problems before your users do: Once the script was built, it was time to test. As the number of virtual users climbed into the tens of thousands, it revealed all sorts of issues we were able to address before they could impact our clients.

One notable finding was that it did not take long before chats started to hamstring us. To combat this, we created chat box scripts simulating user chatter, and as they ramped, other areas of our webcasts experienced interruptions. As a result, we re-wrote some of the message handling procedures to prioritize operational packets over less-critical packets like chats.

Other limit thresholds were hit due to sheer message volume. Again, we first encountered this as we ramped up chatbots. For any number of chats, a number of attendees, and the average size of chats, the formula for chat size per second is given as Size*(Count/Sec)*Audience. This helped us determine when we would encounter certain limits.

If a thousand users each chat every five seconds in a chatroom with 10K users (a total rate of 200 chats/s), and chat packages average 500 bytes, that would mean our servers were uploading 500B*200 chats/s*10,000 listeners ≈ 1 GB/s. Realizing this could not scale with multiple active webcasts, we implemented several safeguards to prevent the risk. This included compressing our JSON packages significantly to minimize the overhead in each packet.

What Did INXPO Takeaway from their WebSocket Testing with NeoLoad?

• “Load testing taught us to examine all of our communications more critically to help determine which payloads absolutely had to travel via WebSocket for performance reasons vs. what could use HTTP requests (many of which routed through CDNs).”
• “… load testing results made it obvious which parts of our infrastructure needed expansion in order to enable product experience across a larger INXPO audience.”
• “WebSocket has also proven to be a boon for us, but one requiring hard testing/learning to implement.”

After putting lessons learned into practice, WebSocket testing has helped INXPO create a world-class webcasting platform – success that has fueled growth and challenge. With NeoLoad, they have been able to identify and address before ever becoming an issue for their customers.

[By Jonathon Wright, Digital-Assured]

In the digital economy, every business is a software business, and how we create software is forever changing. We now need to challenge the business value of lengthy think-make-check cycles can deliver in an increasingly competitive, consumer-centric marketplace. Merely compressing these cycles by adopting iterative or agile methods does not fully address the challenges faced in terms of innovation, speed and quality. Technology is no longer the only enabler, we must continuously learn and evolve. The approaches and techniques that worked yesterday, may no longer be the optimum approach for tomorrow.

Figure 1 – Cognitive Adaptive Technology – Digital Quality (ISBN 0692885994), Chapter 2 – Digital Experiences (DX)

Historically, the human process of creating software has been based on linear, sequential activities underpinned by engineering best practice. But now the real challenge is to confront heightened end user (UX) and digital experiences (DX) which are shifting faster than ever before. The user expectation to release new capabilities and features within a matter of days or even hours, has put real pressure on the industry‘s fragile methodologies of which tend to disguise linear activities within an iterative model, combined with simple continuous delivery vessels.

“Digitalization of processes, behaviors, experiences and interactions, will result in continuously, adaptive, innovative, disruptive, business models.”  – Digital-Assured.com

Therefore we must adopt a new multi-modal adaptive approach that supports accelerated, distributed enterprise collaboration, coupled with communication enabling rapid feedback loops. Whether you label this as Digital, understanding the journey of the consumer is an essential part for the survival of every business in the future.

Figure 2 – DesignOps: The Digital Manifesto, https://leanpub.com/digital/

The simple model above is the new delivery paradigm; from ideation to design, experimentation to hypothesis through to benefit realization. By embracing Digital Engineering practices (artificial intelligence (AI), analytics, autonomics and machine learning) to take a value-driven delivery approach to the digitalization of processes, behaviors, experiences and interactions, will result in continuously, adaptive, innovative, disruptive, business models.”

Figure 3 – ‘Solution Thinking’, Ray Arell, Cognitive Adaptive Adoption webcast series, nuCognitive

Businesses must learn to pivot by utilizing solution thinking techniques such as heuristics to aid problem solving, help simplify decision making, based on the cognitive learning and the discovery of abiotic solution components.

Figure 4 – Cognitive Adaptive Adoption – Multi-Modal Delivery Modals.

Adoption of these techniques require governance (archology) the goal being to break down chaotic, complex and complex modern day solutions landscapes (archaeology). For some customers, these landscapes equate to hundreds if not thousands of upstream and downstream applications, endpoints or nodes (organisms) within ecosystems of ecosystems.

Figure 5 – Cognitive Adaptive Learning – “Complexity-Informed Organizational Change”, Ray Arell, nuCognitive, 9th September 2016,  adapted from the Cynefin framework by Cognitive Edge. CC BY-SA 3.0

The emergence of cognitive adoption is driven by trends towards omni-channel content delivery, utilization of big data and an improved customer experience. Digital business systems need to be extremely responsive to customer sentiment, work across a variety of devices, be resilient in the face of unpredictable failure modes, and process huge amounts of unstructured data.

Figure 6 – Cognitive Adaptive Intelligence – Power of Analytics & Autonomics – Descriptive, Predictive and Prescriptive

Here is my talk on ‘Cognitive Learning – Evolution, over Revolution’ on November 15^th to explore how artificial intelligence (AI), analytics, autonomics and machine learning can help you become part of the next generation of insight-driven businesses.

Jonathon Wright

Jonathon is a strategic thought leader and distinguished technology evangelist. He specializes in emerging technologies, innovation and automation, and has more than 18 years of international commercial experience within global organizations. He is currently the co-founder of Digital-Assured based in Oxford in the UK.

Jonathon combines his extensive practical experience and leadership with insights into real-world applications of the core principles underpinning Performance Engineering and Enterprise Digital (Smart Cities, IoT and AI). Thus, he is frequently in demand as a speaker at international conferences such as TED, Gartner, Oracle, EuroSTAR, STAREast and STARWest. Jonathon is the author of several award-winning books.

About the first Neotys Performance Advisory Council #NeotysPAC

Neotys organized its first Performance Advisory Council in Scotland, the 14th & 15th of November 2017.
With 15 Load Testing experts from several countries (UK, France, New-Zeland, Germany, USA, Australia, India…) we explored several theme around Load Testing such as DevOps, Shift Right, AI etc. By discussing around their experience, the methods they used, their data analysis and their interpretation, we created a lot of high-value added content that you can use to discover what will be the future of Load Testing.

You can read the Jonathon Wright’s presentation here.
You want to know more about this event?

Today’s post is part one of a four-part series, and covers the following topics:

• Part 1 – Introduction to Performance Testing
• Part 2 – Establishing a Performance Testing Strategy
• Part 3 – Modeling Performance Tests 
• Part 4 – Executing a Performance Test

Introduction to Performance Testing Overview

Applications are being released to the market with increased complexity every day and done so using shorter development cycles. This requires an embrace of Agile development and testing methodologies. Performance, as part of the global user experience, is now the primary determinant of application quality.

“Old school” sequential projects marked by static Qualification/Implementation/Test phases, which push performance testing until the end of the lifecycle, are sure to experience a performance risk along the way. This proposition is no longer acceptable by today’s application quality standards. This blog series will provide practical information on how to execute efficient performance testing in this new and more demanding environment.

The State of Complexity in Modern Apps

One of the main drivers behind this shift to modern load testing is the growing complexity of the IT landscape:

• Most users are using mobile, thin clients, tablets, and other devices to access information.
• Complex architecture(s), simultaneously shared by several applications, is being built.
• New technologies offer a range of solutions (AJAX framework, Rich Internet Application (RIA), WebSocket, and more) aimed at improving the applications’ user experience.

Historically, applications have been tested to validate quality across several areas: functional, performance, security, etc. Testing phases answer to user requirements and business risks. However, the dialogue has changed; the conversation is no longer about quality, it’s about user experience. The user experience is a combination of look-and-feel, stability, security, and performance.

Performance: Imperative to a Successful User Experience

Performance is the factor in the success of the user experience. This is due to advances in technology, architectural complexity, and the locations/networks of the users. Load testing was a nice addition to the development process but has quickly become an essential testing step. Load and performance testing answers the following questions:

• Is the application capable of handling a certain number of simultaneous users?
• Is the average response time for pages acceptable under this set load?
• Does the application revert to normal behavior after a load peak?
• How many users can the application handle while maintaining acceptable response times?
• What is the load threshold above which the server(s) begin(s) to generate errors and/or refuse connections?
• Do(es) the server(s) remain functional under high load or does it crash?

Like any testing activity, performance testing requires proper methods and logic.

When an application passes performance testing but then fails in production, it is often due to unrealistic testing. In these cases, it’s an easy but misguided blame to point at the testing itself or the tools used to execute it. The typical problem resides with test design, especially when done without the correct basis. It’s necessary to ask, “what did we need to know which, if we had the information ahead of time, would have allowed us to predict this failure prior to production?” In other words, “how can we deliver and execute an efficient performance testing process?”

Phases of a Load Testing Project

Methodologies like Agile and DevOps allow for application creation to quickly address customers’ needs. The practices of updating project organization require closer collaboration between teams. Within these methodologies, the project lifecycle is broken out into several sprints, each responsible for delivering a part of the application. In this environment, the performance testing process should follow a specific workflow (as noted at the top of the post).

A performance testing strategy should be implemented at an early stage of the project lifecycle. The first step: Performance Qualification. This defines the testing effort of the whole project. An “Old School” approach to performance testing would force the project to wait for an assembled application before performance validation would begin. In a modern project life cycle, the only way to include performance validation in an early stage is to test individual components after each build and implement end-to-end performance testing once the application is assembled.

Learn More

Discover more load testing and performance testing content on the Neotys Resources pages, or download the latest version of NeoLoad and start testing today.

In part two of this four-part series focused on putting performance testing to work for you, we will explore “how best to establish your performance testing strategy.” 

Establishing a performance testing strategy is the first and most important step of performance testing. It helps you define:

• The Performance Testing Scope
• The Load Policy
• The Service Level Agreements (SLAs) and/or Service Level Objectives (SLOs)

It’s never possible to test everything, so conscious decisions about where to focus the depth and intensity of testing must be made. Typically, the most fruitful 10-15% of test scenarios uncover 75-90% of the significant problems.

Risk-Based Testing

Risk assessment provides a mechanism by which you can prioritize the test effort. It helps determine where to direct the most intense and deep test efforts, where to deliberately go lighter (to resource conserve for the more intense scenarios). Risk-based testing can identify significant problems more quickly/earlier in the process by helping focus on the testing of the riskiest aspects of a system.

For most systems, performance and robustness problems occur in these areas:

• Resource-intensive features
• Timing-critical/sensitive uses
• Probable bottlenecks (based on internal architecture/implementation)
• Customer/user impact, including visibility
• Previous defect history (observations noted across other similar systems during live operation)
• New/modified features and functionality
• Heavy demand (heavily used features)
• Complexity of feature set
• Exceptions
• Troublesome (e.g., poorly built/maintained) system components
• Platform maintenance

Here is a list of questions presented by industry-expert Ross Collard to help you identify different performance risks:

Situational View

• Which areas of the system operation, if they have inadequate performance, most impact the bottom line (revenues/profits)?
• What uses the system is likely to consume (resources per event) regardless of event occurrence frequency? Resource consumption should be significant for each event, not high in aggregate simply because the event happens frequently, and thus the total number of events is high.
• What areas of the system can be minimally tested for performance without imprudently increasing risk, in order to conserve the test resources for the areas which need heavy testing?

Systems View

• Which system uses are timing-critical/sensitive?
• Which uses are most popular (e.g., happen frequently)?
• Which uses are most conspicuous (e.g., have high-visibility)?
• What circumstances are likely to cause a heavy demand on the system from external users (e.g., remote visitors to a public website who are not internal employees)?
• Are there any notably complex functions in the system (e.g., exception handling)?
• Are there any areas in which new and immature technologies have been used, or unknown/untried methodologies?
• Are there any other background applications sharing the same infrastructure, and are they expected to interfere or compete significantly for system resources (e.g., shared servers)?

Intuition/Experience

• What can we learn from the behavior of the existing systems that are being replaced such as workloads/performance characteristics? How can we apply this information in the testing of the new system?
• What has been your prior experience with other similar situations? Which features, design styles, sub-systems, components or systems aspects typically have encountered performance problems? If you have no experience with other similar systems, skip this question.
• What combinations of the factors, which you identified by answering the previous questions, deserve a high-test priority? What activities are likely to happen concurrently, causing heavy load/stress on the system?
• Based on your understanding of the system architecture and support infrastructure, where are the likely bottlenecks?

Requirements View

• Under what circumstances is heavy internal demand likely (e.g., by the internal employees of a website)?
• What is the database archive policy? What is the ratio of data added/year?
• Does the system need to be available 7×24?
• Are there maintenance tasks running during business hours?

The answers to these questions will help identify:

• Areas that need to be tested
• Test types required to validate app performance

Component Testing

Once the functional areas required for performance testing have been identified, de-compose business steps into technical workflows that showcase technical components.

Why should business actions be split into components? Since the goal is to test the performance at an early stage, listing all important components will help to define a performance testing automation strategy. Once a component has been coded, it makes sense to test it separately to measure:

• Response time of the component
• Maximum calls/second that the component can handle

Moreover, component testing supports JMS, APIs, Services, Messages, etc. allowing scenarios to be easily created and maintained. Another major advantage of this strategy is that the components’ interfaces are less likely to be impacted by technical updates. Once a component scenario is created, it can be included in the build process, and feedback on the performance of the current build can be received.

After each sprint, it is necessary to test the assembled application by running realistic user tests (using several components). Even if the components have been tested, it is mandatory to measure:

• System behavior with several business processes running in parallel
• Real user response time
• Sizing/availability of the architecture
• Caching policy

The testing effort becomes more complex with the progression of the project timeline. At the beginning, the focus is on app quality, then concentrated on the target environment, architecture, and network. This means that performance testing objectives will vary depending on the project’s timeline.

Performance Testing Environment

It is imperative that the system being tested is properly configured and that the results obtained can be used for the production system. Environment and setup-related considerations should remain top-of-mind during test strategy development. Here are a few:

• What data is being used? Is it real production data, artificially generated data, or just select random records? Does the volume of data match production volume forecasts? If not, what is the difference?
• How are users defined? Are accounts set with the proper security rights for each virtual user, or will a single administrator ID be re-used?
• What are the differences between the production and the test environments? If the test system is just a subset of production, can the entire load or just a portion of that load be simulated?

It is important that the test environment is an exact replication of the production environment (some differences may still remain, which is acceptable). Even if tests are executed in the production environment with actual production data, it would only represent one moment in time. Other conditions/factors would need consideration as well.

Devise a Test Plan

The test plan is a document describing the performance strategy. It should include:

• Performance risk assessments highlighting performance requirements
• Performance modeling (logic explanation(s) calculating the different load policy framework)
• Translation of the primary user journey into components
• Description of all other user journeys with specific think time/business transaction metrics
• Dataset(s)
• SLA(s)
• Test description(s) being executed to validate performance
• Testing environments

The test plan is a key artifact of a well-designed and executed performance testing strategy, acting as evidence that a team has satisfactorily accounted for the critical role performance plays in the final end-user experience.

In many cases, project teams ensure the delivery of performance test plans as part of feature work during planning and development cycles by requiring them as “Definition of Ready” ingredients. Though each feature story or use case may not require the same level of performance testing, making the thought process a hard requirement for completion leads to better systems and an improved mindset over the end-to-end quality of what the team delivers.

Learn More

Discover more load testing and performance testing content on the Neotys Resources pages, or download the latest version of NeoLoad and start testing today.

In part three of this four-part series focused on putting performance testing to work for you, we will explore the “incorporation of modeling into your performance testing.” 

This post is part two of a four-part series collectively covering the following topics:

• Part 1 – Introduction to Performance Testing
• Part 2 – Establishing a Performance Testing Strategy
• Part 3 – Modeling Performance Tests
• Part 4 – Executing a Performance Test

[By Stephen Townshend, Assurity Consulting]

Continuous performance testing is the concept of building automated performance testing into an automated deployment pipeline. I see it being promoted everywhere from social media and conferences, through to clients specifically requesting it. However – does it really deliver?

In this blog I want to address five areas of concern.

The limited value of component testing

The most common way I’ve seen continuous performance testing implemented is to benchmark the performance of components as new code is deployed. Components could be anything from an API, an application function, or even a system within a system-of-systems.

However, component focused performance testing only addresses a small percentage of the performance risk:

At best, component focused performance testing allows us to track the response time and capacity of components in isolation relative to previous builds. It does not necessarily tell us about how the solution will perform in the real world, under real conditions.

Many performance issues occur not within the components of our solution, but with how they integrate together. In this situation, if we need to understand the performance of the entire solution before releasing it to our customers, then end to end integrated performance testing is the only way to mitigate that risk effectively. I have heard of component-only performance testing approaches which have ended disastrously.

So, given that we often need end to end performance testing anyway, and that the percentage of risk covered by component performance testing is low – is continually performance testing the components of our solution worth the effort?

Losing sight of the big picture

A side effect of fixating on components is that it becomes easy to lose sense of the big picture. There’s different facets to that:

• Losing sight of the overall solution we are testing
• Losing sight of the business purpose of the solution

The second point is particularly important. Once we lose connection with what is important to the business, we lose the ability to prioritise what we do based on what actually matters.

For example, I did some work for a new credit card business. They built a brand new green field solution from the ground up – and my job was to assess the performance risk, build a strategy, and implement it.

As I learnt about this solution I quickly became overwhelmed by the number of systems, systems within systems, integrations, network protocols… how was I ever going to performance test all of it? And then I took a step back and looked at the business purpose of the solution. In reality customers only do a few select things most of the time – apply for new cards, check their transaction history, and make purchases using their cards.

My approach was eventually to drive load throughout the solution from the outside in (simulating the core customer activity). This achieved two things:

• It was magnitudes simpler than trying to break up the solution into components and test each one
• Because of the realism the results were more valuable – we had a better understanding of what the customer experience would be in the real world

The caveat of this approach is you need great monitoring to at least track requests throughout the solution to understand where the time is being taken (as a minimum).

Test asset maintenance

Most performance testing requires test assets. Whenever the application under test changes there is a risk our test assets will break, or worse – the tests will still run but something else has changed which means our tests no longer represent reality (e.g. an additional AJAX request we are not requesting). It stands to reason that in a rapid delivery lifecycle there is more opportunity for this to occur due to the frequency of our releases.

Performance test assets also tend to be fragile because generally they involve simulating network traffic. Very small changes within a system can have a profound impact on the network traffic – even bouncing an application server can regenerate every dynamic ID on the pages of a web application.

This begs the question – how much manual effort is required to maintain our performance test assets within this rapid delivery model? Will we be able to keep up with the rate of change?

Many load testing tool vendors tell a pretty consistent story about how they are CI/CD or DevOps ready and promote continuous performance testing. I find this a very interesting claim. What does it mean to be CI/CD or DevOps ready?

Naturally there needs to be a command line interface or callable API’s to run tests and retrieve results in an automated way, but you also want some way to be able to re-apply correlation rules automatically (to reduce the manual effort between iterations). Most load testing tools come with some form of automated correlation engine, but manual effort is always required. For example, what about handling:

• Client-side encryption?
• Cookies being set by client-side JavaScript?
• Extracting the value from the middle of a re-direct chain?

On top of this, there is all the other stuff performance testers need to manually include such as:

• Think-time
• Pacing
• Parameterisation (datapools)
• Logic (loops, conditions)
• Structuring the project for ease of use and reporting

The fundamental issue with (almost) every tool on the market is that all of this useful logic we build into our test assets is tied to the recorded network traffic. Whenever we re-record the traffic, we lose everything except those automatic correlation rules.

My point is not to criticise load testing tools, but to point out the risk – that the manual intervention required to maintain load testing assets within a rapid delivery lifecycle is not scalable.

Automating a pass or fail outcome

To fit the ‘continuous’ model we ideally want a “pass” or a “fail” verdict at the end of our performance tests. The question is, how do we determine this?

The simplest way is to define NFRs or SLAs. For example, say we want to automatically performance test a set of ten RESTful APIs. We define an NFR that all of these must respond within 2 seconds 95% of the time. What happens if one of our API’s takes 2.1 seconds at the 95^th percentile?

Should we fail the build? Does that make sense from a business perspective? Probably not. So how do we do this better?

The most common way I’ve seen this done is to capture a baseline or benchmark and compare back to it during future builds. If this is a sliding benchmark (e.g. we always compare to the most recent run) then we lose the ability to pick up gradual degradation over time.  For example, we could have an SLA which says our API must not be more than 15% slower relative to the previous build, yet over 30 consecutive builds the response time could degrade by 10% and it wouldn’t be picked up.

A better way to go about this is to compare back to multiple previous runs, say the past 15 successful builds. I think this is a good way to go about it – but it’s hard.

We need to build a rules engine which is capable of looking back over each of these runs to draw meaningful conclusions about performance. I know people who have attempted this and it is much harder than it sounds. Our rules engine needs to be:

• Maintainable
• Configurable
• Understandable

…and our rules need to be relevant by relating back to real business need and being mathematically justified.

And here I am only exploring one dimension – response time. What about resource usage? The thing about performance is we do not know in advance which resource is going to be the bottleneck during a test. Think about how many counters are available in Windows Perfmon – most of them are irrelevant most of the time, and each of them requires customised rules around what is acceptable or not.

This is not an impossible problem and I think we are moving in the right direction, but it’s not something I have yet seen adequately addressed in the real world.

Production-like environments

Ideally we would run our performance tests in production, or a production-like environment. It’s one of the principles of DevOps – let’s stop building and testing our software in environments which are so radically different from the real world.

From experience, however, this is often not the case. Especially in the continuous delivery context. The table below explains some of the common ways our test environments differ from production.

So what can we learn from an environment that does not match production? It lets us compare performance relative to previous builds in the same environment. It does not necessarily tell us about what performance will be like in the real world.

Given this, my question is – is any of this testing activity worthwhile if we do it in an environment which does not match the real world?

When is it appropriate

I do think there are some situations where this kind of testing activity is more valuable than others. Most importantly, if you are a big multi-national business with millions of users – it makes sense. Do as much performance testing as you can as early as possible to find as many issues as you can (because you stand to lose a lot from performance issues). However, for a smaller company in a country such as New Zealand you will end up putting in a similar amount of effort to mitigate a much lower level of absolute risk – which begs the question, is it worthwhile?

Other factors which make a project more suitable than others include whether the software is being built bespoke, the presence or absence of production-like environments, the maturity of the deployment pipeline and the testability of the software. Another factor which I picked up from the PAC conference was having really good monitoring – ideally APM.

Reflection after the PAC

My perspective has changed slightly since attending the PAC.

In particular, I realise that I work in a unique situation – a small country with a small economy, where the value of this kind of activity is questionable. But overseas there are situations where this makes a lot more sense. I still think there are plenty of situations where continuous performance testing is inappropriate, but not always.

I also think there is a whole other discussion that needs to happen about what it takes to implement this in an organisation. It’s not about technology or process, it’s a culture shift. Either we need to get our development teams more engaged in the performance of their software, or we need our performance specialists to be integrated better into our software teams.

It’s also important to understand that continuous performance testing is not a replacement for end to end performance testing. It is simply an extra activity we can do to find performance issues earlier. As long as we all understand that, I’m much more comfortable with continuous performance testing as a concept.

I have my own thoughts about how we can move forward in a constructive way:

• Assess performance risk early. Build a connection between what we as performance testers do and business need so we can prioritise what we do based on what actually matters.
• Build more testable software. Either build or choose software which is testable in terms of the architecture and network traffic. Then we have a chance.
• Build better performance specialists. Let’s build a new generation of performance specialists who are not tools focused but can stretch in both directions – both to relate back what they do to business need, and to help investigate and diagnose performance issues when they find them.
• Promote a culture of performance awareness. Within our industry and organisations let’s make sure everyone understands what performance is and why it matters. And within our own community of performance specialists let’s be more honest about how hard some of this stuff is, and encourage deeper discussion so we can all learn from each other and do our jobs better.

Stephen Townshend

Stephen is a software performance specialist based in Auckland, New Zealand. Performance is about more than just testing – performance risk is business risk. Stephen’s view is that his job is to identify, prioritize, and manage performance risk in a way which is appropriate to each customer’s unique situation. This allows us to focus on what matters to the business the most, first.

He specializes in the two ends of an engagement – the risk assessment and strategy up front, and the investigation and diagnosis of issues in production or during performance testing.

Stephen is also a professionally trained actor for theatre and film which coincidentally makes him an expert at both performance and performance.

You can read the Stephen Townshend’s presentation here.
You want to know more about this event?

There has been many articles related to the potential performance issues related to deployment of Spectre & Meltdown Fixes/Patches.

• Intel CEO Krzanich: “We believe that the performance impact of these updates is heavily dependent on the workload” and further “We expect some workloads to have greater impact than others.”
• Game developer Epic has released a few metrics that show a 20% performance dip as the meltdown patch is applied to the Fortnite backend server.
• Red Hat’s performance team categorized performance results for Red Hat Enterprise Linux 7 (with similar behavior in Linux 6 and Linux 5) for a variety of performance-based benchmarks. They measured an 8-19% performance impact.

Spectre and Meltdown patches: dramatic losses that directly affect your productivity

This makes it more than ever necessary to run performance tests on your applications before going live in production when the application servers will receive these patches. We recommend that you do a performance test before and after deploying the patch, to be able to quantify the performance impact.

We are currently running a set of performance tests before and after deploying the patches on one of our labs application, and we will provide you with our benchmarks in the next days.

Neotys is here to help you with your tests, and to let you assess the impact of these patches with NeoLoad our super fast load testing solution and best in class expertise:

• With NeoLoad, you can test your applications extensively for performance. NeoLoad’s unique analysis capabilities will most likely uncover some optimization potential so you can level out the meltdown patches. Download NeoLoad here
• No other performance test tool lets you develop test scenarios as fast as with NeoLoad. In fact, our customers have told us they are 3x-5x faster with NeoLoad! Read more about NeoLoad here
• We can assist you with your tests through our network of highly specialized partners Contact Us Now

The objective of load testing is to simulate realistic user activity on the application. If a non-representative user journey is used, or if the appropriate load policy is undefined, the behavior of the app under load will not be able to be properly validated. Modeling Performance tests doesn’t require a technical skill set, just the time to fully understand the ins and outs of the application:

• How users work on the system; their habits
• When/where/how often they’re using the application?
• Is there any relation between external events and activity peaks?
• Is the company business plan related to application activity?
• Is the user community going to expand in different geographies?
• Is there a marketing plan focused on app promotion? If yes, who is the audience?
• Are there any layers of architecture shared with other systems?

To fully understand the application during performance modeling, the following roles should be involved:

• Functional Architect
• Business Analyst(s)
• Project Leader

Of course, having different functional representation will provide varied feedback. The goal should be to understand the application, the end users’ habits, and the relation of the application to the current/future state of the organization.

System logs or database extractions can also be useful. These easily point out the main components and functional areas currently used in production. In addition, they will retrieve the number of transactions/hour per business process.

• When considering the user load, it is important to be aware of the difference between the number of concurrent users and the number of users within the expected audience. An application delivered to 5,000 users may only have 500 users accessing it simultaneously.
• If there isn’t any estimation on the maximum concurrent user volume, the number of users can be calculated based on the number of transactions/hour per business actions.

Establish Service Level Agreements (SLAs) & Service Level Objectives (SLOs)

Service Level Agreements and Service Level Objectives are the keys to automation and performance validation. The project leader and functional architect need to define ideal response times for the application as there are no standards to leverage.

SLAs/SLOs enable the Performance Engineer to provide a performance testing results status perspective. With these, performance testing can be easily automated to identify performance regression between several releases of the application.

If the importance of component testing and performance testing in an early project stage is properly understood, the component level SLA or SLO must be defined to enable successful automation.

Test Selection

Each test run will address one risk – status on the performance requirements.

Most projects only focus on reaching an application’s limit. This is important to validate sizing and architecture configuration. However, it most likely will not answer all of the performance requirements.

There are other tests that need to run to properly to qualify app performance. These tests will help validate performance requirements:

• Unit Test: Many projects overlook the power of unit testing. Don’t start running load tests if the performance is not acceptable with one single user. Unit testing allows the application to be instrumented with deep-dive analytics tools and provides useful information to the devs. This step should be a mandatory part of any load testing strategy.
• Connection Test: Let’s say that each morning, all of the application users connect to the system at roughly the same time, then grab a coffee or conversation with a colleague. Even if there are no major activities on the business components of the application, the architecture has to handle this peak user session scenario. It should be obvious – to ensure that the system won’t break every time this happens. Lives will be much easier for the ops team members by validating this point. This test will ramp up all the expected users.
• Production Activity Test: Every application has major business actions. These actions often result in data being updated or inserted into the database. Therefore, there should be a good understanding of the number of records created per day/hour. Production activity testing ensures that the system is able to handle the load related to the number of transactions/hour expected in production. This test will load/validate the behavior of the business components within the database. Two main requirements for this type of test include the number of transactions/hour per user journey; appropriate think times.
• Peak Test: Every application experiences peak volumes. Even if those peaks appear as infrequently as once or twice a year, it is mandatory to ensure that the application can handle it. Peak testing’s purpose is to simulate an important increase of users during a short period of time (e.g., a retail site’s simulation of activity resulting from typical holiday sales frenzy).
• Soak Test: The application/architecture will most likely be available for several days or even weeks on end. In the absence of dedicated times set for maintenance tasks, it is important that the architecture will run without failure over a long period of time. Soak testing runs a constant number of users over extended timeframes. With this test practice, it will be easy to identify any memory leak/network conjunction so that you can easily monitor the stability of the environment’s current configuration.

• Batch Test: Some apps are designed with asynchronous tasks or batches. What is the impact of a batch on the real users? Batch testing will run the production activity, triggering the execution of a batch at a specific test moment. The purpose – identify if the batch has any impact on the user experience.

A number of other load tests could be utilized depending on the constraints of the business and the locations of the users. There are always events related to the company/organization that will affect the application load.

For example, the load designed for a trading application will depend on the opening hours of different markets. The app will have three typical phases: one for Asian users, a second for European users combined with additional Asian users, and the third for the U.S., which combines users from all three. Every time the markets open or close, there are peaks on trading activity. Therefore, testing would include different load types combining various business case combinations.

On the other hand, there are also tests that are designed to validate the platform availability during maintenance tasks or a production incident:

• Failover Test: The purpose of this test is to simulate a production failure on the environment. This test type is mandatory to validate the availability of the environment and to make sure the failover cluster mechanism reacts properly. When the architecture has N nodes, it is important to validate that the N-1 or N-2 nodes can handle the expected load (to avoid event cascading during production issues). The ops team is probably also interested in whether they can complete their maintenance tasks without having to set the app into a maintenance state. Most high-availability applications have many nodes on different layers of the architecture: Web Servers, Application Servers, Database, etc. There should be one test/layer.
• Recovery Test: The opposite of the degradation test, the recovery test is initiated by stopping one of the nodes and then restarting while the app is loaded. Its purpose is to see how the node reacts to a heavy load post-re-start.

Additionally, there may be operational situations to include during the load test to measure application behavior when the cache is being cleaned.

There is yet another point that could affect the performance of the application: the data. Often when load testing a test database less data is used, or at least a DB anonymized for security purposes. The anonymous process usually creates records starting with the same letters/digit. So, all of the indexes used during the load test will be incomparable to those of the normal production database.

Data grows rapidly in production. DB behavior is quite different depending on its size. If the lifetime of the database is long, then it might make sense to validate the performance of different sized DBs to determine if there is a limit differential between a light/heavy database. To achieve this, a large dataset pointing to various areas of the database should be used (e.g., never use a list of names where all the accounts start with AA or AAA2). Instead, keep a representative set of data from A to Z.

The type and complexity of the test will change during the project lifecycle:

Include Think Times

• An important element of performance design, think time is the time needed by a real user between two business actions:
  • Time to read information displayed on-screen
  • Time to fill out information in a form
• Other real user actions that do not cause interaction with the application servers

Because every real-world user behaves differently, think times will always be different. Therefore, it is important:

• To gather the think time of each business step in each scenario. Avoid using the same think time for each step. There are always screens displaying information that end users need to read or forms they need to fill out before step progression takes place. There should be a specific think time for each action.
• To calculate the average minimum think time and the average maximum think time/action.
• Let the load testing tool randomly choose a think time from a specified range (min and max).

Imagine the application is a castle with big walls and doors. Components that will be validated (via load testing) are inside the castle. The wall will represent proxy servers, load balancers, caching layers, firewalls, etc.

If a test is run without including think times, then only the main door would be hit. Okay, it may break, but there is a big chance that it would lock everything out after a short time. Be gentle and smart, and hide from the defense. The main door will allow entrance, and the components located inside of the castle will be able to be properly load tested.

Validate the User Experience

As mentioned earlier, once the application is assembled, testing objectives will change. At some point, the quality of the user experience too needs validation.

Mobile applications, rich Internet applications (RIA), and complex AJAX frameworks are challenging in the way most are used to measure response times. In the past, measurements have been limited to download time and Time to first byte (TTFB).

This approach does not make sense because much of the rendering time that contributes to user experience depends on local ActiveX/JavaScript or native application logic. As such, testing measurements cannot be limited to TTFB and download time because the main objective is to validate the global user experience on the application.

Measuring the user experience is possible by combining two solutions: load testing software (NeoLoad) and a browser-based or mobile testing tool.

The load testing tool will generate 98% of the load on the application. The browser-based or mobile-based testing tool will generate the other 2% of the load to retrieve the real user experience (including the rendering time) while the application is loaded.

This means that the business processes and transactions to be monitored by the browser/mobile-based solutions need to be carefully identified.

Running tests without monitoring is like watching a horror movie on the radio. You will hear people scream without knowing why. Monitoring is the only way to get metrics related to the behavior of the architecture.

However, many projects tend to lack performance monitoring due to:

• The lack of tools
• The fear of the requirements needed to enable monitoring

Though monitoring is not limited to the operating system of the different server(s), its purpose is to validate that each layer of the architecture is available and stable. Architects took time to build the smartest architecture, so it is necessary to measure the behavior of the different layers.

Monitoring allows for a more comprehensive understanding of the architecture and for the investigation into the behavior of the various pieces of the environment:

• Operating System: CPU, memory, disk, and network utilization
• Application Server: memory utilization, garbage collector, thread utilization, sessions
• Web Server: worker, number of sessions
• Database: buffer pool, cache utilization, number of transactions, number of commits, % indexed queries
• Caching Server: hit ratio

Many projects use production monitoring tools to retrieve metrics from the architecture. This approach is not recommended since production monitoring has a large granularity between each data point (every 2-5 minutes). In load testing, it is important to have monitored data collected at least every five seconds. The Performance Engineer needs to be given every opportunity to identify bottlenecks. When a peak appears for only a few seconds during a test, it is vital to have enough granularity to point out the bottleneck.

Monitoring requires technical requirements such as a system account, which services to be started, firewall ports to be opened, etc. Even if it seems difficult to meet those requirements, monitoring is possible if there is proper communication with operations; sending them these requirements in advance will facilitate that communication.

A key takeaway for monitoring: Anticipate requirements at an early stage.

Learn More about Modeling Performance Tests

Discover more load testing and performance testing content on the Neotys Resources pages, or download the latest version of NeoLoad and start testing today.

In the final piece of this four-part series focused on putting performance testing to work for you, we will cover where the rubber meets the road – “performance test execution.”

This post is part three of a four-part series collectively covering the following topics:

• Part 1 – Introduction to Performance Testing
• Part 2 – Establishing a Performance Testing Strategy
• Part 3 – Modeling Performance Tests
• Part 4 – Executing a Performance Test

[By Limor Leah Wainstein, Sealights Technologies LTD]

Testing the software you develop is vital in ensuring customer satisfaction with an application and its reliability. However, merely testing software is not enough—you need to know how to test properly, which encompasses several important factors.

The following post is the first in a three-part series on software testing, including recommendations for how to do it effectively. Part one provides an introductory view of software testing best practices, strategy development, and test modeling. Future posts will highlight some software quality metrics, how to conduct risk analysis, and more.

Software Testing Best Practices

While it’s true that each development team probably has different opinions on the best way to test software, it’s imperative not to drift too far from the prevailing best practices when testing.

Established best practices act as a signpost—they point you towards the right way to test the applications under development, minimizing the risk of defects escaping into production. Best practices are classed as such for a reason—it’s because they drive effective testing efforts.

Deviating too far from established and agreed upon ways to do things carries with it a greater risk of something going wrong with software tests, such as missing important errors that make it through to the customer.

Some examples of software testing best practices include:

• Testing throughout the software development cycle
• Writing tests for maximum coverage—even if 100 percent is not possible, always try to maximize how much of the application and its behavior you test
• Keeping detailed notes on all tests to ensure unambiguous test reports
• Automating all possible tests that you can automate

Following the best practices for software testing doesn’t guarantee an effective testing effort, but it undeniably decreases the risk of something going wrong.

Use a Solid Software Testing Strategy

A test strategy is used to outline the testing approach that a company takes when testing its software. The test strategy includes details on the testing objectives, time and resources required for testing, the testing environment, and descriptions of the types of testing planned.

Quite apart from developing software to meet particular customer functional requirements, all applications must also run smoothly under a variety of conditions. Performance risk relates to the risk that the application doesn’t function because of memory issues, changes in how customers use the software, code changes, or other potential performance bottlenecks.

The standard method for testing software to identify performance risk is to simulate different production loads and check how the software copes with strenuous use. Load testing is often conducted at the end of the development phase, when bugs are more costly to fix.

A potential benefit of designing a solid test strategy from the outset is being able to shift from a reactive performance testing approach to a proactive performance engineering approach.

Everyone takes part in performance engineering—it occurs throughout the development cycle. Designing a solid test strategy leads to risk-based testing, in which all performance risks related to a software release are identified during test planning, and a risk mitigation plan is drawn up. The software can thus be developed to minimize those performance risks from causing issues later on in the development cycle.

Load testing still plays an important role in verifying the functioning of the application under specific loads, but the point is that the right test strategy can ensure applications are more likely to pass their performance tests because potential performance issues have been dealt with earlier.

Model Your Tests

It’s imperative to create functional test cases that attempt to validate the realistic behavior of the system under development. Testing teams can achieve this by creating a model of the system that needs testing. The model is an abstraction—a partial representation of the system’s desired behavior derived from the software requirements. A set of realistic test cases can then be generated from models automatically using model-based testing (MBT) tools such as fMBT or TestOptimal.

Modeling ensures realistic test cases can be generated. An additional advantage of using a model is lower project maintenance because it is easier with a model re-generate test cases for new features.

Summary

• Good software testing depends firstly on a firm understanding of established best practices for testing software. While each team has its own perspectives on how to test, it helps not to deviate too far from these best practices.
• Implementing a solid test strategy helps with identifying key performance risks and building software with those risks in mind. Performance testing is, therefore, less likely to become a bottleneck because teams have dealt with risks earlier in development.
• Creating a model of the system under development helps teams create realistic test cases. Furthermore, a model requires less maintenance later on when test cases need to be made again to test new software features.

About Limor Leah Wainstein:

Limor is a technical writer and editor at focused on technology and SaaS markets. She began working in Agile teams over 10 years ago, and has been writing technical articles and documentation ever since. She writes for various audiences, including technical on-site content, software documentation, and dev guides. She specializes in big data analytics, computer/network security, middleware, software development and APIs.

Each performance testing activity is mapped out in the following workflow:

Performance test objectives will need to adapt depending on results. The Performance Engineer will analyze these results between testing and tuning. Ultimately, the main objectives of a performance test are to identify bottlenecks to uncover solutions. Below are the three key steps in executing an effective performance test.

Performance Test Design

Historically, the design phase has been referred to as “scripting,” but new technology has made this phase much more complex today.

Performance test script creation is essentially a software development project. Sometimes, automatic script generation from the recording is mistakenly interpreted as “whole” script creation, but it is only the beginning. Limited to select cases will automatic generation provide completed scripts. In most non-trivial cases, it’s just a first step. Scripts need to be correlated (procuring dynamic variables from the server) and parameterized (utilizing different data for different users). These operations are prone to errors because changes are made directly to the communication stream. Any mistakes at this point can be very dangerous because these types of mistakes usually cannot happen in the real-world where the users interact with the system through a user interface or API calls.

AJAX applications, WebSocket, and polling technologies generate regular requests. These requests are not generated by the interaction of the user within the graphical user interface. Instead, many calls are generated by one or more individual page components. For example, on an eCommerce website, there is always a direct link to the user cart from the pages. This link needs to show the user the exact number of products currently in his basket. Most of the time, the number of products is displayed via polling request (a technical call performed every five seconds). This type of call needs to be understood by the Performance Engineer. All “polling” mechanisms would need to be removed from the business step of the user journey and placed into a dedicated thread.

Another good example is adaptive streaming technology. Many testers often rely on the record/playback approach. Unfortunately, this type of logic won’t benefit any testing strategy. The goal is to qualify the behavior of the application under realistic load, not to validate the caching. Replaying back the recording request with no correlation will only call the cache of the application: web and application cache.

An additional important element of the design step is the dataset. Using a small set of data will generate the exact same query on the database. As previously mentioned, the point of a load test is not to qualify the efficiency of the database cache or simply deadlocks.

After the script is created, it should be evaluated for both single and multiple users, and with different data. Do not assume that the system works correctly when a script is executed without error. Instead, ensure that the applied workload is doing what it is supposed to – catching and logging all errors found. It can be done directly by analyzing server responses or, in cases when this is impossible, indirectly. It can also be done by analyzing the application log or database for the existence of particular entries.

Many tools provide some way to verify workload and check errors, but a complete understanding of exactly what’s happening is necessary. NeoLoad allows the user to create assertions to make sure that the response of the server is “normal,” (e.g., the content is as expected). For example, receiving “out of memory” errors in a browser page instead of the requested reports would qualify as unexpected content. It is important to catch such content errors with the help of assertions.

NeoLoad and many load testing solutions will automatically capture HTTP errors for web scripts (e.g., 500 “Internal Server Error”). If only the default diagnostics are relied upon, it can’t be verified that the business transactions are doing what is expected.

New web design tends to avoid HTTP errors and displays exception errors in the applications. So if checking is limited to examining only the HTTP codes, it can’t be determined if the scenario is achieving the expected actions on the application.

Performance Test Execution

When executing the actual test, take these challenges into consideration:

• Load testing architecture
• Effective test monitoring

Load Testing Architecture

The main objective is to load test the application, NOT the load testing software. Most load testing solutions have several components:

• Controller: Orchestrates the test, stores the results data: response time, hit(s), throughput, monitoring metrics, errors etc.
• Load Generator: Runs the load testing script against the application. This component will need to handle several hundreds of concurrent virtual users.

To avoid being limited by the load testing architecture, there needs to be enough:

• Load generators to achieve the expected load
• Network bandwidth between the load generator/application

Sizing load generators is a crucial step in defining the number of virtual users (VUs) that a load generator can handle. This number of VUs depends on many aspects:

• Technologies and complexity associated with the application to test
• Network connection capacity

Starting a “ramp-up” (or “scalability”) test of the application with one single load generator will help determine the maximum capacity of one machine. After determining the number of VUs a single load generator can handle (and the number of test VUs), it is easy to calculate the number of generators necessary to run the test. To pinpoint the load generator’s load limits, it requires that you keep an eye on its behavior under ramp-up test load, specifically:

• CPU
• Memory
• Throughput
• Hit rate
• Virtual users load

The first significant breaking point in the CPU, memory, throughput, and/or hit rate metrics represent the performance limit of the load generator. This point must be correlated with the number of VUs generated. Serious issues may occur with the load generators if pushed beyond this number of virtual users. A 20-30% security margin out of the sizing limits is recommended for the load generators in a heavy load test. It is also important to avoid using the controller machine as a load generator. The controller is the heart of the test. It is always better to lose a load generator than a controller (the whole test result).

Effective Test Monitoring

As mentioned earlier, there is limited time to run different tests to then proceed to monitor. Time shouldn’t be wasted by looking at the test data while the test is running. If response time starts increasing due to web server saturation, stop the test and start tuning the web server. Almost every time testing is started on a new application and/or environment, most of the layers (app/web server) are not configured or tuned for the load of the application. This means that every test on a representative environment needs attention from the Performance Engineer to properly tune the environment.

Every new test utilizing a new dataset also needs attention. For example, stop the test if every test user is generating errors when it logs into the application. On the other hand, if there is a good understanding of the application and the environment, test automation can be started without concern over application behavior. Once the test is finished, analyze the results.

Test Analysis

Load testing results analysis can be a job unto itself. A comprehensive knowledge of the following is required:

• Load testing design
• Technical layers involved in the application
• Modern architecture

Although, this post doesn’t delve into results analysis how-to, here are some recommendations for reporting on the project’s performance testing results.

• Almost all load testing solutions allow for the creation of complex graphs that correlate data. The Performance Engineer’s first inclination will be to show all of the graphs in the report.
• Before creating this report, it is important to understand the role and the technical skills of the person validating or reading the report.
• Based on these qualities, several different types of reports can be created. For example:
  • A technical report for developers and operations showing only important graphs
  • A decision maker report giving a very simple status of the application’s performance
• The main purpose of the performance testing report is to give a clear status on application performance. Result reports should be simplified, focusing on these three main themes:
  • Response times
  • Availability
  • Scalability
• A graphical presentation (e.g., three pie charts) makes the results easier to understand for the decision makers.
• Ultimately, the performance report needs to highlight if the performance requirements (identified during the performance strategy phase) are validated.

Learn More about Performance Testing

Discover more load testing and performance testing content on the Neotys Resources pages, or download the latest version of NeoLoad and start testing today.

This concludes our four-part series collectively covering the following topics. Feel free to review earlier installments should you have missed anything over the last few weeks.

• Part 1 – Introduction to Performance Testing
• Part 2 – Establishing a Performance Testing Strategy
• Part 3 – Modeling Performance Tests
• Part 4 – Performance Test Execution

As Performance Engineers, we are experiencing role evolution every day. This means that our challenges are changing too. Things like statistical analysis for load policy build, project input requirements gathering and scripting aren’t the primary obstacles anymore. Instead, these have been replaced by tasks such as:

• Configuration of the automated testing pipeline (including script and test project alignment)
• KPI definition in order to detect regression

The traditional Performance Tester used to configure the monitoring from their performance testing tool. In today’s world, the monitoring piece does not adequately provide the level of detail required for developers to be able to correct performance issues the way they used to. Of course, this is the reason why APM has become such an important weapon in the Performance Engineer’s arsenal – outputting details requested by the developers, including visibility on app usage in production.

As you know, NeoLoad’s load testing platform enables seamless integration with the toolchain. This includes off-the-shelf integrations with the leading APM solutions available:

• AppDynamics
• CA Apm
• NewRelic
• Dynatrace

Dynatrace

It’s been two years since Dynatrace last released their platform (delivered via AI engine), but they’re at it again about to release an updated version. The team recently gave me an opportunity to review the offering, which included some firsthand testing with Andreas Grabner, DevOps Activist at Dynatrace.

With my testing behind me, I must say, that I was very impressed with Dynatrace’s ability to identify problems/bottlenecks during load testing. For more details on the testing with Andreas, check out the Dynatrace Perfclinic video.

Rather than going on and on about my own test results, I’m sure you’d prefer to learn how you too can take advantage of the new integration benefits associated with Dynatrace and NeoLoad. Here are a few key highlights:

• Dynatrace allows several options to build load testing integrations:
  • Sending load testing metrics to Dynatrace
  • “Tagging” web requests generated by the load testing solution
  • Dynatrace metrics in a load testing solution
• The new integration gives Performance Engineers the ability to analyze test results directly within the Dynatrace platform via dashboard.
  • Dynatrace will help pinpoint bottleneck(s)

Learn More

To take advantage of all the options provided by Dynatrace, click here. For a detailed tutorial describing how to enable this integration in NeoLoad and Dynatrace, please download this How-to.

You can also watch the Kitchen TV recorded by Henrik Rexed (Neotys) and Andreas Grabner (Dynatrace) during the first edition of our Performance Advisory Council.

[By Bruno Da Silva, Leanovia]

By the time your web application goes into production, it likely will have gone through several rounds of tests during development. But, no matter the number of unit and regression tests you’ve run, there’s really only one way to know if your app can handle realistic user load. Performance testing allows you to validate your application capacity such that it performs under various loads and conditions.

IT Performance 101

Before diving further into performance testing, and how to get the best from the practice, I’d like to just remind you of the IT performance drivers.

IT performance boils down to the balance of three aspects:

• Application: How your application is architectured and developed. Experience shows that bad performance may come from improper design and/or development.

 What is the application supposed to do? Is the implementation adapted? Are the design patterns/frameworks used correctly? Are the components designed to scale? Is there a single point of failure or bottleneck in your architecture?

• Resources: What your application relies on to ensure its working order. Experience also tells us that poor performance could be resultant from insufficient environment sizing.

 How are clustering and high-availability implemented? Are you limited by network capacity or virtualization? Can your system handle a heavy load from all your applications at the same time?

• Load: How your application and resources are used. Based on experience, another determinant of bad performance can be usage.

 Are use cases/user profiles identified on your application? What load peaks are you facing? Are you testing a new application or an existing one?

Performance Testing Strategy

Testing for performance stability is challenging because it is hard to predict the load you will be facing in production. No matter the case, there is always a point where your application will start behaving poorly or even crash. Because of this, you run the risk of not being able to process an unexpected heavy load. However, you can identify your bottlenecks prior so you can build a highly available and resilient system.

I recommend you consider incorporating the following phases when performance testing:

• Load Testing: Helps with evaluation of your compliance with performance requirements under a defined load.
• Stress Testing: Helps with evaluation of the system or one of its components under heavy load. This is essential in order to identify potential bottlenecks, actions you need to consider for resource capacity planning.

Load Test Design Best Practices (two approaches)

• Shift left: Continuous testing performance during application development. This ensures performance from the earliest stage of the application, and the guaranty that you’ve included all use cases for final testing. When designing load test scenarios at the end of development, chances are, you will miss something.
• Shift right: Improving your load tests with production feedback. The information received based on real usage is valuable insight which helps you build better load tests, check for performance stability in future releases during regression testing. This is also a great opportunity to check whether your application and infrastructure need further adjustment.

By integrating load tests into your development cycle, you will recognize immediate delivery improvement and efficiency gain.

In order to do so, you will need to carefully monitor your environment. This is essential because it gives you a good baseline when conducting analysis on test and production behavior. Moreover, it enables you to measure the impact of an application on shared components, which again, aids the capacity planning effort.

Data Generation Best Practices

Data is key when executing load tests. It helps you build more accurate use cases/user profiles. That is why you should consider data from the earliest stage of development.

When planning your scenarios, try to identify the activity you want to simulate. There are several types of phase activity – the following are considered typical:

• Initialization: The first uses of the application, which means the database is not loaded and where many creations are likely to be made.
• Campaign: The database will be loaded (likely impacting performance) and the usage is more about consultations, editions, and some creations.

When generating datasets and database dumps, you want to make sure your user profiles are realistic enough for your tests. Based on my experiences, here are a few tips to consider:

• Double-check that the rules you implement for generating data are correct. Failing to do this will prevent the ability to identify performance issues that could come from specific functional cases.
• Make sure to vary profiles according to reality. Users might have a higher impact on the database because of too many relationships. Vary number/size when it comes to files as well. A fresh visit or a returning user will not have the same impact on cache, etc.
• Use references among your tests for better consistency. It is good to keep data up-to-date when the application is evolving, but should be re-usable when checking for performance stability.

In case the application is already live, you might want to generate your database dumps from production data. However, you might not be able to do this because of data protection laws. As a result, you need to proceed to data anonymization.

You have to identify and substitute personal with dummy data. Afterwards, it is essential to automate this process so you save time in the future. Doing so will allow you to build the best data reference when conducting production-like testing using reliable data.

Performance in DevOps

As a consultant, I recognize that for most clients, IT performance is not the priority and only comes as a concern when performance failure is detected. Thus, it is quite difficult to implement performance governance according to the best practices. Thankfully, I had the chance to elaborate on performance within a DevOps context at Leanovia. Here are the categories and goals I try to achieve when addressing performance.

Performance Testing

Find the appropriate tool for load/stress testing according to your needs. You should consider tools that implement framework parameters in order to save time when testing an application being developed or a similar one.

Monitoring

Finding the right tool for monitoring is critical as this helps fuel the performance analysis of your load tests. This is invaluable to your knowledge base, as you are able to compare your tests with production cases to address them more efficiently.

Continuous Performance

Ensure performance from the start. Capitalize on your automated performance tests to avoid performance regression. Continuously update monitoring so nothing goes undetected.

Deployment

Automate the application deployment for improved consistency between tests. Experience shows you are more efficient and less error-prone if you are able to restore the environment easily.

Last, and most important, you should containerize your injectors so you can create instances on- demand and not bother with tedious installations.

Data Generation

Capitalize on data you need for your load tests for reference during regression testing. Don’t forget to automate data generation with the proper tool, or by developing re-usable scripts with more complex data.

Bruno Da Silva

After IT studies at the UTC in Compiègne (France), Bruno Da Silva joins Leanovia for an epic journey. Passionated about IT performance, he works as a consultant to address challenging customers issues. In less than a year, he obtains NeoLoad and Dynatrace certifications in order to work with state-of-the-art technologies regarding load testing and monitoring. The adventure of becoming a Performance Ninja Warrior is just beginning.

He also loves chocolate, video games and skiing ; not necessarily in that order.

You can read the Bruno Da Silva’s presentation here.
You want to know more about this event?

For their Agile and DevOps-focused application development teams, leading companies want to achieve both speed and quality without compromise. That’s where NeoLoad 6.3 comes in, helping to meet increasing user demands, bring apps to market faster and more frequently, and release flawlessly with confidence, every time.

What’s New in This Release?

NeoLoad 6.3 continues Neotys’ focus on enabling Agile and DevOps teams to execute superior performance testing while shortening the build/test cycle, enable collaboration across teams, and establish Performance Testing as a trusted partner within the software delivery chain. Enhancements in this release focus on NeoLoad’s platform openness and core load testing features.

Enhanced Dynatrace Integration

NeoLoad 6.3 includes improvements to NeoLoad’s native integration with Dynatrace and new bundle of Advanced Actions available from Neotys Labs leveraging load testing analysis via Dynatrace’s OneAgent. The expanded capability means tighter integration with a key element of the testing ecosystem, faster test cycles, and pre-production (“Shifting Right”) & Component/API-level testing (“Shifting Left”).

These new integration features make it easier and more effective to interact with Dynatrace APM deployed in the testing environment during NeoLoad app load testing. With this setup, testers can precisely analyze how the whole application and its ecosystem are behaving.

API Enhancements

Users of NeoLoad Web are now able to modify the success status of a test through the API, enabling testers the ability to define and customize success/failure criteria for tests. Testers can programmatically (using the REST API) define what success and failure mean, based on test results data accessed (REST API). The NeoLoad RUNTIME API has also been enhanced to allow testers to automatically and programmatically launch several tests during the same Cloud session.

Access to NeoLoad Web URLs from the Inside

A tester/developer writing “glue code” within NeoLoad’s GUI that integrates with other tools can easily access URLs related to NeoLoad Web making the integration tighter. The tester/developer can access the following URLs:

• NeoLoad Web front-end
• Entry point of NeoLoad Web APIs
• Test Run in NeoLoad Web

Testers/developers can also access Proxy settings and NLW-related URLs (in different situations and points of the product):

• From the standard command-line output (Stdout) when running a test; the URL of the running test is clearly displayed
• When using the NeoLoad GUI Runtime API access to running test URL
• When using either the Custom Action API or the JS API, access available to both the URL of running test and NeoLoad Web front-end URL

See the custom Advanced Action API, JS API, and Runtime API.

Password Encryption in Test Scenarios

NeoLoad 6.3 includes enhanced security for test credentials to protect against unauthorized user access.

Microsoft Monitoring Enhancements

Monitoring a Windows machine or Microsoft Server (SQL Server, IIS, .Net) is now easier with NeoLoad 6.3’s new authentication capabilities for Microsoft environments.

Learn More

Discover all of NeoLoad 6.3’s new and enhanced features on our What’s New and Technical Features pages, or download the latest version and start testing today.

By now, you’ve probably heard about Spectre and Meltdown. In the last few weeks, there’s been many a blog on the patch topic:

• Spectre and Meltdown attacks against microprocessors
• More details about mitigations for the CPU speculative execution issue
• Microsoft says chip fix may significantly slow some servers
• Meltdown, Spectre bug patch slowdown gets real – and what you can do about it

As a Performance Engineer, this patch conversation does give me a little heartburn, but I know that I need to test before jumping to any conclusion. Therefore, I took one of our spare Intel Xeon W3550 servers and installed:

• OS: CentOs 7
• PHP
• MySQL
• Apache2
• Demo application

What’s The Testing Scope?

In order to test the impact of the patch, I decided to run two different test types.

1. CPU Behavior: Launching several types of “CPUburn” to measure patch conduct by generating CPU load, io, disk, and memory activity. This test doesn’t simulate the real activity of the server, but it is a simple way to measure patch impact.
2. Traffic Simulation: Generating load test by reproducing web app traffic. The objective of this test is to detect if the patch is impacting the user experience of PHP/MySQL website.

Of course, those two tests will be executed before and after patch installation so that you can easily compare patch’s impact on the CPU or the end user.

So, let’s review patch test results against Spectre and Meltdown…

Test CPUBurn

With the test complete, let’s compare the behavior of the CPU with the help of stress:

CPU System

The behavior of the CPU System is almost the same after the patch.

CPU User

Similar to the CPU System results noted above, the behavior of CPU user is exactly the same. From this first test, I am not able to make any conclusion on the impact of this patch on CPU behavior.

Load Test on Apache

User Behavior

By comparing the results before and after the test, NeoLoad is automatically providing a ratio on the expected KPI:

• Response time
• Element/sec

The response time has increased almost 10% for key transactions.

Transaction/sec

Because of the increase in response time, I am not surprised to observe fewer transactions/sec for select key transactions. The application is able to generate -1% transaction/sec for each transaction.

CPU System

In this scenario, the patch increased the CPU System usage by 16%.

On the other hand, the CPU user (Apache) is similar:

CPU Idle

Because of the increase of the CPU System, the server has 15% less CPU availability post-patch deployment.

Conclusion

From the two tests conducted on my server, I was able to see two different behaviors: no impact; a small impact on the CPU and response time. So, it may be disappointing to you in my lack of providing specific test influence specific on the general behavior of this patch on the application. To be clear, the patch could have a negative impact on the behavior of your system depending on:

• CPU: Does the system have a recent or old CPU?
• OS: Depending on the operating system, the patch partially resolves the issue
• Application type

If you don’t want to jeopardize your production and/or users, I’d recommend running a load test to measure the impact of the patch on your application.

Learn More

Discover more load testing and performance testing content on the Neotys Resources pages, or download the latest version of NeoLoad and start testing today.

It’s the beginning of a new year. For us employees, aside from solidifying 2018 sales or marketing plans, it likely includes last-minute payroll or human resource adjustments to ensure our individual profiles are current.

Technology Drives Innovation & Sophistication

No doubt, a company like PrismHR, the largest payroll, benefits, and HR software platform in their space, is likely on the receiving end of your company’s update process.

As a cutting-edge software provider, it wasn’t that long ago that PrismHR was fairly new to the performance and load testing practice. Fact, when PrismHR first partnered with NeoLoad, they were only testing against single parts of their product – a process that worked for them in the beginning but has evolved into something more sophisticated.

Since those early days, PrismHR has expanded NeoLoad’s workload to include architectural (OS, Middleware, third-party tool updates) and software changes prior to release into production. The team now feels a great sense of confidence both in the process as well as with their ability to re-test “typical” scenarios whether updates are being made to components or the whole product itself.

Delivering Real, Positive User Experiences

Primary to the end-users experience is the data that PrismHR has been able to collect and monitor. With the ability to measure system resource consumption and speed, it feels strongly that it has the pieces in place to evaluate and continually improve end-user experiences. “Having this new data made us think about all of our systems. We have tools for monitoring at the system level but needed a way to measure product speed at the user level. With NeoLoad, we are able to evaluate the end-user experience,” says Andrew Peace, Software Integration Manager at PrismHR.

Continuous Integration: What’s Next with NeoLoad?

Andy’s team looks forward to assimilating NeoLoad into its Continuous Integration (CI) toolchain – running load tests as part of their daily build process. “This will allow us to uncover performance issues earlier when it’s easier and less costly to fix as well as push us toward putting test automation to work for us,” states Peace confidently.

For the full story of PrismHR’s evolution of NeoLoad, read the case study. Or, to start testing with NeoLoad yourself, click here.

[By Todd DeCapua]

“I am sick of the hype.”

In my discussion at the Performance Advisory Council, I performed a ‘deep dive’ session covering these 3 topics, reviewing solution options and practical approaches that work, enabling us to apply them now and into the future.

My position is, “I am sick of the hype…let’s get practical…and get sh*t done!”

For each of the 3 Top Challenges, I led the discussion, using slides and conversation along the topics of: Introduction, Deep Dive, and Future and How-To’s. This turned out to be a great format, and one that encourages an active discussion across the council, and perspectives we spoke about more as our time together continued.

Challenge #1: Big Data / Machine Learning / Artificial Intelligence

For this blog topic, I thought it would be interesting to share where some of the most active dialog happened, that being the “Deep Dive” in contemplating many of the key learnings we have faced.

• Do we know all the data points?

When starting and even delivering, are you sure you know all the elements of data you are going to need, and will new or changing inputs change the outcomes. This has become a very interesting challenge, especially in looking at very large and unstructured streaming data sets. Not to be taken lightly, and a challenge many of us on the council have faced, while others recognizing it is something they will be challenged with soon.

• Have all the algorithms been defined?

Do you ever know if you have defined or ‘tuned’ all the algorithms? This is an important question, and I would suggest fact, which you need to come to grips with in your journey. Of course, as you get deep into the world of algorithms and what they and their results represent, some will also be challenges to think introspectively…”is there a way you could automate algorithms to improve the algorithms?” Continue to challenge the hypothesis, and enable the machines to continuously learn and apply, so to continuously prove and optimize your findings.

• Where are all the ‘processing engines’?

As you think through where all of this compute, or multiple compute areas within your stack, where is this or are these happening? For example, will you plan to have much of the compute done within the edge, or do you have it closer to the core…and why? As you continue to work through this question and why, think of resiliency, and what happens if the processing fails and/or the network between edge and core become poor or disconnected…then what happens, and does it matter? Are there other power, heat, elements or other outside environmental forces which also need to be taken into consideration when making this seemingly simple decision or influence?

• Are there unique protocols?

Looking across much of the data of today, sensors often provide a lot of information, these communication and transport protocols continue to evolve at a very rapid pace. Thinking of how your framework with manage and grow with this changing core element is a consideration not only for ingestion…but, also for processing and results…whatever form that may take. Are there known protocols today, which will limit processing given scale and/or complexity within your environment or other consumers of your results?

• What are the volumes/size of data?

Knowing we are speaking of “Big Data” and the accompanying Machine Learning and Artificial Intelligence…we need to consider and build forecast models for many factors, including: data growth, storage, processing power, network traffic and conditions, and much more. Often these considerations are an afterthought…and the massive and accelerated growth of data and storage, becomes a quickly crippling task, as the desire for more becomes exponential…often not by human intervention, but that of the machine.

• How do we create, reset and refresh?

Well, this is one from a lot of experience, and trial & error. When thinking of the BD, ML and AI; how will you be able to create these data sets and environments, and do you have reset and refresh strategies in place, so to be able to maintain the development and deploy cycles required. Many times, the rule of seven comes into play…knowing if you have 1 copy, you will end up with 7 as you perpetuate and accelerate through the development cycle….so, how much disk will you require and how can you quickly (minutes) reset and refresh your environment(s)?

Challenge #2: Blockchain

For this challenge, I wanted to share some of the “Future and How-To’s” insights we discussed. You will find content I used to guide the discussion from both Gartner and also Mattias Scherer. For Gartner, they have published Top 10 Strategic Technology Trends for 2018 and quoting the document stated, “Blockchain is a powerful tool for digital business because of its ability to: Remove business and technology friction; Enable native asset creation and distribution; Provide a managed trust model.” The below illustration also provides a use case “The Blockchain Ledger” on the left with some “Extended Characteristics” to the right, showing some common platforms on the bottom.

The below illustration “Figure 2.4.1” is representative of an example transaction flow involving 4 actors total: Client, Endorser A, Endorser B and Committer C; flowing through 6 steps represented.

This conceptually highlights 3 elements I highlighted during our discussion and have learned is important to use to shape this dialog.

1. Define business case

Before making a significant investment of time, resources and/or money; it is important to define your business case. This may seem like a simple point…but, it is interesting to observe how many individuals and organizations have seemingly jumped in and just ‘got started’ without a vision for direction or potential end state or why. How will this capability enable you to differentiate and accelerate your business for your customers? Let us find a quick and easy opportunity to pilot and recognize value, or not, quickly.

2. Outline workflows

Taking the next step is to start understanding the relationships, showing the key handoffs which nearly eliminate the delays and errors, accelerating and growing the value to your end users….all while doing it faster and cheaper. What will you discover, could be risks and/or opportunities, remember to focus on your customers and supply chain.

3. Public vs. Permissioned blockchain

Another big topic of discussion was around organizations and shall they be using public or permissioned blockchain environments. Does this impact the performance of the blockchain? How are you testing the performance across blockchain today? If public and line of sight to all of your supply chain, are there conditions that apply, which would be different within a permissioned blockchain? Perhaps there is a competitive advantage if you setup and permissioned blockchain where you own the intellectual property and domain expertise, so to be first to market and deliver the full and complete automated capability through the supply chain and servicing the customer. What would that mean for you and your organization?

The last topic I covered within the challenge is research provided by Mattias Scherer titled, “Performance and Scalability of Blockchain Networks and Smart Contracts.” This provides a 46 pages thesis published in 2017, which covers in great detail a number of performance and security related to Blockchain.

Cited: Scherer, M. (2017). Performance and Scalability of Blockchain Networks and Smart Contracts (Dissertation). Retrieved from http://urn.kb.se/resolve?urn=urn:nbn:se:umu:diva-136470

Below is the Abstract published by Mattias for his Independent Thesis.
“The blockchain technology started as the innovation that powered the cryptocurrency Bitcoin. But in recent years, leaders in finance, banking, and many more companies has given this new innovation more attention than ever before. They seek a new technology to replace their system which are often inefficient and costly to operate. However, one of the reasons why it not possible to use a blockchain right away is because of the poor performance. Public blockchains, where anyone can participate, can only process a couple of transaction per second and is therefore far from usable in the world of finance. Permissioned blockchains is another type of blockchain where only a restricted set of users have the rights to decide what will be recorded in the blockchain. This allows permissioned blockchains to have a number of advantages over public blockchains. Most notably is the ability to split the network into segments where only a subset of nodes needs to validate transactions to a particular application, allowing the use of parallel computing and better scaling. Moreover, the validating nodes can be trusted, allowing the use of consensus algorithm which offer much more throughput.

In this paper, we compare public blockchain with permissioned blockchain and address the notable trade-offs: decentralization, scalability and security, in the different blockchain networks. Furthermore, we examine the potential of using a permissioned blockchain to replace the old systems used in financial institutes and banks by launching a Hyperledger Fabric network and run stress tests.

It is apparent that with less decentralization, the performance and scalability of Hyperledger Fabric network is improved and it is feasible that permissioned blockchain can be used in finance.”

Challenge #3: Internet of Things [IoT]

There are a lot of great discussions that started within my IoT dialogue. For this blog, I felt it would be good to start at the beginning, with the “Introduction.” I always find it interesting with this and many topics, how many people skip over the definition, and immediately get into providing their point of view…only to learn later the slight variance in perspective…which the results would have benefitted greatly from simply defining the topic/perspective first.

So, the definition I shared was, “IoT refers to the connection of devices (other than typical fares such as computers and smartphones) to the Internet. Cars, kitchen appliances, and even heart monitors can all be connected through the IoT. And as the Internet of Things grows in the next few years, more devices will join that list.”

The below illustration provided great discussion points, as we went around the room, and spoke about several examples….which many of us have worked on several of these scenarios, and shared some of the challenges and learnings we realized. One of the realizations we all shared was how ‘futuristic’ this all seemed only a few years ago, and now looking around, much of these are commonly implemented and instrumented in our everyday world. What many agreed has not evolved as quickly are the practices that are used to ensure everything works as designed, not only under common scenarios but also those extreme conditions or exceptions, which often are use cases that should have been recognized and products designed to perform well under those conditions.

To learn more about these topics, the discussions we had around each, and pickup a few items you can take away now to apply to the 3 top challenges you are facing…I wanted to provide the full recording to you. Please let me know what you learn and share it with others. Looking forward to the opportunity of connecting with you soon.

Todd DeCapua

Todd DeCapua is a technology evangelist, passionate software executive and business leader. Some of his roles/titles include: Senior Director of Technology and Product Innovation at CSC, Chief Technology Evangelist at Hewlett-Packard Enterprise, Co-Founder of TechBeacon.com, VP of Innovation and Strategy Board of Director at Vivit Worldwide, Independent Board of Director at iHireTech, and Independent Board of Director at Apposite Technologies. Is also a very active online author and contributor and co-author of the O’Reilly published book titled, “Effective Performance Engineering.”

You can read the Todd DeCapua’s presentation here.
You want to know more about this event?

IoT’s Big Bang

The Internet of Things (IoT) is a vast network of connected, physical objects or “things.” These “things” can be devices, vehicles, wearables, home appliances, sensors, and actuators. Each has an assigned IP address or other unique identifier and can collect and transfer data over a network without human assistance or intervention.

Gartner indicates that “The Internet of Things installed base will grow to 26 billion units by 2020.” That’s a whole lot of devices that will connect to the applications that drive manufacturing, supply chain, asset management, and many other industries! Regarding scope, McKinsey Global Institute postulates that IoT technologies could have a total economic impact of $3.9 trillion to $11.1 trillion per year by 2025. IoT is a rapidly expanding technology that is quickly transforming the process of the information collection that powers business decisions.

IoT Performance Testing Challenges

IoT has not only created opportunities for industries, companies, and devices to work together in new ways, but also for companies to commercialize the IoT data they gather. The business race to embrace IoT is driving an unprecedented level of complexity, data variety, and data volume within IoT ecosystems and a laser focus by business to monetize (or at least derive maximum value from) resulting data. Several factors conspire to make IoT performance testing hard.

Unbridled IoT Growth and Complexity

Consider that every IoT device has its hardware and requires software to make it work. This software needs to integrate with whatever IoT application software will be issuing commands to the device, and the IoT application needs to understand and analyze the data gathered by the device.  The ever-expanding number of “things” on the Internet and the variety of data they generate have increased the complexity of IoT ecosystems by orders of magnitude. A heart monitor collecting data about a patient uses different software/protocols and may communicate with its host application differently than an information gathering sensor in a car.

Devices and Data Ubiquity

The plethora of “things” and the data they gather is driving the requirement for more/faster integration with IoT host applications as well as support for data analytics initiatives that derive value from the data. Businesses thirst for the benefits they can reap from real-time insights and analytics outcomes. The result of this can put additional pressure on testers to validate the IoT ecosystem and components under test.

IoT Uses New Protocols

As “things” hit the market, they frequently create the need for new protocols to manage device-to-device, device-to-server, and server-to-server communications. The state of the IoT ecosystem is already incredibly complex because there is little standardization regarding the IoT protocols used. At this writing, the most common protocol seems to be Message Queuing Telemetry Transport (MQTT) since it performs well in high latency, low bandwidth situations. However, Constrained Application Protocol (CoAP) and Extensible Messaging and Presence Protocol (XMPP), DDS, and REST APIs over HTTP are commonly used as well. These new protocols, and the diversity of IoT devices they support the demand that performance and load test tools evolve to support QA test.

IoT Load Testing Considerations

In the IoT world, already complex architectures are further muddled by different versions of operating systems and firmware, the vast variety of things, and the volumes of data they manifest. Together, these variables present testers with new, and often perplexing, performance test challenges, mainly when load factors related to “the cloud” are incorporated.

Consider the following cloud-based use case example:

Weather forecast models indicate that a massive winter storm is descending on the Eastern seaboard. Smart thermostat vendors, such as the NEST, anticipate that intelligent thermostats within this region will communicate with and send computational updates to their IoT host servers (Nest’s private cloud) all at once. Vendors want to ensure that all relevant device data and “states” are captured by Nest servers and that appropriate communications and updates sent to devices.

As you can see, the daunting number of possible IoT combinations to stress test can vex the most seasoned tester. Defining what subset of variants to test and what load factors to account for is a critical precursor to your overall test initiative. Success demands a thoughtful approach and leverages action planning.

IoT Load Test Action Planning

First, gather information to understand which devices and software versions are present in your target IoT ecosystem. It may not be possible to test all variables, so your test scenarios need to focus on the most popular device/protocol combinations. Make sure that your strategy addresses the following:

• IoT complexity means that QA must test all the new devices that communicate with the network and account for factors such as latency, packet loss, network bandwidth, load, etc. When not accounted for, the impact of these factors in the IoT ecosystem can be catastrophic – consider a car is not responding to a corrective command while the driver is behind the wheel.
• Diverse usage conditions can impact the stability of the end user’s internet connection and play a significant role in determining test results. Therefore, test plans must ensure that data is detected and stored during service disruptions.

Next, choose the right testing tool – one that supports the underlying protocol(s) used by the IoT application under test. The platform needs to address things like sensors, devices, and actuators in the physical layer; the IoT protocol(s), databases, analytics components, and the processing of computations in the system layer. Also, requirements of the host IoT application such as business IoT requirements for healthcare or manufacturing vs. consumer IoT requirements for wearables and smart home.

Whatever tool you decide to use, your IoT performance and load test strategy should encompass the following parameters:

• Simulating the interactions of devices and sensors
• Scaling from thousands to millions of devices
• The continuous sharing of data. Because IoT devices connect to a server, performance testing must focus on the communication between devices and the network infrastructure.
• Supporting IoT protocols specific to the ecosystem under test
• Ensuring that notifications, requests, and responses are sent and received in proper form
• Providing appropriate data capture, integrations, and performance with analytics systems

Load Test Best Practices 

Align your test plan with the IoT platform: Understand the types of devices and protocols used on your IoT platform and design your test plan accordingly. You’ll also need to gather user requirements related to peak loads expected at given times. Test cases can be initiated defining typical and atypical use cases of the objects.

Prioritize test cases: Identify key areas that require extensive test effort and time. Evaluate how specific scenarios, such as the loss of data, will affect the IoT application under test (think manufacturing production) and how messages get passed in real time. Other scenarios, such as analytics of sensor data across geographies and devices require that applications can support anticipated loads – testers need to measure response times and track device stats.

Learn More

Discover more load testing and performance testing content on the Neotys Resources pages, or download the latest version of NeoLoad and start testing today.

Deb Cobb

Deb Cobb has deep expertise in enterprise product management and product marketing. She provides fractional product marketing, go-to-market strategy, and content development services for software and technology companies of all sizes. To learn more about Deb, her passion for collaborative thought leadership delivery, review her portfolio, or to connect directly, click here.

This week’s Amazon’s order processing issue perfectly underscores the importance for Performance Testers to always be planning for that “rainy day.” If you were unaware of the event, on the 7th of February the Amazon.de order processing system was shut down due to a voucher issue posted on the third-party portal mydealz.de. This one-time use discount code voucher was executable across multiple items in the Amazon cart and included a special “trick.” If you added an item from a specific discount category to your cart, you could add more items from any other category (by design, the discount on other items was to be removed during the final order step). What users found, however, was that the discount credit remained in the cart with no discrimination to the item total!

Unfortunately, the addition and removal process of certain items from a special discount category was not properly implemented. In fact, EVERY buyer was given the 10 € credit when shopping! The resulting untenable traffic spikes caused the need to shut down the ordering system before further order processing & revenue losses continued.

Is everyone with a retail storefront susceptible to customer-facing issues like this? How poised are you to avoid a similar fate? 

Load Testing can help

Load testing is a necessary critical step any retailer should include to help prevent these problems. A complementary approach is to implement a strict internal IT process discipline, especially if you’re managing multiple different third-party services’ impact on behavior under load. Simulation of interactions across all these entities can be even more of a challenge.

The best defense is to arm yourself with knowledge; check out our web services load testing best practices guide, including the use of service virtualization to simulate third-party services.

As part of our ongoing effort to share the full breadth of NeoLoad’s capabilities, particularly, as the product continues to evolve, Neotys will be publishing a series of Tutorials, that will be delivered with each new release of NeoLoad.

With this ongoing educative approach, Neotys hopes to present these “how-to” tutorials – accessed either in the NeoLoad Documentation online or directly from the NeoLoad GUI in the NeoLoad User Documentation. The first of these tutorials were delivered with NeoLoad 6.3:  “How to extend NeoLoad capacities using Custom Advanced Actions.”

Putting NeoLoad’s Custom Advanced Actions to Work

• Tailor the platform to whatever application(s) you wish to test – Using NeoLoad Logical actions, JavaScript, or NeoLoad Advanced Actions API
• Extend NeoLoad’s load testing capability – Reproduce communication with the application under test using non-native/custom protocols
• Extend NeoLoad’s monitoring capability (retrieval of monitoring data from a third-party)

For those unfamiliar with Advanced Actions, they extend the behavior of a User Path. While NeoLoad includes several examples of these, Neotys Labs provides other Advanced Actions as-is or as a starting point for customization.

Explore this Topic

For more information about this topic, take a look at:  “Custom action API: Create a Custom action.”

Discover more load testing and performance testing content on the Neotys Resources pages, or download the latest version of NeoLoad and start testing today.