Quite often, we use load testing as a means to measure, monitor, or even observe performance metrics and other characteristics of a system under test. Those characteristics can include availability, reliability, scalability, and also security. Each attribute tends to have its own set of metrics or behavior.
Performance commonly refers to response time metrics. Response time metrics are measured differently, depending on the software test tool and audience. Sometimes response time is measured as wall clock time, or the elapsed time it takes to complete a specific action or boundary within the test. Other response time metrics might focus on protocol level measurements, such as network timing (connection time, latency, time to first byte, time to last byte, etc.) or even navigation timings at the browser level (first contentful paint, first CPU idle, first meaningful paint, etc.) Response time metrics are often the focus of performance measurements, but there are other metrics of interest around performance.
Concurrent users are another standard performance metric in load tests. Concurrency or the number of concurrent users is a measure of the total number of people/actors/users/processes/threads using a resource within a predefined period. The resource can be anything related to your application or system under test but is often a measure of real simulated users accessing your application, such as your web site, for example. Not to be confused with active users, concurrent users are more often a subset of all unique/active users accessing a web site.
Throughput is a measure per unit over time. Throughput metrics are most commonly expressed as a rate, such as the average amount of network throughput in bits per second or the number of transactions that have passed or failed per minute.
I like to observe these three key performance metrics together, response time, concurrency and throughput, as the combination of the three can illustrate the behavior of the system under test, as well as help identify any obvious performance bottlenecks.
Performance can also dive deep on any number of parameters provided from real-time analytics. It is not uncommon to monitor response time metrics as part of real-time analytics in application and performance management tools. It is also commonplace to compare performance metrics from load tests with metrics from application performance management tools, structured logging, and observability platforms.
Availability, while perhaps not concerned with response time performance, is often worried about server availability or uptime. Availability might also focus on failover and recovery test scenarios, with detailed metrics around the load balancer.
Scalability commonly focuses on removing bottlenecks or ensuring that a server can be scaled up, or a web site can be scaled out while ensuring that systems are sized correctly and remain cost-efficient.
Reliability might be focused on the application software itself, making sure that the correct responses are returned to the user from the server, for example.
Security can touch on all scenarios around performance and are often part of nonfunctional testing requirements. Load testing, in combination with application penetration testing, is an excellent way of identifying and exaggerating common vulnerabilities or even simulating Distributed Denial of Service attacks, for example.
Load testing helps you explore these characteristics of a system under test, in varying load conditions.
For example, you might be interested in observing performance under high load conditions as measured by concurrent users or response time. Perhaps your test scenarios are more focussed on seeing a load balancer in a failover condition while monitoring availability. Or maybe you are using load testing to identify and fix reliability issues such as performance bugs. Other software testing scenarios might include simulating or re-creating performance bottlenecks, as observed in real-time analytics platforms. Load testing can also be used as stress testing to find at what point your system will break. Sometimes load testing acts as a form of regression testing where you can validate your performance requirements have not slipped or fallen below previously agreed targets. Other times load testing helps confirm or check performance as measured by nonfunctional testing requirements. Load testing suits many purposes as a discipline. As a result, we see many different testing tools and test scenarios conducted on Flood by many varied audiences and user groups.
Load testing is an essential part of modern development and operations as it helps teams build and deploy software with confidence in production environments. The impact of not thoroughly load testing your products is often experienced as a loss of customers, revenue, or even integrity and trust in your brand. Having a website launch fail, or a critical business component unable to scale to meet demand due to bottlenecks, or a customer-facing, revenue-generating part of your system was not available is detrimental to any business.
Hindsight is a beautiful thing, and most who have experienced the hard-learned failure in front of customers or their business certainly wish that they had started load testing earlier. There are never scenarios of load testing too soon.
Not to worry, though, it is never too late to start load testing, and Flood’s cloud-based testing platform makes it easy to start. In the many varied software development lifecycles, there are as many diverse opinions on when to do load testing -- shift left, shift right, early, late, production, non-production, end of the release, throughout the release, linked to CI/CD, the list goes on. The reality, though, is you should be load testing at a time that makes sense to your requirements, and Flood’s cloud-based testing platform enables you to do this whenever it suits your needs, with load generators that can be provisioned on-demand.
While there are many styles of both, both in effect cover similar goals, and sometimes the goals are evident in the title. Load testing implies a simulation of a load profile. Performance testing suggests measuring performance. Stress testing indicates testing something until it is stressed or breaks. We often use the terms interchangeably but tend to prefer the term load testing, as it ties more strongly to the intent of simulating load and measuring for performance (or scalability, or reliability, or availability, the list is long!)
At Flood, we have a broad cross-section of industries that conduct load testing. IT, Finance, Services, Retail, Government, Media, Communications, Utilities, Gaming, and even Construction industries all tend to load test.
Typical drivers for load testing within these industries include one or more of the following reasons:
- need to service a large number of concurrent users or high demand
- need balance load or confirm correct operation of a load balancer
- need to balance costs or provide capacity planning
- need right-sizing or scaling of infrastructure
- need to control the operational expenditure of IT resources
- need to conduct any form of website load testing traffic at scale
- need to quantify or qualify nonfunctional testing requirements or performance metrics
- already have a high risk or historically weak performing applications
- are simply concerned with the performance characteristics of their applications and underlying systems
A load test at Flood is anything that uses JMeter, Gatling, Selenium, or Element (Puppeteer) to define user behavior at either the protocol level, for example, HTTP, or at the browser level, much the same as a real user interacts with your application or site. You can upload your load test scripts as scenarios.
Once you have a load test scenario defined, you can run these on our distributed load generators. At Flood, we support over 15 geographical regions in multiple cloud environments, including AWS and Azure, as well as the option to run from your on-premise or other cloud infrastructure using Flood Agent. We place no limit on how or where you choose to run your load generators, and it is entirely up to you.
Once you have your load tests up and running, our testing platform provides real-time information about your load test that you can easily share with colleagues who may also be interested in real-time analytics from your website load testing. All of our reports include a variety of performance-related measurements to help you analyze the impact on performance metrics.
Load testing tools are the software that you use to affect load. The majority of load testing tools do this at the protocol level, whereby you can simulate making HTTP requests over the wire. These tools also include the ability to parse the response from a target server or application, as well as use content from the response to assert a value exists, or re-use values in a future request.
More recently, Flood has been a strong advocate for simulating load at the browser level. While this approach trades in complexity for concurrency, when applied with the economy of scale that cloud-based website load testing provides, customers, can realize the benefits of browser-level load testing tools. Load testing tools such as Flood Element, based on Google Puppeteer, simulates realistic user behavior in the browser, which helps circumvent much of the complexity and high technical skill required to get started with protocol-level load testing tools.
At Flood, we are unashamedly big fans of open source testing tools. When we talk about the best software testing tool, we like to frame that in terms of picking the right tool for the job.
We pride ourselves on remaining tool-agnostic and support several leading open source tools, including Apache JMeter, Gatling, Selenium WebDriver, and now our tool, Flood Element. All of these tools have a robust open source community.
JMeter and Gatling are purpose-built performance tools that generate load on the HTTP protocol level. By far, JMeter is the most popular tool on our cloud-based platform, but numbers alone don’t signify the relevance or importance of a load testing tool to a particular customer. For example, some of our customers favor Gatling for its domain-specific language, or ability to write tests in Scala. JMeter is popular with many customers for its wide range of supporting JMeter plugins, as well as the many blogs, articles, and even videos on how to use this software testing tool to its best extent.
Flood Element is our open source derivative based on the prevalent Google Puppeteer library, which itself provides a high-level API to control headless Chrome or Chromium. This approach helps you simulate load at the browser level. Browser level load testing means you can reproduce most things you can do manually in a browser. If it’s in a browser, Flood can load test it!
Flood Element includes the ability to generate screenshots, crawl a Single Page Application (SPA) and reproduce pre-rendered content such as Server-Side Rendering (SSR). It also proves immensely useful in simulating realistic user behavior within a browser such as forms submission, UI testing, and even keyboard and mouse input. The test scripts for Flood Element use TypeScript which gives you inline documentation and script validation before pressing launch, to catch the small issues which otherwise prevent test scripts from running during your load tests.
In the past, load testing was the domain of specialists, with a focus on skills needed to simulate not only load but also the necessary in-depth experience to identify, tune, and fix performance bottlenecks.
While performance engineering is a vocation itself, in modern teams, we have seen the evolution of performance testing to include not only specialists but generalists in the field as well. At Flood, many of our customers have no performance engineering background at all. Many customers fit into what is now called Site Reliability Engineering, yet many more fit into more traditional Operations or Development roles.
At Flood, we advocate for all customers interested in load testing, by helping them circumvent traditional barriers to getting started. With a focus on open source, providing simple ways to provision load generators and quickly arrive at analyzing results, we believe load testing is for everyone.