This post is a preview in a multi-part series from Nicole van der Hoeven's API Load Testing ebook.
API load testing may sound daunting, but we aim to make it simple. API load testing is one of the easiest ways to get started with load testing.
In this post, we’ll give you a run down of everything you need to know about load testing. Let’s start with the basics.
What is an API?
A quick Google search will tell you that API stands for Application Programming Interface. In simple terms, an API is a set of rules and instructions for how to communicate with an application. These rules are necessary so that the application server can understand requests and process them accordingly.
What does an API do?
When a user accesses a web app through a browser, his clicks on the page and other interactions in the browser are sent as requests to the server. A request could be as simple as this:
GET: http://flooded.io/order/10001
This request could be the browser’s way of communicating with the server that it wants to retrieve the page on flooded.io corresponding to order #10001, for example. Because the request is in a format that the application server understands (it’s in its API), the application server replies and responds with the resources for that page. So the API defines a language and syntax that clients (such as web browsers) need to use in order to interact with an application.
Why do we need to test API’s?
Most people understand the need to test via the user interface of an application, but testing via the API is just as important. Imagine what would happen if the request above returned order #100 instead of #10001, for instance. Though the user interface may seem to be acting completely normally, we could have serious security and business implications. This is why it’s a good idea to do some functional testing at both the API and UI level to ensure that the application is returning the expected responses.
Why do I need to load test my API in addition to functional tests of the API?
Let’s say you’ve done your functional testing and it’s all passed. Your app goes live with much marketing fanfare, and hundreds of potential customers start to trickle in. Then— you realise through your monitoring tool that your customers are encountering errors when trying to view their orders. But didn’t you test that already?
See, even though you’ve tested that the request itself works, you also only tested it with one user at a time. The missing part of this puzzle is testing the API with more than one user, to make sure that the API is able to scale to multiple concurrent tests. Enter load testing.
What is API load testing?
Load testing is applying the traffic you expect to see in production systematically to your application servers in order to determine how the application behaves before it goes live. It's a way to check whether your application is robust enough to handle the load you want it to handle before your users find that out for you. API load testing is going through this process by simulating multiple users sending requests to your server at the same time.
There are different ways that you can run load tests, but API load testing is one of the easiest methods and most cost effective options. That’s because it’s simple to script and there are a lot of industry open-source tools that you can use to generate and execute your scripts. This is part of the reason we created our own Test Builder-- it’s a simple interface that allows you to quickly start a load test without needing any coding experience.
What API load testing will NOT show you?
API load testing is a powerful tool that we recommend all teams should add to their testing toolkit. However, it is not a silver bullet and should be used as part of a comprehensive test strategy. Here are some limitations of API load testing:
- API load testing does not simulate real users interacting with elements of your webpage. In contrast to a customer opening up a browser and navigating to your site and filling out forms, an API load testing script will consist only of the underlying requests that are being made by those on-screen elements and sent to the application server. Exactly how the customer triggers those requests, and what buttons they’ve pressed, is irrelevant. It’s all about the raw requests.
- API load testing doesn’t give you an idea of how user-friendly your application is. It doesn’t measure front-end performance or how quickly pages render in different browsers. While all the resources that the server returns can be downloaded by your API load testing tool, there is no browser on which to run them. This means that any javascripts or AJAX that runs on users’ browsers are not executed. This includes scripts like those required to trigger Google Analytics, which prompts many an engineer to question the results of an API load test due to the inability to see the traffic come in on Google Analytics.
- API load testing isn’t what you’re looking for if your application is a single-page web app that relies heavily on client-side scripts to dynamically populate and update the page. If that’s what you’re looking for, you’d be better off looking at running browser-level load tests using tools like Flood Element, Selenium or Tricentis Tosca.
What API load testing WILL show you?
Now that we’ve talked about what API load testing isn’t good at, let’s talk about what it IS good at. API load testing has a lot of benefits when utilized in the proper context.
- API load testing can be a great way to simulate users at a low cost. This is especially true if your application already has a robust API. Since most interactions on the browser-level get translated to requests anyway, generating load this way is incredibly efficient, allowing you to hit your server with requests while bypassing the UI layer and the resource overhead that that entails.
For example, at Flood we’ve baselined some of the tools that we support to see how many users we can run on an AWS m5.xlarge instance:
Selenium (UI Level): 5 users
Flood Element (UI Level): 50 users
JMeter (API Level): 1000 users
Gatling (API Level): 1000 users
You’ll see that the browser-level test tools, Selenium and Flood Element, can run significantly fewer users on the same sized node as can the protocol-level tools, JMeter and Gatling. This isn’t due to an inefficiency in the tools but rather to the fact that browser-level tools inherently need more CPU and memory to start separate instances of browsers per user and render pages graphically. You can run more users per node by carrying out API load testing with protocol-level tools.
This efficiency translates directly into cost savings because every one node that you don’t need to execute your tests on is one fewer node that you’ll have to pay to provision (whether on premises or in the cloud).
For this reason, API load testing is one of the most cost-efficient ways you can get started with load testing, allowing you to scale up your load relatively cheaply while getting immediate results.
API load testing will generate load that is a good representation of load you could expect to see in production. This gives you an accurate estimate of how your application architecture will behave and highlighting any performance bottlenecks that can be fixed prior to go-live. Given the narrowed scope of API load testing, it is great at pinpointing even the most minor of degradations in API performance.
What Makes for a Good API Test?
Astute readers may be asking at this point, What makes an API load test good? I’m glad you asked. In short, it all comes down to planning. Stay tuned for the next post in this series!