New technologies often require new approaches for testing. The internet has grown to service more than 55% of the world’s population. One of the reasons for the incredible growth in internet usage is streaming of news, live sports, series, movies or music.
HTTP Live Stream (HLS) has become the industry standard protocol for delivering video content across the internet. Migration from legacy protocols to HTTP has increased since broadband and mobile connections became faster and content distribution networks became wider.
If you are releasing an application for Audio or Video streaming you will certainly need to load test it with a realistic traffic pattern. Doing so will ensure you are able to deliver your stream smoothly without interruption as users join your streaming event.
There are several ways to approach load testing a streaming application:
- For those with load testing skills, Apache JMeter or Gatling are the best option for testing HLS, as they can provide the greatest efficiency of threads per load injection node/server
- For those that are new to load testing Flood Element or Selenium may be an easier option to learn quickly
Testing with a browser is typically an easier option that provides more realistic metrics, but it comes at a cost, especially for extremely high load scenarios. These browser level tools require a larger server footprint per virtual user, so the cost per each virtual user will be higher.
Given these additional costs, it makes sense that JMeter is the most popular tools for testing video with high load. Even though it may be the most popular tool, there are several key issues that can come up when load testing video streams with JMeter which users should try their best to avoid:
Pitfall #1: Failing to test for live content demands
Live streams typically have lower quality and have a single language than on demand streams, due to the need to support a much larger volume of concurrent viewers. If you are going to test on-demand streams - these typically have higher qualities and the possibility to switch audio languages. So, it is good to simulate a representative mix of these as part of your scenario. For live streams it is recommended to simulate higher volumes of users or sudden spikes of connections and disconnections.
Pitfall #2: Forgetting to test for the various devices supported
Desktop users usually have access to a higher speed connection and higher resolution screen, and consequently demand higher video resolution than mobile users. If your application supports both Desktop and Mobile devices, we typically advise customers to test 40% Desktop and 60% Mobile. We also can adjust the workload to 80% of threads to retrieve their most popular device option i.e. Full-HD. The remaining 20% can be set to retrieve all video options randomly, to ensure all options are working properly before release.
For further information, please check out this great article on how to simulate bandwidths in JMeter.
Pitfall #3: Not thinking about your cache
One of the questions we are most commonly asked is whether you should enable the cache in your JMeter test. We believe that you should follow what is typically seen in production and enable caching for your test. To ensure a realistic test result, make sure you set the cache elements reasonably, i.e. 6 seconds for high definition quality files usually means a 2 MB cache. Keep in mind the server resources needed to support this for load testing. If you run with 1000 threads on the server, the memory needed would be around 2 GB.
Pitfall #4: Not considering audio requirements
Audio is almost always attached in the video pieces that you will be streaming, but many companies forget to test it. One major piece we see overlooked is how to test the support for multiple languages requiring extra audio files set in the manifesto. Additionally, keep in mind that your audience might switch the audio language midway through the stream.
Pitfall #5: Not looking at Latency
Latency is the worst enemy of HTTP video streaming, even with CDNs being introduced to fight it, it can still sink a stream. Generally, we should see video files of 5 or 6 seconds. For our test, we can download in parallel 5 or so video pieces which can be a good buffer of roughly 30 seconds. Simulating the entire video playtime in this manner can be a tedious task, but it’s required to calculate for the TPS or RPS we require for our testing.
Writing a JMeter script for load testing your high concurrency video stream won’t be easy, but testing it well before users do it for you is necessary. Looking out for the most common pitfalls will make sure you don’t fall into the traps that have tripped up countless other customers along the way.
Once you have gotten your test created, make sure you check out Flood to scale this to your peak concurrency scenario. We can help you to simulate your streaming with millions of concurrent users on JMeter and Gatling. By using the cloud, we can get you up and running with your streaming load test in a matter of minutes.
Please don’t hesitate to let us know how we can help you to accurately test your streaming service with realistic load tests run through our distributed network of load generators.