Here at Flood, we are asked every day for replacement options for HP LoadRunner (now owned by MicroFocus). While LoadRunner has been the leading load testing tool in the market for many years, it has lost favor with many teams over the last few years for a few key reasons:
- Cost: HP LoadRunner typically costs in the mid six figures for an average size deployment. If you are looking at a large deployment, seven figures is not out of question.
- Perpetual Licensing: typical LoadRunner licenses are still perpetual and difficult to scale up and down
- Decreased Need for Non HTTP Protocols: with the migration of many packaged applications to the cloud, companies have less need for the wealth of protocols that LoadRunner can support
Replacing HP LoadRunner with StormRunner
HP, being wise to the changes in the market, decided to introduce StormRunner as their own LoadRunner alternative. The main aims of StormRunner were to provide a solution that had:
- Native cloud integration
- Support for popular open source tools like JMeter
- More flexible, consumption based licensing
StormRunner quickly became a popular replacement solution to customers that had become disillusioned with LoadRunner.
Comparing Flood vs. StormRunner
StormRunner and Flood both rely on a similar set of features in order to deliver a seamless testing experience in the cloud.
Shared Features Between StormRunner and Flood
Both tools offer a robust set of cloud functionalities:
- Browser based load testing (TruClient for StormRunner, Element for Flood)
- JMeter, Gatling, Selenium Integration
- Analysis Reporting
- APM Integrations to tools like AppDynamics, New Relic, and Dynatrace
- Ability to execute tests in a shared cloud
- Ability to execute tests in a private cloud account with AWS or Azure
- REST API for integration to DevOps Pipeline
StormRunner Specific Features
StormRunner offers some unique features not found in Flood, including:
- Support for a wider variety of protocols (non-web like SAP GUI)
- Ability to use LoadRunner tests
- Sitescope, an dedicated monitoring solution to use in place of AppDynamics, New Relic, and Dynatrace
Flood Specific Features
Flood offers some unique features not found in StormRunner, including:
- Host your own AWS support (soon to be deprecated in StormRunner)
- Ability to use Tosca tests (UI and API) as load tests
- Ability to write tests in the web with Test Builder
While the pricing models for StormRunner and Flood are similarly consumption based, they do vary in that Flood is truly consumption based (on the # of servers used to test) vs. StormRunner is based on the number of Virtual User Hours consumed (which is an approximate representation of the # of servers used).
Protocol Level Comparison: JMeter
Looking at the cost of an identical test, which might be a one time test with 20,000 users in JMeter for 1 hour, the costs would be as follows:
- Flood: 20,000 users for 1 hour would require 20 nodes (load injectors) for 1 hour, which comes to a total cost of $149
- StormRunner: 20,000 users for 1 hour would require 20,000 virtual user hours (Web), which comes to a total cost of $4,500
Verdict: Flood is 30x less expensive than StormRunner in this Protocol Level scenario
Browser Level Comparison: TruClient (StormRunner) vs. Flood Element (Flood)
StormRunner and Flood are two of the only solutions on the market that offer support for load testing via real browsers. This technology is great for measuring full stack performance as users would really see it, and is often times the only way to test technologies like AJAX or Single Page Applications.
Looking at the cost of an identical test, which might be a one time test with 5,000 users in TruClient/Flood Element for 1 hour, the costs would be as follows
- Flood: 5,000 users for 1 hour would require 100 nodes (load injectors) for 1 hour, which comes to a total cost of $499
- StormRunner: 5,000 users for 1 hour would require 5,000 virtual user hours (GUI), which comes to a total cost of $3,200
Verdict: Flood is 6x less expensive than StormRunner in this Browser Level scenario
Conclusion: Choose the Best Choice For You
Both StormRunner and Flood offer a robust set of features that are great for agile teams looking to do testing in a cloud enabled offering. Since each team has their own requirements, it is important to compare the different criteria here and asses which solution is right for your use case.
If you are interested to try out either product, you can trial either product here:
- StormRunner: free trial that allows for 10 tests with up to 100 concurrent users
- Flood: Free trial that allows for unlimited tests for up to 5 node hours