JMeter and Gatling are two of the most popular open source tools on the market today, with each tool commanding thousands of loyal followers. Here at Flood, we have a wealth of customers using either (and in some cases, both) tools.
Gatling vs. JMeter?
While our customers will frequently ask us which testing tool we prefer, the truth is these are two tools that accomplish a similar aim (performance testing or load testing) with a much different approach. Which tool you should use will depend on a variety of factors including (but not limited to) what apps you are testing, what your skillset is, and how you want to run your tests.
In the spirit of helping our customers compare and determine which tool is right for them, we’ve had our experts compile some of their biggest pros and cons of each testing tool below to help you in your research:
JMeter
Overall, we tend to see JMeter adopted as a close replacement to LoadRunner with a robust UI and plugin/support ecosystem to allow for performance testing of a wide array of protocols.
Pros:
- UI for less technical users (could be viewed as a con to technical users)
- Robust support and training ecosystem
- Proven history with frequent major releases, lots of contribution
- Expansive array of supported protocols and plugins
Cons:
- Requires use of UI, no scripting capability (could be viewed as a pro to non-technical users)
- Less SCM friendly
- More resource intensive (bigger deal for massive scale tests)
Here is a deeper look at the top 3 reasons we love JMeter:
History and support
One of the major arguments for using JMeter is its long history. It was first created in late 1998, which means it's had nearly 20 years to iron out any lingering defects and establish itself as one of the most reliable, mature load testing tools around, open-sourced or not. The JMeter project on Github has seen 105 releases and 15,722 commits to date - significantly more than Gatling. Its user base is also quite active. Nearly anything you want to do in JMeter can be solved by a quick Google, and there are myriad tutorial videos, blogs, and even online courses built around JMeter.
What isn't built into JMeter by default can be added by way of plugins. Pretty much anyone can create plugins for JMeter, and there's even a Plugins Manager to easily browse and install plugins. Independently-run JMeter Plugins lists 72 plugins. By contrast, Gatling has less than 20 plugins listed.
GUI-based scripting
While Gatling is pure code and can be daunting for someone new to it, JMeter comes with a user-friendly interface and, for simpler scripts, is mostly drag-and-drop. This means that it's faster to learn in the first place as well as easier to share among a team, as it's easy to see the structure of a script at first glance. Debugging a script is also easier due to the interface, which allows you to switch between the raw response and the rendered HTML. JMeter does still support code via the beanshell and groovy samplers, but they're by no means necessary. In testing as in life, simplicity is key.
Extensive Plugins with Many Protocols supported
JMeter supports hundreds of 3rd party plugins that extend the functionality JMeter out of the box. JMeter is able to handle some of the more niche protocols that Gatling currently cannot support. Some of the most glaring omissions on Gatling's side are TCP, FTP and JDBC which might drive a user to pick JMeter over Gatling for those specific cases.
Gatling
Overall, we tend to see Gatling adopted as a continuous load testing tool that is a larger shift from legacy tools and more often leveraged by those with technical skills
Pros:
- Easy hooks into CI/CD pipelines
- Ability to create completely scripted tests without a UI
- Less reliance on plugins and 3rd party add-ons
- Better efficiency for massive scale load tests
Cons:
- No UI (might not be a con if all team members are technical)
- Mindset and resource shift, potential retraining from HP LoadRunner (could be a pro for teams looking to change)
- Smaller open source community, fewer contributors and training materials
Here is a deeper look at the top 3 reasons we love Gatling:
Easy integration into CI/CD Pipelines
Gatling is actually built & developed using Travis CI by the Gatling development team. Gatling project structure as well as the fact that you typically use an IDE to edit a DSL as code, instead of working in a GUI when writing your scenarios lends itself to being very CI/CD friendly. There are also plugins available for popular CI tools such as Jenkins.
Gatling has strong integrations like JMeter, connecting to popular Build automation software and IDE's. Additionally, there are plugins available for leading build tools such as Maven and SBT. Eclipse and IntelliJ IDEA are supported as popular IDE's for script development.
Full Scripting with Powerful Assertion Language
Scala is the scripting language that forms the basis of creating Gatling scenarios and projects. Scala works everywhere and was designed to be concise. It is built to run on the Java platform (JVM) and is compatible with existing Java based libraries which can aid in advanced scripting and complex load testing scenarios.
Gatling comes with a really nice assertions language that is purpose built for undertaking functional checks under load. An example is being able to verify that a particular transaction takes between a given time range i.e 1-5 seconds. This is particularly useful if you are measuring transactions against pre-known SLA requirements.
Purpose built performance architecture
Gatling uses the Akka toolkit in which it's core architecture is a fantastic fit for a load testing tool as it implements an "actor-based concurrency" model. This concurrency model doesn't use a "user-per-thread" approach like other tools - instead it uses actors that interact with each other directly. Therefore it has the capability of running multiple users per thread, which may result in slightly more efficient running of high load scenarios as we benchmarked here (link: https://flood.io/blog/benchmarking-jmeter-and-gatling/) .
Conclusion
JMeter and Gatling are two of the most popular open source load testing tools on the market, and we love both for their unique strengths. Teams will need to assess their current applications, skillsets, and existing infrastructure to determine which tool fits the need for each test. In the end, both tools can rise to the challenge of load testing the most used apps in the world, so you can’t go wrong investing in JMeter or Gatling. The success of your load testing initiatives will likely depend much more on the quality of the people doing the test and the load testing approach leveraged than the specific load tool selected