Integrating Flood with Azure Devops for Continuous Testing

Starting March 2020, load testing will no longer be part of Azure DevOps. So what can you do? Well, here's a way that you can integrate Flood into your Azure DevOps pipeline. All you need is the included Powershell script and a Flood account, and you can start load testing every build of your application.

Azure Devops is an evolution of Microsoft’s VSTS (Visual Studio Team Services) and the latest effort to develop an efficient platform for building and delivering products.  This powerful platform includes the ability to track development progress, manage testing efforts, automate delivery pipelines, and more.  However, starting in March 2020, one key feature will be missing - load testing.

Fear not - it is easy to integrate cloud load testing scenarios into your Azure Devops pipelines using Tricentis Flood’s API. This customizable integration combined with Flood’s Explorer dashboard enables you to identify whether your most recent code changes have resulted in a performance improvement or regression - with traceability down to the exact build!

Before You Start - Required Prerequisites

We have made it straightforward to implement a Continuous Load Testing capability with the combined usage of Azure Devops and the Flood platform. The following prerequisites are required to set up this capability:

  1. Github Repository - for your application code as well as the Powershell/Shell and test scripts to be located in.
  2. Tricentis Flood account - the API token is required to launch Flood’s against your web application.
  3. Azure Devops account - to be used to run your build Pipeline and associated load tests
  4. Azure Pipeline - this feature of Azure Devops is used to build your application and run the scripts to launch and verify your load tests in Flood.

Step 1: Setting Up Your Github Repository

Azure Devops supports Github where you can store not only your application code but also the load test assets (i.e. JMeter, Flood Element, or Gatling scripts) required to run continuous load tests.

Additionally, we’ve included the Powershell/Bash scripts required to launch and verify load tests.

The Github repository used as an example for this article is located here: This contains all you need to run Continuous load tests using Azure Devops. Feel free to fork it for your own testing purposes.

This repository holds the following files:

  1. - simple instructions on what this repository contains.
  2. azure-pipelines.yml - the YAML script that outlines the steps that our build process will follow.
  3. startflood.ps1 - the PowerShell script that launches the load testing infrastructure, and executes the Dummy JMeter script in a new Flood.
  4. verifyflood.ps1 - a PowerShell script that monitors the Grid infrastructure while it is initiating, and monitors the Flood execution as well as verify the results to signify an overall PASS or FAIL condition.
  5. - an (alternative) Bash Shell script that monitors the Grid infrastructure while it is initiating, and monitors the Flood execution as well as verify the results to signify an overall PASS or FAIL condition.
  6. JMeter & Element scripts - for this example the JMeter script (containing a Dummy HTTP sampler) is used however you can use any support test script depending on your requirements.

Please Note: You are able to use PowerShell for both starting a Flood as well as verifying test results. Azure Devops supports both PowerShell and Bash Shell scripts usage.

Step 2: Setting up a new Azure Devops Pipeline

Signing up for a new Azure Devops account is a simple process and currently includes a 12 month free option to use their most popular features including the Build Pipelines we use in this article.

Once signed up and logged into the Azure Devops platform - you will need to create a new project. We’ll create one called FloodBlogDemo as follows:

Once the Project creation process has finished we’ll automatically be sent to the Project dashboard - we’ll need to access and select the Pipelines menu option.

Under the Pipelines menu, click on the New pipeline button.

In the first step of the process for creating a new pipeline - we need to connect the Github repository where all our code and load test assets live.

Choose the GitHub item from the list of options.

Since we have already allowed access for Azure Devops to my GitHub account - a list of recently used repositories will be listed as follows:

Choosing our purpose-built flood-azure repository will allow this Pipeline to use the contained test assets automatically as part of the build process.

Note: If you have not setup an Azure Devops integration with Github before - you will be redirected to an authorization page asking you for permission for Azure to access your Github repositories in order to do things like Checkout’s etc. as part of the Pipeline process.

The final step before the successful creation is to review the YAML file that outlines the steps that this build process will take each time it is run.

Each line describes the conditions and steps for the Pipeline to run with - we can go through these to better understand what will happen when the Build Pipeline executes.

  1. trigger - this step outlines under what conditions this Pipeline will trigger. In this case it will trigger any time changes are made to the master branch.
  2. pool - this step allows you to specify properties of Build agent pool. In this case we are using Ubuntu Linux latest VM image which is the default option.
  3. steps - under steps we can include some pre-build steps. This can be used for example to post to a Team chat platform (such as Slack) to notify users that a build has been initiated.
  4. task - these steps are where the heavy lifting occurs. We have two PowerShell tasks - one task that launches the load testing infrastructure and runs the test itself. The second task monitors the infrastructure being launched as well as the Flood execution itself. It also verifies the results of the Flood and raises an overall PASS or FAIL pipeline condition depending on how the test goes.

We are able to run this Pipeline and this can be done immediately by clicking on the Run button. However there is some further setup required before this will run successfully.

We need to edit the Pipeline in order to add a variable that the scripts use to interact with Tricentis Flood specifically.

Click on the Edit button on the Pipeline main view.

We need to add a variable that contains our secret Flood API token - this can be retrieved from your Flood account in the API Access menu from the Flood user profile dropdown menu. Copy paste the API token value into Azure Devops as follows:

The variable name should be named flood_api_token and it is very important that the value should be kept a secret.

We also need to add a flood_uuid variable which keeps track of the unique Flood identifier which is used in the scripts as well.

We’ll need to tick the option for letting users override this value as well.

Once these variables have been added - click Save and then Run the Pipeline:

This will initiate a manual Build process taking all the steps in the YAML file and you will be able to see the results for each step as it takes place in real-time.

You should see something like the following once the Pipeline is active:

You should also be able to login to Flood and see the test running while this is happening.

Once the Flood is finished - you should see the end result of a complete Pipeline Build process that has dynamically launched load testing infrastructure, run a load test, and verified a simple SLA and marked the build as PASSED or FAILED based on those load test results!

Step 3: Verifying the success of your Continuous Load Test

There isn’t much point in running a Continuous Load Test if you can’t tell if it’s successful or not. That’s why in our Bash Script - we have implemented a simple check to base the overall PASS or FAIL test result based on the amount of failed transactions we observed.

We have some logic in the script to query the load test report via the Flood API and retrieve the mean error rate. We then verify this value is 0 to indicate no failed transactions were observed and hence a successful build.

You can extend this logic to verify things like response times against a business requirement such as an SLA value.

Putting It All Together

Flood and Azure DevOps are a winning combo to build your applications quickly and with performance.  By integrating the two together, you will have immediate feedback on the impact of each code change to your applications ability to scale.  We’d encourage you to take advantage of our example scripts to get Azure Devops integrated to your flood free trial instance.  You can head on over to Flood to sign up for a free trial which will include 5 free node hours as well as access to our API’s.  Simply follow these instructions to get your trial integrated to your Azure Devops account and start load testing with each and every build!

Start load testing now

It only takes 30 seconds to create an account, and get access to our free-tier to begin load testing without any risk.

Keep reading: related stories
Return to the Flood Blog