In normal times, the New Jersey Economic Development Authority (EDA) issues around 300 loans over the course of a year. But these are extraordinary times. With $15M in grants and loans suddenly made available through the State’s COVID-19 response programs, the EDA needed to build new apps to process loan and grant applications, fast. The apps had to be built quickly (in a matter of days), and needed to perform rapidly (e.g., process 10,000+ applications per hour).
How does a government agency immediately scale up from apps that issue 300 loans in a year to apps that could handle more than 30X the average annual load in just an hour? They call in the state’s Office of Innovation: a two-pizza sized team of self-proclaimed “tech nerds” that thrive on high-pressure situations like this. More officially, the team “works in partnership with the Governor’s Office and State agencies to create innovative policies and technologies that address complex public problems by working differently: [they] apply data science and public engagement to understand problems, and use the latest digital technologies combined with strategic policies to develop solutions that improve services while also reducing costs for the State and the taxpaying public.”
Predicting unprecedented demand
The EDA’s first COVID-19-response program to launch offered small businesses $5M in grants, up to $5,000 per grant. To support the program, the Office of Innovation team rapidly built an app with the following components:
- A form-based front end collects and stores user data, then sends it to Microsoft Power Automate.
- Microsoft Power Automate takes the raw data, integrates it with data from other agencies (for example, to check if the applicant is in good standing with the Department of Labor or has any taxation delinquencies), then sends it to the legacy system.
- The legacy system, a homegrown integration with Microsoft Dynamics, performs additional processing according to business rules.
- If the grant is approved, the applicant automatically receives a PDF to sign and specifies where to send the money (via ACH).
This was a first-come, first-serve program, so they knew the composite application would need to handle a high volume of concurrent users. How many? That was the (five) million dollar question. To get an idea of what they should expect, Ross Dakin and team looked at Google Analytics data for a wizard that helped people determine what business assistance programs they were eligible for. Based on that, it seemed safe to assume that the application would need to support 3,000 concurrent users.
Ready for launch
Could the app support that volume of traffic? The team had selected a vendor with a high degree of confidence, but there was no room for ambiguity. That’s when they decided it was time for load testing.
With load testing top of mind, the team stumbled upon an article about companies offering free software to governments and non-profits working on COVID response projects. The article mentioned a load testing tool: Tricentis Flood. A few calls were made, and soon Foulk (a Tricentis partner) was walking the team through the load testing process. Load testing revealed that the app could actually support a much higher volume of concurrent users than they anticipated, and no further tuning was required.
Mission accomplished
With this insight – and several cool-looking reports that showed how much traffic the app could support without slowing or failing – everyone was confident that the development was complete, and the app was ready for go-live.
When the EDA grant program launched, peak traffic reached nearly 8,000 users per minute – double what they expected – and the app still exceeded performance expectations. In the first 76 minutes, 10,000 grant applications were submitted. By the next day, more than 26,000 people had applied.
A week later, EDA launched a program to distribute $10M in 0% loans. This benefited from the insights gained in the previous program, and within three days they received submissions from over 3,000 organizations requesting a total of a quarter billion dollars. The following month, EDA plans to run yet another program, this time 10x the size of the first. They are confident in the system’s abilities to handle the unprecedented load, thanks to the testing performed by Foulk using Tricentis.
The demand for both the loan and grant programs was quite a change from processing 300 loans over 365 days – but everyone rose to the challenge.
“You never want to try something new when the stakes are high, but we had no choice given the scale of the COVID-19 pandemic; we had to create new systems that could scale better than our legacy technology,” remarked Ross Dakin. “The Tricentis and Foulk teams helped us validate that the new systems we created could indeed handle the traffic generated by so many New Jersey small business owners seeking assistance. I’m now completely sold on the value of load testing (and so are our executives) – next, we want to get baseline performance measurements on all legacy systems and anything net-new that we create. No more unknowns.”