It's primarily used for automation. We're a pretty complex environment. We have hundreds and hundreds, if not thousands, of applications. We have more than 200 Agile teams which are doing builds across five big locations in the US.
We have five large business units: financial, property and casualty, enterprise applications, marketing, and emerging business. And then we have a shared-services organization. Across this, we have more than 200 Agile teams that do build work. A lot of the time, these agile teams focus on developing and testing the work that has been handed to them. When Tosca came in, one of the things we started thinking about was how Tosca could help us facilitate some end-to-end testing.
When I say end-to-end, that doesn't mean in one particular business solution area or in one particular department, but rather, how do we go across departments? If we have to create a retirement plan, the work is not just in the retirement area. It has to flow from a lot of different applications and different business units, and that facilitates the end-to-end.
The way we are organized right now is by department. Initially, our scope was to see if we could do end-to-end, but we reduced the scope because that would have meant that many people had to be ready at the same time to consume the work. So what we said is that if we can use Tosca to do end-to-end for an application, then we can use orchestrator tools like Jenkins or Concourse to create an end-to-end flow from a business perspective.
To give you some numbers, we will be harvesting a saving of almost $1 million, in the first area that we tried this. We had 36 manual testers and we were able to go down to 14 quality engineers, so we are seeing some savings. That was the biggest. In other areas, we are seeing savings from reducing by three or four people.
In terms of cases covered by testing automation with Tosca, it's very difficult to put a number on that. Where Tosca has really made a difference is where we had manual testing only and the percentage was zero. In the area that I just mentioned, where we went from 36 to 14 testers, they were at zero percent automation and they're already at 40 percent. The goal is to be 80 percent by mid next year.
Out of our 200 teams, we have not finished all the waves yet for Tosca. Around 50 percent of them are already using it, if not more. In January of 2018, the number of associates we had doing manual testing — I'm not even talking about contractors — was around 370. Our projection is that by the end of next year, we are hoping to go down to 230 associates. That's about a 37 percent decrease in the test-analyst workforce. Most of that is going to be enabled via automation using Tosca.
We have seen lead-time change impacts as well.
We are seeing some defect numbers dropping down and we are still operationalizing. The biggest is Speedplay, where we were doing testing manually. Now, with automation, we are able to execute some of those regression tests sooner.
Tosca enables us to run the entire regression test suite immaterial of where changes have been made. We now have much more confidence in the areas where we've implemented it. We have a high level of confidence in the changes that we are making because we know we have a regression suite that we can run.
We have seen workflow improve. Before, we wouldn't have thought off coordinating among different applications. For example, application A does their testing and tells application B, "Here is what we need." Application B goes into their cycle and they do their work and say, "Okay, we're ready." That might have taken a day, two days, five days, or a week. Now, we have examples where we have been able to directly call the database for application B and retrieve the information. What used to take six days, we are able to do in six hours.
One more example where Tosca helped a lot recently was when there was an issue where our retirement plan holders were seeing incorrect information on their PDF statements. There were 32,000 PDF statements that needed to be validated after the fix was done and it would have taken 60,000 hours for one person. So we had a couple of our folks create scripts. It took them two days and they executed all the 32,000 PDF validations in one day. Tosca didn't only help us in terms of time, because it was more an issue of cost-avoidance, but it helped us gain the trust of business because business signed off on it. Business was the one which had said, "We need to validate each and every PDF manually," and that's where that estimate had come from.
We are seeing a lot of these success stories. We expect, once we have the full implementation done next year, that we'll see many more examples.
Overall, by next year, we will be looking at a total reduction in testing costs of about $14 million, across the span of three years. That includes both build and run. Build is very difficult for us to harvest because if we reduce the money in build — if we take away two people — the demand grows and they add two developers. But overall, from a build and run perspective, we are looking at forecast savings of $14 million. This year alone, we have proved that we have reduced the overall spend by $3.5 million.
Another big use case which where we have been helped a lot is through Tosca BI, which helps with large-volume validation from a data perspective. For a lot of our data lines, we have an automation framework, but it doesn't do as well when it comes to comparing huge volumes of data from source to target. That's a place where Tosca BI has helped us because it can do those large file and data comparisons in a very short time.
