Worksoft Certify Reviews

Initially we were automating the regression suite for SAP ECC.

From there we moved into a web application called HVAC Partners, which is something that we've developed that is a type of customer portal. That application also connects to SAP, but it does some other things that don’t necessarily connect to SAP. It is a kind of front end for quotes and sales orders that go into SAP, but it's also reporting status of orders and status of warranty claims and the like for the customers.

From there we moved into the Middle East SAP ECC instance and automated their regression suite and, from there we rolled out S/4HANA for our service business. With S/4, SAP releases updates every quarter. Because the S/4HANA instance is in the public cloud we have two weeks, essentially, to regression test and test any new functionality. We started with the last release and we did about 70 percent of the testing with Worksoft. We also used the S/4 automation tool, which is more for unit testing so it's not as valuable as Worksoft. We're wrapping up that automation in about the next month and we'll be moving on to a European rollout of S/4. We'll just start working our way across Europe and those implementations.

We have it on a virtual server and we're using remote desktop access for the offshore automation engineers to access it.

We're using it with Fiori and it's working fine. We have integration in and out of S/4 to Salesforce.com so we also automated those. The test cases were end-to-end. We start in Salesforce, which is a web application, with, for example, a quote, and then it goes into S/4 and gets reviewed and approved. It then goes back to Salesforce with the approval and a sales order is entered that ends up going back to S/4. And then there's fulfillment, back and forth, and eventually billing and collections. We were able to do that whole automation with Worksoft, plugging into Salesforce as well as integrating to S/4 and doing the S/4 automation, back and forth. It's been incredibly useful. We saved something like 80 percent of the time it would have taken to manually test, using this tool.

In terms of using the Capture feature without knowledge of testing tools, we brought on some new support people. One of them is our web support person and she had no background in Worksoft. She's been using it to do all the initial captures for our HVAC Partners. She's been able to use it very easily. Our more experienced automation engineers will follow up, after she's done the Capture piece, and troubleshoot some of the stuff that she might not understand yet. They're working with her so that she does learn it. But she's been able to use it very easily.

Worksoft’s ability to build tests and reuse them is very good. We ended up obsoleting the tests and not using them with the other tool we used, whereas now, we rerun these, at a minimum, every month. We do that for a few reasons. One reason is to keep the health of the tests up. Suppose a material is obsoleted. The test that has that material in it is going to fail because it's going to say, "Material not found." Or suppose a customer is no longer a customer and he has been blocked or archived. We run the tests to make sure that the scripts don't need any changes. We also use them in case a process has changed. We're releasing changes to SAP about every two weeks: support tickets, enhancements, maintenance, etc. If a business process changes, then the automated test needs to change to reflect that change. Running them every month, at a minimum, helps make sure that everything is healthy.

The other reason is to identify anything in our quality system that could unintentionally impact other things that the programmers didn't realize. We've caught a couple of those in queue and they said, "Okay. I didn't mean to do that. I only meant to change this one thing,” but it changed all kinds of things and we were able to catch that before it went into production. So the reusability is fabulous if you create the tests properly: no hard-coding, and you’re using data tables to hold any of your field selections, and you're using good automation standards, so you create and consume your data. If you create it and consume it, when you rerun it, it does the whole thing again. You don't have to worry about finding a sales order that works, for example. You really have to create a logical test design to make it reusable but as long as you do that, it's very reusable.

It dramatically reduces the time we spend on testing. Before we started using this tool, everyone was pretty much doing testing manually and test events were taking from two to six weeks. What they did in two to six weeks, depending on the scope of what they were doing and how many people they had involved, we can usually do in one to two days.

The most dramatic was when we finished the Middle East automation. They were bringing up another company code and they wanted us to run regression testing on all of their current company codes, about seven of them. We completed it in about four days. The IT director came to us and said that it reduced their labor by 93%. “Quite frankly,” he said, "we would never have been able to do all of that testing. We would have had to engage a minimum of 28 people, and it would've taken them a minimum of eight weeks, and we still would not have been able to do all of the tests. We wouldn't have gotten them done." We were able to do it in a fraction of the time and with a broader scope than they would've been able to do. They would've done as much as they could and then they would have gone live and hoped for the best.

And we've also been able to use it for other things like certain recurring tasks that had been done manually. We had people who were manually monitoring Tidal jobs, which are batch jobs that have been scheduled to run. If a Tidal job fails, somebody has to go in and figure out why it failed and either restart it or fix it, and then rerun it. These are jobs like billing jobs and we automated them. They probably spend 15 minutes a week on billing jobs now, whereas we had somebody doing this about 12 hours a week. And then that person would have to send out an email to whomever the relevant person was saying, "Hey, check your batch job. This isn't running." They now spend about 15 minutes running it. It saves the emails to the users, documents the results in a spreadsheet, and puts it out to a SharePoint where the auditors can pull them any time they want. It was the same thing with monitoring the claims jobs. We've done a few things like that which have added to the value.

Automation using Certify has also saved testing time, big-time. As I said, the Middle East: 93 percent. For the S/4HANA project, what we did in three or four days, they had been taking two weeks and not getting through at all. With the release, you don't get to say to SAP, "Hey, testing is running behind, we need another week," because it's in the public cloud. Like it or not, they're going live. The drill is supposed to be: You test during week one and you remediate in week two and you go live that weekend. We got our stuff done, 70 percent of the work, in about three days, and it was our first time, out-of-the-gate, so it'll go easier with the next release. The rest of the team took the entire two weeks to do their 30 percent. And within the 30 percent they were doing, a lot of them were smaller tests. We were doing end-to-end tests that go through Salesforce and S/4, etc.

In terms of defects, the value is finding the defects prior to moving something into production. There are two I'm thinking of that we found in Mexico. One of them would've brought shipping to a halt and the other one would have brought receiving to a halt. If you shut down factories, even for a short period of time, there is this domino effect. The value of those finds is huge. And this wasn't even something that the guys making this change were testing. They were testing the piece that they changed, which was working. What they didn't realize is that they changed all items instead of just that subset. It was a minor goof in the programming. It was just too broad of a statement. 

I started in IT about nine years ago and we did total manual testing. We would have defects in the high hundreds to 1,000 during the implementation testing. Now, we're probably under 100, so it's much lower. It could be that we're just getting better at implementations.

We use Worksoft Certify to test our SAP System. We have a global instance of SAP, which we started implementing in 2012, and we are still in the process of implementing. We have rolled out SAP to about 80 percent of our manufacturing and distribution. Right now, the remaining projects are a small distribution center and sales offices. We have ongoing projects, and three times a year, we release a new version of SAP. We rolled out SAP to a new geography, and we also added new features for our business users. Thus, as part of those projects, we use Certify to do regression testing of our existing business processes, and we also use it in the project to test new functionality.

When we are rolling out in a new country, we do a configuration for that new country. We use the automation test to test the business processes and prices of that new country. It is sort of semi-automated. Our business analysts generate sales orders from the new country, and we will run them through the shipping orders to cash, the shipping steps, and the concrete steps. Then, we get a set of documents to review. The business analysts review those documents to make sure the order is processed correctly. So, it's not fully automated, but it does help cut down on testing a lot when we roll out to new countries. 

For regression testing, that is fully automated. We have tests where the software checks the results and either returns a pass or fail. These are run as a regression suite anytime we push a change to production.

We do use it for the end-to-end testing of packaged applications, primarily SAP. We do have some plugin applications that we use it to test, which are part of the business process. We use Salesforce for CRM, and we have a custom built eBusiness application. While we don't do extensive testing of those applications using Certify, when the business processes touch one of those applications, we do cover those application with another certified test.

It has cut down on the amount of low level, grunt work that business analysts have to do and can free them up to do more critical thinking. Before we had test automation, we were running tests and relied on people, which was very time consuming. A business process test might have 100 to 150 steps across different applications, and we don't have a single person who has expertise in all those applications. When executing a manual test, we have to balance the test between different people to do their steps. In a typical project, we might have 100 to 150 of these types of tests running. The coordination of the testing process where you have to have different people available at different times is very time consuming and inefficient. What automation has done is cut that cycle dramatically because automation does not have to worry about having to find the right order management or warehouse person to do their steps. The automation just runs through, then the business analyst can review the results afterwards. Therefore, it has been more efficient, cutting our testing part down by almost two-thirds to 75 percent.

Our automation tests are more robust than our manual tests. We found our test lab would grow over time because we didn't have a lot of discipline within the team for manual testing to have a master test which could be used repeatedly and revise as necessary. So, they were creating a new test for every specific little thing that they wanted to test. They were  setting up these manual tests where they had ten to 15 tests which tested the same thing, but not quite. Therefore, it became a bear to manage. Whereas, with automation, because it is more controlled, we have a core set of about 125 automation tests entered into our library. That's in our change control. Therefore, we know exactly what the state of our tests are. 

If there is a new business process or new wrinkle in a business process, we didn't have a defined process, so now we are updating automation tests. The quality of the data that we're getting out of test from automation is much higher than we received out of manual testing. If we know the automation suite is parsing, then the application is working properly. With automation, we have more confidence that if the test is parsing than the application under test is working correctly.

We have automated quite a number of test cases in last year.

We have a weekly release. A weekly release means every week that we have testing going on. The particular year where we started is called the asset management area. We could never run our Intuit testing, and we have a whole lot of Intuit testing. This one product helped us pass the testing with the Nighthawk testing, which is working on the Nighthawk manager. That's the one that we use. We can switch it on in the night and run the testing, then come back in the morning and see what has been completed. If there is any fail, we can even analyze it. We can use the evidence document to pass it on to development team to tell them, "This is where we failed."

It has improved on our defect management time. It has improved our test execution time. I don't need to manage these things, just sit somebody down to look at how the script runs. There are a lot of ways that it has helped us.

We have been working on test automation for SAP-based platform for the last 15 years and leveraging Worksoft for SAP test automation in last 10 years. 

We have some fantastic use cases on how we can leverage Worksoft in large SAP transformation programs, especially for major global rollouts. If you look at some of the industries, like retail and logistics, we can see a lot of consolidation. When there is consolidation, we need to merge two SAP landscapes to a single landscape instance. It is a very complex transformation program. The testing is labor-intensive work in such a large scale IT landscape program. Worksoft has played a key role in delivering these complex integration programs on time, because we automated the core processes using Worksoft Certify. This helped us to automate the testing scenarios in new companies being integrated. At the same time, we test the existing company codes on the existing landscape to ensure that business as usual is not impacted.

If you look at the industry trend, the next six or seven years will be SAP S/4HANA migrations and adoptions of SAP cloud applications, whether it is SAP SuccessFactors, Ariba, or Fieldglass. We strongly believe that WorkSoft can play a pivotal role in delivering large-scale SAP S4 HANA programs since Worksoft can support the implementation of SAP cloud applications. We have delivered a couple of major SAP S/4HANA programs by leveraging Worksoft Certify. 

One of the key things that we have done is introduce test automation on day one in system integration testing (SIT). We don't want to wait until the program is over, then start the automation development. We start the automation right from the build phase, then we are able to deliver test automation on day one to the SIT.

If you look at the customer's experience, we have seen that if you take any testing engagement, whether it is large-scale system integration testing activities, UAT, or regression testing, and have seen that companies involve their business workforce to do lot of manual testing. Then, the business needs to take from the business critical activities and focus on testing. Worksoft significantly reduces the testing effort involved, so they can focus on their business critical activities. 

Our primary use case is for SAP regression testing. We wanted to have our all SAP processes for regression to be optimized using Worksoft Certify. It took around one year for us to take that process and learn how it works, gradually improving it until now. 

We do regression testing on a weekly basis using Worksoft Certify. This does not include just SAP. We are also using the web-based tools when we are integrating end-to-end processes. When we are moving within one end-to-end process, we are shuffling between all the scripts and then going back to SAP. It works quite well.

At some point, we will have an RPA initiative to use this solution. For example, we need to obtain data from our development environment to check reports and verify them, then create transactions using Worksoft Certify.

We have more than 160 end-to-end processes. In our repository, we can have one transaction process or more than 40. There are big end-to-end processes and there are smaller ones. There are processes that can take hours to run and there are processes that can take a couple of minutes to run, but most of the end-to-end processes have multiple transactions. In weekly/monthly releases, we don't have any type of manual testing. We are just relying on Worksoft Certify for its test automation.

We also use Worksoft Certify for test automation in projects.

For scheduling, we are using an Execution Manager from Worksoft Certify.

Our business and IT users have been able to use the Capture feature, even without knowledge of testing tools. We have different IT people from each business area, like production planning or material managements. These people, without even having any knowledge of programming or Certify, just need to know how to use Capture. They are using it because it's really easy to use. You just have to initiate a Capture and do the same work that you are doing every day. Then, Capture will record everything.

You don't need to do any coding when creating end-to-end processes; it's a codeless because it is just recording. We don't need a specific person (or multiple persons) to work with the processes to do coding for the automation, which saves time. It increases the confidence of the process owner or IT person working on it because we provide them the capability to record their own process without doing any coding and just using Capture. They are confident that whatever they have recorded is their own process. It's not someone else recording their processes. The test automator can polish that process, but we are not recording it since they initiated the Capture. Overall, this saves time.

It has reduced our test maintenance time by more than 50 percent because we don't have to do manual test processes. We have saved over 150 man-hours monthly. It has increased our delivery times. We went from 200 man-hours down (three weeks work time frame) to approximately 40 man-hours (three days work time frame).

Using Worksoft Certify, we have found and repaired more bugs, though not that many. Users cannot manipulate information during the process. We schedule our tests at the night time and leave them working. Next morning, when we come into the office, we just check our system. These tests are pass and fail. For the failed test, we have screen captures where we can go and check the screenshot to know what the issue is. No one has to sit in front of the screen when you are running regression tests or scheduling them. I can schedule them and continue with my other daily routine work.

Our primary use case is test automation. We have an SRP solution called One1ERP on an ERP platform. We started the automation in Worksoft Certify, getting more than a thousand test cases automated.

Nowadays, we also have automation for a web application in our HR area, so all our HR processes on the platform, Pega. This started to be automated since November last year. We have 40 test cases automated on this application in one year. 

Improvement means for us that we have to be better in quality. Due to automation, you can run every automated test case twice a week. If you do it manually, you do it once per release. This is a quality improvement.

We always have to be more efficient. E.g., if we can directly automate test cases, so if we are using older test cases, then by just switch the company code (national company), we don't have to do it manually. We are able to directly automate the test. This saves money, which is always important for our top management.

We use Worksoft Certify for testing and non-testing. 

• Under the testing umbrella, we use it for regression testing, disaster recovery, validations, and system refresh testing. This has been our main focus. 
• Recently, we have started getting more requests for more RPA-light type of work, which is not testing. It is using the same skills, process knowledge, and tool sets to do work that would replace manual repetitive tasks with automation.

We use Worksoft Certify for some basic ServiceNow functionality or Workday releases, weekly and quarterly releases. For SAP Hybris and ECC, we are also using it for our internal security protocol testing. So, we test Office 365 and Windows 10 compatibility. We test some Excel functionality and file sharing, as part of our security protocols. The most in-depth end-to-end testing that we have is in SAP.

We have done a lot of manual testing. We still do a lot of manual testing for our projects. We've eliminated a significant amount of manual testing with our system refresh, automation, and for technical upgrades where changes are known. However, for projects with new enhancements and functionality, we are having a slower time penetrating into them. With regression testing, we have completely replaced it with automation. 

Now, we are trying to shift as much as we can to start automating processes earlier in the project lifecycle, but it has still been a bit of a challenge. This is one of our stretch goals for this next year. The non-testing area is where we have had the most growth over the last six months.

The freshest example is some of the RPA-light activities that we have done. We have been able to save on a lot of manual work for some very highly skilled, expensive resources. This has been able to free up a significant amount of their time so they can spend more time on innovation and more creative, value-add activities. That's been one of the more rewarding things that we've done, and the most appreciated. 

We run our tests and our full suites every week. We have them scheduled in batches so certain sections of our scripts run every day, then we run them through the whole suite every week. That is how we maintain them by running and repairing them.

To run them every week and make repairs takes us maybe two hours. Because they are scheduled on batches to run, it's lights out. They are pretty hardened at this point, so there are not a lot of repairs for data, etc. If there is a break, it is usually because there has been a change to a process that we were not aware of. Our automated scripts are the company's best business process documentation, as we don't have a business process management tool. Therefore, if anybody wants to know about our business processes, they come to us for something like training or new production support resources. They will come and watch our automation if they want to understand how the systems work.

Its primary function is test automation. We record the test procedures and execute them in an automatic way. It gives us reports, in a centralized way, of all the information that we need to manage test automation for the ERP system and a few other systems as well.

We moved from manual regression testing, once a month, to continuous testing every second day, with the same scope or even an increased scope now. We don't need to focus on the manual regression tests, so we've freed up hands for support activities and new developments. I don't think that it has changed how our organization works directly, but we have become more efficient and more flexible.

In our estimation now, we save about 160 hours of manpower every month because we use Certify. It's 80 hours of robots executing the tests, and they're doing it roughly three times faster than humans. To do the same manually, and keep up with the general pace of development, we would spend 240 hours on testing every month.

In terms of codeless, end-to-end process automation across packaged applications, technically speaking there is some programming or development, like you would learn in middle school. But you don't need to be a professional in any programming language to run it. That's how we've done it. We don't deploy any additional software packages with it. We haven't developed anything else, other than using Certify steps to run end-to-end test automation.

Nevertheless you still have to have a mindset of a developer or an engineer. You need to have understand cycles and other primitive functional blocks. Still marketing-wise, I would call it codeless due to its simplicity. 

SAP is our core and the people involved with it are not always developers. Some of them are analytics or system experts. They don't know how to write C++ or Python or Java. But they can do testing automation in Certify themselves without help from a centre of competence (CoC). The only time CoC will participate is when truly complicated logic required such as runtime variables staking.

In usual situations, the skill set to start doing test automation needed is much lower.