Worksoft Certify Reviews

Primary use case is for SAP support testing.

People focus on what they actually want to test and define it in a more detailed way. It shines a light on what they are testing, along with the speed and adaptability.

• Reusability
• Scheduling ability
• The modularization of testing: It brings a more functional level, as opposed to the technical level.

A feature that I am looking forward to in version 11 is a search capability, where you can search within the script themselves for keywords. That will be really helpful.

More than five years.

It is stable from an application standpoint. Sometimes, there is complexity within the virtual machines that they run on, or the hardware, depending on the configuration that their client has setup. Sometimes, this is a bit of an issue to overcome, but I don't see this as a Worksoft Certify issue.

If done properly, it's very scalable. It creates automation which can be run from an SAP point of view. I can create automation which runs in any SAP environment that I want it to run in, as long as the UI is the same.

Their technical support is very good. They have different aspects of technical support:

• If you're standing up a project, they can be with you every step of the way. 
• When you just submit a ticket, they'll respond to you and get back to you within their service level agreement.

Worksoft Certify's codeless scripting appeals along with the use of the Capture feature which helps in those initial phases. It also helps to translate the business requirements to the automation team. If there's a separate team, this is a little better too.

There is recognition in the industry that automation capabilities, like Worksoft Certify, create value for companies, as things are only getting harder and larger. Companies are integrating systems to try to align their processes.

The initial setup is relatively straightforward. The team helps guide you. The technical support is there. It comes down to the training that's applied, and having the people who understand the use of the tool.

The typical range that we see is between 50 to 90 percent improvement in speed capabilities. Another aspect comes with the depth that they're able to test. It increases their capabilities overall to test at a broader level.

Our customers often evaluate Tricentis along with Worksoft.

From an Accenture point of view, our customer chose Worksoft because of our alliance with them. It's the capability that we've built and the knowledge base that we bring along with that.

Capture 2.0 is good for new implementations or new processes. It's not as useful when you get into more mature automation. Its documentation and automation are fairly straightforward.

There is a bit of a learning curve around the Worksoft Certify tool. The best practices, which are lined out by Worksoft, as long as those are followed, then this leads you to understanding the tool and using it in a proper way. If you don't get started off on the right foot, it will be hard to course correct. So, it is vital that they get started on the right foot and understand the best practices. The product's learning curve is relatively good.

Automation is only as good as the functional knowledge that is used in order to create it. This tool works extremely well. Manual testing and automation testing are two different animals. You have to look at automation in a different way. Simply taking manual scripts and automating them, and you're not going to get the full value out of a solution like Worksoft that you could if you were to rationalize the testing and come up with an automated approach.

When you're manually testing, it's about having the least number of clicks possible. Every click for a human is time. With automation, clicks don't cost anything. You might approach the testing in a different manner. It would take a human multiple times longer, but with automation, it makes sense not only from that specific test case point of view, but also from a reuse factor. When you're going to use a certain business process that you've created, then reuse it for different work streams.

The primary use case is more on SAP for automating all our manual business work, as well as doing regression testing. We do end-to-end regression testing on SAP.

The product has worked very well with SAP.

Worksoft is used for manual processes that needed to be automated. This has helped us make our processes be faster and more efficient because there are no manual interruptions or errors.

For my processes, Worksoft makes them faster when creating scripts.

Capture 2.0 is a very good feature, on which we can record very easily and get documentation generation and testing acceleration. 

The product had some UI issues. In the next release I heard the UI issues will be lifted up (version 11), I am excited about it because the product will have more UI features. We are thinking of upgrading our existing Worksoft Certify from 9 to 11, when it is released, as this will be good for the company and help all our users.

The stability is pretty good what I have used sofar with SAP

The scalability is good.

The technical support is pretty good. If you open a ticket, they are able to answer it within 24 hours. They are able to support us.

We previously did manually testing and used other tools. With Worksoft, we can more see the productivity and benefits that it provides. We can also do more testing, making easier for all the users. 

We used the Worksoft for the deployment. They came and helped us. We have a good relationship with them.

Worksoft Certify is an investment. We see value in it more than the money. We see value in it because the user can look at different aspects of testing and what they can do.

This solution has enabled us to automate in order to tremendously save time. Only first time when you are recording and creating the script will you spend some time with it, the rest of the time, you are just executing. If we do one manual process, it could take approximately two hours. The same process using Worksoft probably takes ten minutes.

It help cut test maintenance time too. An eight hour manually project using Worksoft can be done in an hour.

Worksoft is good for SAP to do your automation and testing.

We are not using it for web UI testing.

The primary use case is regression testing and test automation.

We use this solution for end-to-end testing of packaged applications, like SAP, ServiceNow, and some in-house web apps (which we built).

We have already found some defects using the test automation. 

One of the bigger value-adds that we had was extracting data from our warning systems to be inputted into our new learning system.

We automated because of the value. Our business users do a lot of regression testing. They often don't have the time when we need to get the testing done, so this frees them up to do more value-added work for the company.

• Speed, as far as getting tests completed on time. 
• Its ease of use and the value of that. It is a lot easier to maintain test scripts on Worksoft Certify than on other testing tools that we have had in the past.

We have had run ins with some bugs on Business Process Procedure (BPP) and Execution Manager. 

We also did upgrade Worksoft Certify recently to clear up some issues with server fogging.

More than five years.

It scales pretty well.

I would rate technical support as a seven out of ten. Sometimes, we end up going back and forth with Level 1 support. Usually, if we go back and forth, it is email most of the time, and sometimes it is once a day with 24-hour turnaround. We usually understand what they require and trying to give as much information as possible. We still need to tell them to kick it up to Level 2 support or at least have a meeting with them, so we can show them what is going on. Sometimes, it does get a little frustrating, but once we either have the meeting or get Level 2 support, the issue gets resolved pretty quickly.

We had and still use QuickTest Professional from Micro Focus a little bit, but we were approached back in 2008 about getting into Worksoft Certify.

We went with Worksoft Certify because of its ease of use. You don't have to know a scripting language, like with QuickTest Professional. Also, it has ease of maintenance.

The initial setup was not really complex, just a little. We did not understand the full setup; it was just confusing. We really didn't get a full explanation of how things connected together until we actually started to use it. Once, we start using it, things became much clearer.

During the initial deployment where we received execution suite, the necessary information was not really provided. However, it was a fast deployment. We had to meet some deadlines, so people just came in and did the installs, or helped us do the installs over the phone, then afterwards we realized how things were connected.

We used Worksoft for the deployment, who was very good.

We have seen ROI by being able to free up and give time back to the business for other value-added work.

In 2008, we evaluated SAP TAO. The person who came in to do the proof on concept wasn't able to get the test scenario working at all. He had three and a half days to do it, and he couldn't do it. When Worksoft came in, they had the same length of time and were able to get that one plus another one partially working in the same time frame.

Worksoft generally seems to want to make sure you are successful at what you are trying to do. I haven't come across an employee from Worksoft that isn't willing to help. A lot of times you do get that from other salespeople, and that is just not the case from what I have seen from Worksoft. 

The product has a lot of benefits as far as getting testing done. It gives you some value back.

The company is using it across our IT landscape on different projects. We use it to test web applications, SAP mainly. Right now, we're doing a big conversion project. The one that I am working on is a big conversion from an old mainframe system over to SAP. Thus, we are using it to automate all the UAT test cases.

We have also used it in other areas of the company for ServiceNow upgrades and general web design stuff.

For this SAP upgrade or implementation, the business users are creating building blocks in their areas, then they're changing the building blocks into long end-to-end scripts to do a complete end-to-end testing to speed up the UAT. It definitely has a bigger coverage of UAT testing.

• It is user-friendly. We can give the tool to business users, and they're able to use the tool pretty efficiently without a whole lot of training. 
• The reporting features are nice. 
• It is easy to take someone else's work and manipulate it to use yourself. 
  
• It is not heavily code-based, so you can pick it up and automate very quickly.

For Execution Manager, I would like it to be more robust interface and be able to view the remote machines full screen instead of a little window. This would be a great upgrade for us.

I would also like more customized reports without having to print out big reports.

It is a very stable product. When we have had upgrades, it's been pretty seamless. Older versions of our scripts work in the newest update without a lot of rework.

It is very scalable with users. We have business users who are scripting. We have Worksoft developers who do more complicated work. Then, it moves over to the people who do execution and process through Execution Manager, so we have several different layers of users doing different tasks.

The technical support is very good. They are very responsive. When we have created tickets with them, there is usually a 24-hour turnaround time, then we are contacted back. Their interface is good for back and forth communication.

It is not practical to do manual testing anymore due to the volume of it. The amount of data variations is too great to test manually. If we were going to do it manually, we definitely would not have full coverage, where now we can get pretty close to full coverage on our tests.

When you are implementing Worksoft Certify, it is probably best to bring some Worksoft consultants in to help get you setup, and set it properly. Because if you don't get your initial systems, like your folder structures and your naming conventions, set at the beginning, there is a lot of rework to get it to work. We did have to go through that.

We save probably 50 percent of our time. The tool does what it's supposed to do, and we are able to actually work.

Worksoft Certify is a good product. The customer support is really helpful and supportive. They are always upgrading their products to new features, which we like. It is a pretty stable tool, which doesn't require a lot of maintenance.

Our environment has SAP Fiori. They are also doing a HANA implementation. As far as the web, I don't work on that side as much.

I haven't used the Capture 2.0 very much.

My company brought on Worksoft to assist us with an SAP upgrade in order to make it so the testing was less stressful for business users through automated testing.

We configured Worksoft to setup our SAP testing environment after a refresh by regenerating all of our testing users. This sets them up with the right access, the right approvals, and everything on the back-end so we can do our scripts the next night.

The Capture 2.0 feature recording the process automatically as you go through. It saves everyone a lot of time. It allows the business to give IT the process they go through without having to spell it out so it can be recreated as an ongoing test.

There is a learn functionality where Worksoft learns applications that would be nice if Worksoft expanded its support for other applications that aren't web-oriented.

We have had some stability issues just getting stuff setup. It seems to be pretty stable outside of the web UI. We have had some issues with our testing running into nightly backups of database backups, etc. That has caused some issues, but when we get everything ironed out in a nice, controlled environment, it seems to be pretty good. The web can be a bit finicky sometimes, but it's just that the response times aren't always the same. So, it's a little harder for it to be resilient.

The scalability is really good. We had just talked with Worksoft about if there were any limits in terms of how long your scripts can run for or how many can be running at one time on the database. It seems the only concern is the hardware that you are running it on.

We ran into some issues with the version that we were on during the initial setup. We ran into a bug on one version, then they upgraded us to a new version, and we got hit with another bug. So, they had to put us in a beta. That was a little frustrating. However, besides the bugs that we ran into, the install was pretty straightforward.

It saves us time. It makes it so we are not as worried about changes which are going into the system. We know that we have nightly runs to ensure that things are working. In general, with upgrades, we can always rely on the testing to make sure that certain business processes are working. If they do stop working, we know when/where and can tie that back to changes in the environment easier.

Our eCommerce platform is Hybris. We run end-to-end tests where we place orders in Hybris, then we validate the order in ECC. Additionally, when an order is placed on Hybris, our QA environment has a lot of things which the SAP analysts have to prepare to get an order ready, so it doesn't clutter up the system, such as creating deliveries. Worksoft can do this for us as well.

Hybris is out most modern application. Our point of sale system is web-based, and it is in web form. We are on Azure. One of the things that we've been able to do is use Jenkins to put our Azure machines on business hours. We tell them to turn it off at 5 PM, then we tell them to turn it on at 7 AM. This has saved us about 62 percent of computer operations.

Prior to Worksoft, there were three different individuals within the company who worked collectively for about three years trying to automate just one of our smoke tests for our point of sale system. A lot of them got pretty far, but they weren't able to finish. However, with Worksoft, we have been able to automate six of our smoke tests in four months.

The most valuable feature is having a no-code solution for automation, so our QA team and some of our business users can work on automation. Then, they don't have to be developers.

Most of our SAP analysts use LiveTouch. They use LiveTouch along with prebuilt components. Our QA team uses LiveTouch when they need to add things.

I would like to learn how to get better logs for their support team.

It is fairly stable. We have run into some intermittent bugs off and on that we can't explain. Since they are typical Window's stuff, you just kill them.

When it is unstable, there will be times when a test that we are running in Certify will just stop, and it will say, "Aborted." There will be errors. There will be no explanation as to why it happens. It has now happened maybe one out of 20 times. When it happens, I just tell our QA team to stop Certify and restart it, hoping we don't see it again.

Scalability is throttled in part by the system that you are testing. So, how much testing can your system handle? 

I think they came out with a different type of licensing specifically for testing. Therefore, you don't have to use a more expensive user license, you can use an automation license. So potentially, if we had 100 use cases, we could spin up a 100 different machines, have them all run and be done in five minutes. That would be the goal, but I don't know if that would actually succeed or not.

For the couple of the issues that we were really scratching our heads over, we were in communication with the technical support several times, but they never got back to us. The issues are not critical because they're not really blocking anything. They're just annoying.

We needed a first step in order to get into DevOps. The first step was being able to automate our smoke tests and regression tests. They are tests that we use to make sure that our SAP environments are viable and our point of sale system. We chose Worksoft because they were the only people who we could find which were capable of automating SAP right out-of-the-box.

We needed a faster feedback loop. We have a third-party who develops our Hybris application for us and wanted to be able to hook into their Git repository, so when they push a new version, it would automatically deploy and run our smoke tests. Then, I can know within ten minutes if it works.

The new environment was pretty straightforward to set up. There were four servers, and maybe a fifth one, if you wanted to have a separate server for automation testing.

Some of the integration depends on the subject matter expertise on your team. How well do they know ECC and their processes? Then teaching them how to use Certify to build out their processes. So, on a scale of one to ten, it is probably a seven if you are not familiar with some of the development principles, like looping. If you are not familiar with them, then it will become more difficult to build out processes needed. This is just understanding the methodology of doing certain things, not Worksoft specifically.

The IT department and I worked with a Worksoft deployment engineer because of all of our Worksoft infrastructure. She walked us through setting up the database in SQL and the MongoDB with Worksoft Analyze. It was a pleasant experience. Most of the issues that we ran into were because I did not know something.

Because we haven't built out our suite of tests yet, we haven't saved that much time. However, we know that it will allow us to save a lot of time and money, because once we are fully DevOps, we'll be able to spin up and spin down our systems on demand. Then, we will know within 30 minutes whether the system deployed successfully or not.

We did not evaluate any other vendors. We didn't find anything else that did what we wanted.

if you can use Azure or AWS for your Worksoft infrastructure, then use that for ease of deployment. Once you have your environment, then you can save it using Infrastructure as Code. Thus, if you needed to rebuild or repurpose it, you would be able to do it.

We haven't taken advantage of all the current functionality.

We hardly use the Capture 2.0 feature at all.

Worksoft Certify is being used to run automated weekly regression tests across some of our primary SAP systems in line with our Change and Release management strategy. These tests run every weekend without fail. The results are reviewed on every Monday morning to check for failures and to analyse if any failures are associated with the changes scheduled to be transported to the production environment that week. Failures (if any) are fixed and the tests re-run before transporting the associated changes into the Production environment.

We also utilize it for projects that need extensive business-user testing and functional testing. There can be testing requirements which come at short notice which can take three to four weeks of manual testing effort. By using Certify, we have been able to bring timescales down to a few hours of automated testing effort.

Our final goal is to utilise this for 'Unify', our new global solution which is currently being deployed, which will deliver common processes and systems to all sites and sectors, replacing all our existing legacy systems which will demand extensive regression testing. 

We have never had any systematic regression testing regime in the organisation. This has helped in building an automation framework across our SAP application landscape, thereby introducing mandatory regression testing across all our key systems and improving the overall quality across our production systems.

From an audit perspective, results generated from Certify (BPP reports) provides detailed test evidence which is also being utilized for internal training purposes/training guides, etc. The BPP reports also provide details on failures along with screenshots.

We have a variety of complex systems in our landscape, one of them being the Openlink Endur which is a commodity trading and risk management system. We are currently building an automated regression test suite to support application testing for Endur.

Our weekend regression tests are performed in 'lights-out; mode. Tests are scheduled to run at a certain time over the weekend using the Execution manager functionality. Usage of Certify has also prevented some major defects going into Production and we have seen significant savings in all manual testing activities as the business users/functional teams are getting more time to perform 'value- adding' activities.

Post our recent upgrade to Solution Manager 7.2, we are currently in the process of implementing the Test suite functionality and the integration of the same with Certify. We expect Solution manager to be the single source of truth bringing out all the results from Certify which is going to be extremely beneficial from an audit perspective. We have already implemented the integration of Certify with HP ALM in our landscape.

Moreover, we have this reusable asset now which can be run frequently to support all our projects and change requests across our legacy SAP systems. Even last-minute testing requests are being accommodated by utilising the automated regression suite without any dependency on business users/functional users for their efforts. We use it across the multiple projects which need immediate assistance and for our weekly regression cycles. To give an example of a recent project which was a major platform migration from a Data Centre in Asia to Europe which needed extensive Disaster recovery testing and Functional testing/User acceptance testing. The initial testing estimate was approximately five to six weeks, however with the use of Certify we could do extensive testing in less than three hours saving many weeks of manual testing effort.

It is very user-friendly with an appealing UI, unlike a lot of other automation tools that we have evaluated. With sufficient training and adoption of best practices, the tool will certainly help organisations to successfully implement an automated testing framework and eliminate manual testing activities.

The fact that it can be used across SAP and non-SAP applications (including web-based apps like Web Dynpro) is a big advantage for us because we have a variety of SAP and non-SAP applications across the Johnson Matthey IT landscape. Being a 200-year-old organisation, our variety of legacy systems have a lot to benefit from the use of automated testing.

Certify has many interesting features, e.g.: 'PRIMO' which is the image recognition functionality is a life saver in instances where Certify standard functionality cannot identify and learn objects within certain legacy applications.

Regarding end-to-end testing of packaged applications, Certify is primarily used across our SAP application landscape and the Openlink Endur (commodity trading and risk management system). We hope to realise more benefits by implementing Certify across our wider application landscape over the next few months.

We have been using the Capture feature, although not the latest version, the initial version, for process captures was used to create our test designs. It has been a life saver in many instances, without the need to spend any extra effort to create test designs and captures. The test steps get captured in the background which generates an XML file which can be easily imported into Certify, creating the basic test structure which can be improvised/modified to make it a repeatable reusable test. In terms of the amount of time it takes users to create documentation automation using this feature, it is the same amount one would spend to do a manual test. While a person is performing a manual test, Captures are automatically generated in the background. We have used it extensively to build our test designs.

We have requested for some minor new features which Worksoft is considering.

The PRIMO image recognition functionality has room for improvement, especially around its ability to work with java interfaces, Execution manager scheduling, etc. as we have observed. As we explore more of our legacy systems, I am certain there will be a need to use more of the PRIMO features to learn the objects.

Overall from a SAP perspective, it works almost seamlessly.

One to three years.

The version of Certify that we are using has been mostly stable and we have rarely encountered any problems. Our weekend regression test failures are often associated with environmental/system performance issues and not related to the stability of Certify. I have been happy with the overall performance of Certify and how it has helped to optimise our tests.

I am confident that Certify can scale to fit our automation testing needs as we expand the current automation testing framework across the wider Johnson Matthey application landscape. We are also exploring options to identify potential areas where Certify can help support mass data uploads, etc. to benefit other teams in their day to day operations.

We have several concurrent users accessing Certify in our environment, primarily automation engineers, test engineers and tech managers.

Our interactions with technical support has not been the best always and there is room for improvement especially with respect to the time taken to respond to cases. However, with the right contacts and reasonable escalations we have always managed to get quick attention on our issues.

We have always been heavily reliant on manual testing and as a result, regression testing was not systematic and we could never think of implementing frequent weekly regression test cycles which was challenging. We decided to go ahead with automated testing and use Certify because:

1. Manual regression testing takes a lot of time and resourcing is always a challenge.
2. Regression testing not being systematic, the quality was very difficult to measure as we did not have a standard set of manual regression test scripts/sufficient documentation.
3. There was a delay in our time-to-market because all the testing was being done manually and there was no way we could accommodate frequent, weekly, regression test cycles. That meant high business risk, that we would have more defects in the production environment/ more associated costs.

We had all these challenges and we started exploring options to mitigate these risks and automation was identified as the way forward, nearly two years ago. We evaluated various automation tools in the market. It was critical that we had to identify a strategic tool which would cater to our SAP and non-SAP application landscape. Worksoft Certify came in as a big winner ticking most of our requirements.

We went through a lot of initial challenges, mostly around internal resourcing issues. Looking back, I am happy to say that we could overcome these challenges and have managed to successfully implement an automation framework using Certify.

Early in 2017, we decided to go ahead with Worksoft Certify post evaluation of multiple automation tools. Our initial engagements with Worksoft consisted of several onsite workshops to explore the tool in detail along with technical feasibility assessments across our application landscape. These engagements were extremely beneficial and it gave us the overall confidence to adopt Worksoft Certify as our strategic test automation tool.  

We did a pilot implementation with Worksoft to see if we could take this ahead on a large scale before embarking on the major project to build the automated tests. Some key processes across our critical SAP systems were identified as candidates for this exercise. Test designs were created with support from the functional teams and taken ahead for automation build with Senior Worksoft consultants and our internal resources. This 7-week Automation Roadmap Engagement exercise was extremely successful and we learned a lot of lessons from it which helped us plan the next big phase of the automation roll out. It gave us overall confidence across the functional and management teams which subsequently led to securing the appropriate budget, etc.

One of the biggest lessons learned from this engagement was around the ways to structure our teams. This led to us going ahead with a Managed Services model with Worksoft. We have an offshore based Worksoft Automation Services Factory team who helps build our automated tests. The team can scale up/down based on our automation forecasts.

The automation deployment is still ongoing. The initial phase was completed across a five-month span. Currently we are rolling out the second phase of the automation build focusing primarily on our global Unify solution and the Openlink Endur application.

Regarding implementation strategy, we followed an agile two-week sprint approach. Our functional teams continuously created test designs and these were fed to the Automation Factory every two weeks, who in turn developed the automated tests. This was the most practical model, which worked well in our environment.

At its maximum capacity, we have had approx. 10 to 12 automation engineers in the Factory team. Our functional teams are spread across multiple global locations and we had between 3 to 6 resources working on test designs liaising with the business users as required.

From a script maintenance perspective, we spend an average of 4 to 5 hours every week with the current asset of nearly 800+ tests.

We have always worked directly with Worksoft, along with support from our internal resources. Worksoft has been delivering excellent services through their managed services model.

We have a res-usable re-runnable asset built which is saving a lot of time across the functional teams/business user community.

Our final goal is to utilise this for 'Unify', our new global solution which is currently being deployed, which will deliver common processes and systems to all sites and sectors, replacing all our existing legacy systems, which will demand extensive regression testing. 

It is expensive compared to some of the other automation tools in the market. However, the benefits and ROI has proved that it has been a good investment.

We have concurrent licenses.

SAP TAO and Micro Focus UFT.

It was critical to identify a strategic tool which would cater to the testing requirements across our SAP and non-SAP application (including web based apps like Web Dynpro) landscape. Worksoft Certify came in as a big winner ticking most of our requirements.

It is a great product and we have not seen anything which cannot be automated till date in our application landscape.

It is important to do sufficient technical feasibility assessments before deciding to go ahead with Certify and equally important to determine the best implementation approach which will work for your organisation. Functional teams/business users' buy in is critical as the test designs cannot be created without their continued support. Adoption of best practices around naming conventions/folder structures etc. will help in easy overall maintenance of the test assets, which will also help with the generation of development and execution dashboards/overall reporting.

I would rate Certify at eight out of ten. Worksoft has always been very supportive and responsive to our needs and this has certainly helped us achieve our initial milestones successfully. I am extremely proud of what has been achieved so far and looking forward to expanding the automation framework across our wider IT application landscape over the months ahead.

We are using Worksoft Certify to enable our stakeholders to do test automation on the UI level.

We use it for end-to-end testing of packaged applications. We are part of the internal IT department within our company. Most of the time we are using it on our own products. The products and systems that we get are usually preconfigured and prepackaged and we do additional testing, not just for the functionality for the coding that we add to the product, but also on the prepackaged solutions.

We use it for all kinds of applications. Our focus is the web area, including web-UI testing of modern applications. We have two tools in place: our own internal corporate test automation tool, and Worksoft Certify. The latter is a complementary tool, especially in the web area where there are some white spots for our corporate tool which it cannot cover. That was the main reason why we brought in another tool. And for that, it fits perfectly.

We are able to run our test phases faster. Once the scripts, the test cases, are ready and automated, not only are we able to check our systems or landscapes during the test phases, but we can proactively monitor our development and test systems. Proactive monitoring of our systems is very important for us and was not possible before because manual testing is just too time intensive.

Worksoft Certify helped us to increase time savings. We didn't start test automation in general with Worksoft Certify. We did automation before with our own tool, but it helped us to increase the coverage of test automation and to increase the time savings.

We had a success story with two teams. For the execution of the scripts, we had time savings of 82, 88, 95, and 90 percent. And for the speed, it was between nine and 21 times faster than manual execution.

It's not necessarily saving us money, but it's helping us to free up the capacities of our end users to work on other stuff. Instead of doing testing for two days, they can work on bug fixing, developing new features, etc. That person still gets the same paycheck at the end of the month, so it's not saving us money, but it increases the value of our products. It increases the quality of our products. The reason for that is we are not customer-facing. We are dealing with internal teams and internal products. We are not selling anything to the outside. We are with the internal IT department. For the development teams and the sales team or the consulting team it might be different. But we are not really going out, selling our products and getting the revenue for the company. This is done by other teams.

It's script-free, which is really important for our end users because we are usually dealing with colleagues who are not developers and who do not always have the technical background of developing and scripting. It's very useful that there is a nice UI and the tool is script-free.

It covers all of the technologies we need to cover.

And one big advantage of Worksoft Certify is its integration with SAP Solution Manager, the test suite of Solution Manager, with the certification. That is crucial for us since Solution Manager is our test management tool of choice.

There are a couple of small things, technically, that could be improved.

Features we have asked for include single sign-on. It's a bigger project to make sure that our end users do not have to store passwords, usernames, and the like, for the different tools we have. 

We are also working on an additional integration with another tool that we have in place for lights-out testing. That's ongoing at the moment.

Another idea we brought is that the definitions for the objects need to be automated. They need to be recognized automatically by Worksoft Certify instead of changing them back and forth manually. This is also something that Worksoft is currently working on. 

Updates, in general, is a topic that we are working on with Worksoft on a regular basis. For new products, for new UI technologies when they come out, the test-automation providers need to update their definitions to make sure that the objects are recognized correctly.

It is pretty stable. After upgrades, we usually don't experience any big issues. Of course, it's software, so here and there we find bugs, but nothing crazy, to be honest. The availability of the system is pretty good, almost 100 percent. I don't see an issue here.

At the moment we don't have any issues with scalability. We have about 300 end users working with Worksoft. On infrastructure, it's split. We have a WTS environment, a Citrix environment, for those colleagues who want to use the prepared environment. We have other colleagues who are using the client on their own machines, on their own laptops or desktops. The only thing that we saw at the beginning which we need to change in the future is that, due to the latency, we cannot use clients in the US, for example, while having the server in Germany.

If the latency is over a certain number of milliseconds then it is basically impossible to do automation. That was one of the main reasons why we set up the Citrix environment at the very beginning.

We are still in the phase within our company, or within IT services, of training and spreading the topic of test automation, overall. So our coverage, at the moment, is not the entire organization, it's only the IT department. Once we have done this - and it will take at least another year - we will see if we spread using Worksoft and our internal corporate tool as a combination, or tool ecosystem, further into the organization. But this is not our not our team's responsibility so it's not really in focus at the moment. We are pushing for test automation in our teams and there is still a huge demand for training and new teams coming into the topic of test automation.

In general, technical support is good. They are collaborative and responsive. The only thing I don't like - and this is the only complaint I usually have for Worksoft - is that the first-level support is not always the best for working on topics. We sometimes need to escalate to second-level support and then we know that we are getting a colleague who is aware of the issue and is not just playing for time.

We already reported this to Worksoft and asked them to find another way or to educate the first-level support or to make sure that the tickets go directly to second-level support if they come from us. The guys on our end who are reporting the issues sometimes know more than the first-level support.

When it comes to second-level support, we are happy. There, we know we will get the help that we need. The colleagues are responsive and very helpful and, from a quality perspective, they are very good.

In the beginning, there were some issues with the integration, it didn't work the way we wanted. We spent some time with the Worksoft team, with the support and engineering team, in adding some enhancements to adapt the solution to our three-tier Solution Manager landscape. But that worked very well.

We have a very good collaboration and relationship to Worksoft. For example, every two weeks we have calls with them. We'll provide feedback and they take it seriously. They usually provide us with updates, with enhancements, with new functionalities that we need. That's working pretty well.

We started with manual testing and then we started the test-automation initiative. We started with our internal corporate tool and then, as I mentioned earlier, we figured out that we could not cover everything with that. At the end of 2015, we started to check the market. We did some PoCs and we decided to go forward with Worksoft Certify.

There were a number of reasons we went with Worksoft Certify. The Worksoft team did a great job. They came to our headquarters and did the PoC, showing that the tool is suitable for our needs. They did another PoC with our operations colleagues who were running the regressions testing in Singapore. And then there were the technical requirements that we had. Worksoft Certify was able to cover all of them, some of which I have mentioned already: Being script-free, being fully integrated in Solution Manager, and being able to script in a modular way. And finally, the integration between our own internal tool and Worksoft Certify was also important.

The initial setup was pretty much straightforward. I cannot give you too many details because I did not take care of the implementation. But I know that it took us about two weeks to set up the whole infrastructure. It was not really difficult. And we had very good support from Worksoft, from the support and engineering team. They helped us in setting up the database, setting up the connections. It was not a big deal. In total, within two to four weeks, everything was working fine.

On our side, we had a couple of staff members involved in the implementation because our team is the application owner. We had to involve two more colleagues from the database team because we don't have all the authorization stuff, for the databases area, for the servers etc. In total we had about three staff members involved, but not full-time. It was about one work-week for each of them.

Maintenance is done by myself and one of my colleagues, with the help of our database and server/infrastructure team. We don't have authorizations for everything and we are not database experts. There are three or four staff members taking care of maintenance, as part of our job; it's not a full-time job, obviously.

Whenever we need to do a full upgrade, when we need to plan the downtime for the production system, we try to make it on the weekend. I also already recommended to Worksoft that it would be nice to have something like an offline update where the system can be upgraded or smaller changes and fixes can be included without having full downtime. For an upgrade we usually need two to three hours. Afterward, we do a bit of testing, so upgrading takes about half a day.

It was just one or two people from Worksoft and three guys on our side.

We get feedback from all areas that the return on investment is there. Not just regarding time savings, but also cost-reduction. The return on investment in one case was reached at something between five and six runs, which is pretty fast, especially in an Agile environment.

What is also very important for us here is the avoidance of human error during the execution of tests. Usually, if someone is sitting in front of a laptop and doing testing eight hours a day, he or she will make some mistakes. This does not happen with a tool. Another important factor for us is the availability for testing. Usually, it's pretty hard to plan a test phase to bring all the testers to the table and get the time blocked off for the test phases. For the tools, we just need the systems up and running and then it's a matter of minutes to set up the test plans and to run the tests.

At the moment we are rolling out Execution Manager.

At the time we decided to bring in Worksoft Certify, we looked into two other tools. The key difference was that Worksoft was script-free. That was not the case for other tools. And the full integration to Solution Manager was one of the key differentiators between the tools.

If you have done a market evaluation and have decided to go for Worksoft, my advice is to go for it. I would definitely recommend Worksoft Certify as a test automation tool.

The feedback that I get from our stakeholders is that the tool is pretty simple to use. What we usually do is a two-week training, not full time, where the total is about three to four business days, 20 to 25 work hours. From there, most of our colleagues can start working with the tool. Of course, they have questions later on, some difficulties when it gets into special activities. But overall, the tool is easy to use. It's generally found to be intuitive.

In terms of cutting test maintenance time with respect to the scripts, that has not happened. If you need to adapt your scripts, automated test scripts are much more complex and more effort-intensive than manual test cases. But this is the nature of the beast. It will happen with every tool. If a screen changes, if a system changes, then you have to adapt your script for manual testing. For a manual script, you just adapt a Word document or an Excel sheet or the like. But if the process flow changes, you have new windows, new options, then you have to adjust your script for each and every provider that you're selecting. The maintenance of scripts is something that I always discuss with my end users and should never be underestimated.

We are not using the Capture 2.0 feature at the moment. We are planning to use it in the future. But at the moment, due to the heavy workload on our plate, we haven't had the chance to look into and to roll it out. We are familiar with the concept of Capture and it's a very nice feature because it makes the collaboration between business and IT much easier, and business can be involved in test-automation topics and activities as well.

We have three roles in our environment. We have the key players, who are the project managers, the persons responsible for test automation overall in the respective teams.
Then we have the test automation engineers who are responsible for creating test scripts and to maintain them; sometimes they run them as well. And finally, we have the executors, the ones who are running the scripts, checking the details and, if something is not working fine, going back to the test-automation engineers and asking for support and help.

I rate Worksoft Certify at nine out of ten. I'm happy with the tool, I'm happy with our colleagues at Worksoft. We have a very good relationship, we can bring up everything. There isn't much I can complain about. I'm happy at the moment with Worksoft.