Perfecto Reviews

Our use cases are execution on the cloud for web and mobile. With mobile, it could be different platforms, like iOS, and Android. We have functional execution and API test execution. That is primarily what I use Perfecto for. We have a software product that tests the product on different platforms of web and mobile. In an automated fashion, not just manually testing it but also with automating a lot of these scenarios where we execute them, Perfecto runs the execution for us against these different platforms. 

We have 100-plus members registered with Perfecto who use the product at some point. There are multiple deployments across our departments. For example, at the Bank of Montreal, our everyday banking is one application that folks use to do banking. Then, we have other products like stocks, trading, and other products. There are multiple teams who use Perfecto, even though they fall under different suborganizations within the organization. It is open right now for anybody in the company to use it. At this point, there are 100-plus people using it, at least 30-plus teams who may have used it until now, and at least 18 to 20 teams actively using Perfecto within our organization.

Perfecto has definitely helped our organization, mainly from a scaling point of view, with its reporting, and choice of frameworks.

1. It has allowed us to test across different configurations, helping us with our test configurations. 
2. It has helped us with our reporting. Perfecto's reporting allows us to have a clear view regarding what tests have been executed and the health of the application. 
3. It has allowed us to plug in our framework, i.e., the way that we designed it, into Perfecto with modifications. That fluidity and freedom are there for us. We are not tied or reliant on any one particular vendor. We are free to pick and choose with whom we want to work with. 

Perfecto has affected our software quality in a good way. It has allowed us to execute on-demand and on-choice. We also track the number of issues that we find in the product. Every single day, we tag the issues that we found. For example, if something was found by automation, that means it was found by a Perfecto execution. Over time, we realized the real value in tracking those numbers. We can see now that we have clearly been finding issues earlier. It has allowed us to catch our defects earlier, thus improving the quality of our applications.

We use Perfecto for our mobile testing; both real devices and virtual devices. We also use it in web online test cases, for testing the UI.

Overall, Perfecto has significantly helped us because we are not dependent on maintaining the infrastructure for various devices. Everything is maintained by Perfecto, which means that we can focus on testing.

We leverage Perfecto's reporting and analytics using the CI dashboard, reporting dashboard, and the live stream view. This functionality is quite important because it gives us the history of our past test cases and failures. From there, we can determine at which build a particular feature starts to fail.

Perfecto gives us the ability to test using virtual devices, and the testing can be done in parallel. This means that we are able to test against approximately 300 different devices.

In terms of scaling our testing operations, the option to use virtual devices is very important. If we didn't have the virtual device option then we wouldn't be able to scale as much, and our test execution time would increase.

When it comes to test creation, Perfecto supports open-source technologies such as selenium and APM. This is very good because initially, we were using most of the open-source technologies that were available. This meant that integrating with Perfecto was very smooth, as they supported what we were using.

The quality of our software has improved since we implemented this solution. Firstly, it helps that Perfecto maintains the hardware, allowing us to focus primarily on our software. Secondly, we have been able to find more defects in our applications.

We started using Perfecto to help us save costs. We gave out devices to a lot of content partners. At one point, we had 25 content partners, mostly in the US, and we gave them devices every year. We wanted a solution that would help us save on shipping devices to the US to our content partners.

In about six years, I've given 25 devices a year to at least 20 vendors. There are thousands of active devices and the roaming costs are mostly from mobile data, voice data, and text messages. Initially, we got into it from a cost-saving perspective, but then moved into doing our own internal testing and device qualification.

We have a hybrid deployment model. It's a public cloud, but we have credentials to log into it, which provides security. The devices are at our head office in our labs, but the web service is at Perfecto's data center. There's a VPN connectivity tunnel between our head office and their data center, and then it goes out to the web. Most people have all their devices at their data centers, but because of the sensitivity of our usage, we don't do that. Most of the devices that we have on the cloud are pre-commercial devices. They're not released to the market yet. We have a hybrid solution because we are working with sensitive NDA devices.

Everything is managed by Perfecto when it comes to the cloud itself.

Our team is using this solution and all of our preferred vendors signed NDAs. There are people in India, the US, and Europe who use this solution. Currently, in my system, I have 250 users, which includes people in marketing and customer service who log into the device before it launches and use it for training.

We also use the system for tutorials. We can take screenshots of new apps or services on the device. If someone wants to know how to turn on or turn off data roaming, we can click on settings and it will link each page as we click.

We have seen the biggest improvement in cost savings. Initially, we save $2.6 million a year just on roaming charges. Our testing is more efficient because we can automate testing. Before, we did everything manually. Three of our partners love it because they don't have to travel to Canada to test on our devices. They can stay where they are and get access to our devices and our footprint, so it has really improved our relationship with our vendors as well.

We also have devices in the US and devices in Europe so our team doesn't have to travel to do roaming testing.

Perfecto helps us save on the cost and effort of having to maintain our own virtual test environment. Saving costs was the key driver in our business case. We also saved time by automating the test cases, which saves us $2.6 million initially a year, including roaming charges and shipping devices to different vendors.

We started off integrating Robot Framework, which is an automation tool, with Perfecto and mostly we've been using it for web UI automation.

Recently I've joined in with another team member to use it with a Quantum solution provided through Perfecto, instead of Robot Framework. It's like a Java-based implementation. It's already integrated with Perfecto and our use cases for that are web and API.

And, while I'm not involved with mobile, we have projects that are using it for mobile as well.

It's given us some traceability, from a QA perspective, on when certain functionalities are available in online banking and when they are down. 

We have also used it for production automation to replace the manual labor that was done overnight on deployments to production, where someone was having to test them. We offload our scripts to Perfecto. 

And we use it not only for viewing the current execution, but as a historical tool to see our past executions and see when things were working and when they were not and where in the code did something fail. Perfecto is nice because, for Java code, you can see exactly where something failed, at which steps, and it shows you exactly which file actually it failed in. That's really helpful for us in narrowing down where the failure occurred. We know whether to investigate the automation scripts or whether it was a data issue or a defect. Perfecto gives us a lot of visibility into those things.

We are catching issues a lot earlier when it comes to the environment, because if we didn't have the automated jobs running with Perfecto, we would probably let a lot of these environment issues pile up until a lot later. But we're running them daily and seeing where things fail and Perfecto is giving us that video to see exactly where it failed and in which line of code. It's giving us a bit of a bump "left" to see where things fail first. It's definitely earlier than it used to be.

It has also definitely helped to speed up our testing efforts. When the scripts are stable, Perfecto is awesome. If there's not much change, we can run thousands and thousands of test cases every day. Someone from manual testing could execute, maybe, 30 test cases a day. Perfecto can execute, say, 3,000. It's really a huge difference. Part of what I like about Perfecto is that it allows concurrent testing. You can hit multiple browsers or devices at the same time, which makes it a lot more efficient for us to execute a regression.

And in terms of software quality, in one of our online banking projects that we recently delivered to production, we found 300 defects just with automation alone, and we found regression defects in addition. Not all of them could have been caught with manual testing. We improved our application by catching defects earlier, before they went to production, thanks to Perfecto.

We have all these devices and we test our mobile applications on them. We also use it for web testing, as a backup, when the other web browser testing tools that we use are down. We have combinations of mobile devices on which we do manual testing and we use it for automation testing as well.

Our organization mostly focuses on the front-end. We're a "digital factory" for all the UX and UI development testing. We design and develop front-end solutions for our bank. The solution is used by testers to test any changes related to mobile. 

Having Perfecto to test all the various changes to our applications on virtual mobile devices allows us to complete our testing and regression testing within the scheduled time. It enables us to do pretty much everything that we could do in the actual devices. It's really helpful in both automation and manual testing.

The Perfecto device lab has helped us scale our testing operations. It has become embedded with the success of our business and our projects. It would cripple our productivity if we didn't have it. We're very dependent on it. Our testing has been done utilizing Perfecto a lot lately.

Because we don't have a lot of physical devices within our team—we probably have one or two—and because our team is composed of about 50 resources who do testing, we cannot send them the devices, now that we're all working virtually. Perfecto is a perfect tool for allowing everyone to work remotely.

We have 42 devices in our Perfecto cloud base and all of them are being utilized by the testers. Our testers can access one of those devices at any time. As soon as that device is available, they can use it. They don't have to go to the office to use it. They can sit in their houses and perform their activities.

It has also helped to speed up our testing efforts because testers don't have to go to an office. However, the actual testing requires the same steps that you would run, and takes the same amount of time, if you were using the actual device. The savings come from having all your documents or screenshots saved by clicking one button. That's something that helps a lot. Also, the flexibility of changing from one device to another, just by clicking a button, is very efficient.

In addition, Perfecto helps affect our product's quality because it mimics real device behavior; everything is similar. Productivity-wise, we're able to test all the features and we're able to see what we need to see. For example, we can see the behavior of our application on every device due to the screen resolution of the device. If it's an iPad, the size of the screen is different than if you're using iPhone 12 or 13 Mini. When you go to Android devices, it's also different. Having all of these devices inside our cloud within the Perfecto tools, allows us to perform testing and deliver a better quality product.

With Perfecto, we are trying to automate our mobile test cases. These are test cases that run on mobile devices including Android and iOS phones, as well as Android tablets and iPads.

We are a financial institution and we rely on dedicated devices within a private cloud. Due to the sensitivity of our data, it is mandated that we use a private cloud.

I am the director of automation and analytics within our digital portfolio. We use Perfecto for the visualization and reporting of test results, including the relevant numbers.

We leverage Perfecto for reporting and analytics, and in that regard, it provides good reporting features. In terms of analytics, Perfecto still doesn't have predictive modeling or similar analytics features in the tool. In terms of the features that it does have, we use them to the fullest.

We make use of the Perfecto Reports dashboard. Within the framework, we use the Allure reports for automation execution.

We are using the Perfecto cloud for both real and virtual devices. For example, we have some cradles for real devices that we have licenses for, we have virtual devices by means of emulators and simulators, and we have browsers. These are the three types of licenses that we have.

The physical devices are sitting in the Perfecto device lab. The emulators and simulators work to simulate Android and iOS devices, and the browsers are separate again. As the main testing platform for our operations, the device lab is very important to us. We used to do everything in-house, which required on-premises infrastructure to do so. We got rid of the VMs and our physical machines, including our Macs. This is, in fact, one of the reasons that we chose Perfecto.

The fact that the device lab provides same-day access to new devices is a good thing for us. We have an SLA for procurement and if we need a new device, our SLA stipulates that Perfecto has a certain number of days to procure it. We are happy because when a new device becomes available, we get instant access. 

Another example of how Perfecto has helped to improve our organization happened during COVID. Sometimes during the pandemic, when we were working away from the office but still had the on-premises infrastructure, somebody would have to travel to the office only to restart a mobile device or a Mac machine. We no longer have to do that and that's a big relief for us.

When it comes to procurement, when Apple or Samsung launches a new device, all we have to do is pass the request to Perfecto. This is an example of where we save on infrastructure but it's not the only advantage. We are able to execute all of our tests in parallel on different cloud devices. Previously, we were challenged in terms of scalability with local infrastructure. Now, scalability is not a challenge and we can add as many virtual devices, running in parallel, as we need. Previously, we were limited to running a few devices in-house and using a few different browsers.

Perfecto has helped to speed up our testing effort. This is true both in terms of time to market and in terms of execution time. 

Our main use case for Perfecto is for our testers to test their mobile or PC applications.

One of the benefits of Perfecto is that it helps us keep our devices up to date with the right OSs, because we test across different mobile OSs. The solution has been instrumental in helping us to keep our devices at the right levels for our testing.

The solution also helps us to eliminate defects. Any application defect is a bad thing. I'm the "application police." If something is wrong with it, I'm going to give you a ticket and ask you to get it fixed. As a bank, we cannot have defects in the applications and Perfecto is instrumental in that process. That's how it helps us, organization-wide. It really stinks if one of our end-users opens up a mobile app, expecting they can do something but they can't because of a problem with the application. Testing with Perfecto, we find those defects and we get the developers to get them fixed before the end-user sees them.

When we retest and they pass, we're achieving the quality results that we're looking for.

And it has saved us tremendously on the effort and the costs of maintaining our own virtual test environment. We don't have to hire so many personnel to take care of those devices. Perfecto has people employed in its lab to physically work on devices when needed. We would have had to do the same thing if we didn't use the solution. As a high-profile bank, hiring an employee includes doing extensive background checks, and those kinds of things cost the company. Using Perfecto saves time, resources, and it saves on physical intervention with the devices. Overall, it results in huge cost savings for our company. 

We are using Perfecto to do remote testing on physical devices. 

The device is quite significant for us because we are part of a telecom company and quite a lot of testing is sensitive. We need to have private cloud hosting as well as public cloud hosting. Perfecto gives us both options, and we are using both private as well as public cloud. When a new feature comes out, we cannot just put it on the public cloud for everyone to test. That is why we use a private cloud. We can access it over the internet anywhere in the world, which is very important for us.

We use it to work across multiple teams and departments.

Perfecto plays a very crucial part in our day-to-day test executions. We do a lot of mobile testing, so Perfecto is very useful for us for either manual testing or automation.

It is very important to us that the lab provides same-day access to new devices. We have a very fast-moving, agile team. Therefore, we need access to the lab all the time.

We have a small development team who uses their emulators for development purposes. For testing, we still rely a lot on the actual devices. However, at times, when we launch something, we do use the virtual devices as well. Virtual devices are important when testing, although they don't give the same level of experience as real devices. I can understand that it isn't possible to purchase every single device all the time, especially when you are testing later versions of Android and don't have a virtual device or any version of iOS. We have a couple of native application teams who deploy new applications on those virtual devices all the time, depending on the requirements.

Perfecto helps a lot with productivity overall. We have a distributed team across different geographics. Because people are spread out across different countries, not everyone works at the same time, but whenever they work, they want access to the devices. The only feasible solution is to look at the cloud. Perfecto has come in really handy since it is always up and active. People can access it in their own time and at their own comfort, i.e., anyplace and anywhere. So, it has definitely helped with our fast, agile deliveries over the course of time.

Any testing is good testing. The more you test, the better software quality gets. It has definitely helped.