What is our primary use case?
We use it for automated testing. For our company, we do web testing automation. We run our test suites containing Java-based scripts and automate the entire mobile or web application. We've 500 to 600 scripts, and we execute them on Sauce Labs. Sauce Labs provides different browser versions and mobile devices where we can execute our tests and get the results. That is how we use this solution.
How has it helped my organization?
Running 500 to 600 scripts in our local infrastructure takes 8 to 10 hours for the cycle to complete and get results, wherein when we use Sauce Labs and get parallel instances, we can run our test in parallel or concurrently. The execution time reduces from 8 hours to 2 or 3 hours based on the parallel instances procured from Sauce Labs. That's really good. The other thing is that it's very stable. We might face some network issues when we run tests in our local infrastructure, but I have never seen any such issues with Sauce Labs. Their infrastructure is good.
Sauce Labs is optimized for automation and integration with the major CI/CD platforms and developer tools. We've integrated it with Jenkins. There is a Jenkins plugin for Sauce Labs where you can just configure your job in such a way that it can automatically trigger on Sauce Labs. No manual intervention is required. They also support integration with Jira for defect tracking. They have a solution for raising a defect with one click, which is pretty cool. If you feel something has to be reported on Jira for developers to fix, you can just click on that, and it is reported directly on their Jira project. We've also integrated Sauce Labs results with Slack. Once the execution is completed, it notifies us on Slack that the execution is done, and we can also get our results over there quickly.
The testing environments we've used include Windows with Chrome, Firefox, and Edge. We've used Mac with Safari. On the mobile part, we've used Android and iOS. We've also used iPad on iOS. It's important to have multiple types of testing environments available in a single platform. We don't have to look at different solutions. Sauce Labs is a complete package. There are clients who request to execute their applications in N number of devices covering all the operating systems and mobile devices, and having such a solution to look at everything in a single package is very helpful.
Sauce Labs affects the execution cycle because the time taken for us to run the tests in a local setup is a lot more. Sauce Labs helps us with that part. Also, when the developers post any new code, we don't have to be there to run it manually. With the CI/CD and Sauce Labs integration, it automatically happens, and we can just go and get our results, which is very helpful.
The ability to test on additional testing environments has enabled us to make our product available to a larger user base. We also do functional testing for applications. In our inventory, we might not have the latest devices. For example, the device that we have with us might be of a certain version, and it might not extend beyond that, whereas Sauce Lab immediately launches the new version devices in the market, so we can quickly go, test, and report any defects in those versions. The coverage of testing has increased because of the availability of new versions, which has had an impact on the business. We can test and push the code to production after testing on new versions. Later on, if the users with the latest versions of devices report an issue, it will make sense to us.
Sauce Labs can run with every code commit and provide developers with immediate feedback. We have to arrange our architecture in such a way that it does that. If the architecture is right and you have the code in the right place, it definitely provides what you want. Previously, it used to take a lot of our time. We are an India-based team. If the deployment happened at midnight or 12:00 AM, when we logged in by 10:00 AM, we could see that there was a deployment. We would then do the testing, run the automation tests, and give the results back to the developer by 2:00 PM. By integrating CI/CD with Sauce Labs, the moment the development team commits the code for a different set of devices and browsers, it immediately triggers the Sauce Labs pipeline, and we can get the results at the same time when testers and developers log in.
The immediate feedback with every code commit affects the development processes. It depends on the file or script size that we have. For a small number of scripts, it takes one or two hours.
The immediate feedback helps to deliver software faster. When we do manual regression, it takes 40 hours of time. With Sauce Labs, the duration reduces to 20 hours, so the cycle of delivering the code is much faster.
We run our tests in parallel on Sauce Labs. When you have 50 scripts and you run them in series, if each script takes 10 minutes, then your execution time would be 500 minutes with one machine or one instance of a browser. If you have Sauce Labs with five parallel instances, you can run five tests at a time, which will save a lot of time.
What is most valuable?
The solution they provide is very robust. We can quickly connect to their environment with the hub URL. They have a URL that has the entire grid of desktops, web browsers, and mobile devices. They also provide real devices, so you just provide the URL and test your application. You can test not only on the web. If you have any application, such as an APK or an IPA file, you can upload that, and you can test it on different versions. They have more devices than the other solution providers. These are the features that I find to be good.
The knowledge center that they provide for different platforms such as Node.js or Java and different testing tools such as Selenium, Cypress, or Puppeteer is very helpful in getting started for a new user.
The support they provide is also valuable. Once we sign up with them, they quickly raise a ticket for any issue, and they address that within 24 to 48 hours. That's appreciated.
What needs improvement?
With the desktop browser, we can inspect any screen with the web developer option, but they should provide something for mobiles so that we can quickly inspect elements on the device. To write the Selenium scripts, we require web locators. We have to capture them from the local and execute the script on Sauce Labs. If Sauce Labs can provide a solution where we can inspect any of the mobile devices online, it will be very helpful for us.
I also faced an issue where during the execution time, the iCloud password used to expire within a certain period, which used to create pop-ups on the screen making our tests fail. That was one thing that we faced many times. If that can be improved, it will be very helpful.
For how long have I used the solution?
I've been working with this solution for almost three years.
What do I think about the stability of the solution?
It's stable. I used to run 100 scripts for a project on Sauce Labs, and it used to give me results in four hours without any major failures or a 96% to 98% pass rate. There were one or two things, such as the iCloud pop-up on the iPhones, that were causing issues.
What do I think about the scalability of the solution?
It scaled well for our needs. We utilized 10 parallel instances, but they provided us with 15. If you keep on scaling it on a larger scale, the application that you're trying to automate should also support that, but it's scalable.
We are a team of 60, and 20 of us are using this solution. All of us are from the QA team. We provide the service, and we utilize this tool to make sure that everything is running fine. We ensure that every product that we get from developers is properly working, and once we certify that it's working fine, we move it to production for the developers to do their job.
How are customer service and support?
Initially, we didn't reach out to their technical experts because we had the expertise on it. In one or two scenarios, we reached out to them, and they helped us with examples and quick solutions. They resolved the error well, and the solution they provided worked for us. They were very helpful.
They have L1, L2, and L3 support. The ticket goes to the L1 support team first, and if they have the solution, they provide us with that. If not, it's escalated to the next team. They're pretty decent at providing solutions. I'd rate them a seven out of ten.
How would you rate customer service and support?
How was the initial setup?
Compared to the other cloud providers, it was easy. If you are technically strong, it'll take you just one day or eight hours. If you are a beginner to the cloud service providers or automation itself, it'll not take more than three days for you to get started with Sauce Labs. They provide a good knowledge base and good documentation for you to get started, which makes it an easy job.
Integrating Sauce Labs with the framework is a separate task. It took us 40 hours to integrate Sauce Labs with the framework we had.
In terms of maintenance, so far, it's only on the framework side. We just change the version that we want to run. A particular version might get outdated, so we just go and change it in our code, and it'll appropriately take that particular version on Sauce Labs.
What was our ROI?
The ROI is in terms of time savings and reduction of manual effort. It takes a lot less time to provide results. It reduces manual effort and helps in getting the product out earlier in the market. Even the developers can use it for their own testing. If they have something to quickly check in the UI, they can. Initially, they'll move a particular feature to the testing team without testing, but now, because Sauce Labs is there, they can quickly go and see how their UI looks on different versions, and they can fix any issues. It reduces one cycle of work for us as well as for the entire product team. There is an ROI, and we see customers renewing their license for the second year to use Sauce Labs for their solutions. There is a 15% to 20% ROI.
What's my experience with pricing, setup cost, and licensing?
Cost-wise, it's decent. If you have to get the base version out of it, it's the best solution to go with. As compared to other cloud service providers, the pricing of Sauce Labs is decent.
There are monthly and annual packages, but I'm not much familiar with that part. They might be providing too many things in the package, and all of them might not be useful for people. They can modularize the offering instead of giving it as a whole package.
Which other solutions did I evaluate?
There were a couple of recommendations from different clients, which included LambdaTest and AWS Device Farm. We did PoC of those, and they were pretty much aligned with what Sauce Labs provides. The budgeting part was probably the main factor in decision-making, which is taken care of by our onsite manager.
Overall, Sauce Labs has a wide range of browsers and real devices. They also have a good knowledge center. It's a robust and scalable solution with ease of connecting. We have a Center of Excellence team in our company that did the research and found Sauce Labs to be the right solution to pick. It has been in the market for a long time, and we found it to be mature enough for our needs.
It was important for us to go with a cloud-based solution because the infrastructure that we have is limited. The range of devices, the versions, and the machines that we use wouldn't have been sufficient to run the scripts that we authored, so we moved to Sauce Labs.
What other advice do I have?
If you're looking for a cloud service provider that is scalable and stable enough to execute your tests, Sauce Labs is recommended. There is also Sauce Connect Proxy. If your application is running on a local host, you can just connect by using Sauce Connect Proxy, and you can make it global. That is a cool feature that they provide. It has a large range of devices where you can do both manual and automation testing.
Sauce Labs is a cloud-based solution, so in terms of latency, when the application is hosted on a server at a different location, we have noticed a latency. For example, if the application is running on the India servers and the execution is happening on the US-based location on Sauce Labs, the script that runs in two to three minutes in our local setup would take around six to seven minutes on Sauce Labs.
Overall, I'd rate Sauce Labs a six out of ten.
Which deployment model are you using for this solution?
Public Cloud
Disclosure: My company has a business relationship with this vendor other than being a customer: Partner