What is our primary use case?
At our company, Applause, we offer software testing as a service and we always get a lot of interesting, uncommon or challenging use cases from our clients. We sometimes get ones that require specific devices or browsers to work. For example, we have clients who want to mix testing on desktop browsers and apps or test on multiple apps to achieve some kind of scenario; perhaps you are at a restaurant, and you are ordering something on your personal phone/tablet, which shows up on the restaurant's tablet or desktop browser.
Our clients are not only looking for executing the test cases manually, but their target is to automate all of them and be able to integrate that into their CI/CD pipeline and get faster feedback about the stability of the changes that the development team produces on a daily basis.
Sauce Labs covers all of our automation needs and also allow us to do manual testing in case we are verifying bugs or testing something else.
How has it helped my organization?
My main focus is on automation testing and Sauce Labs is an integral part in our success as a company as they offer 1000+ different combinations of desktop browsers, real devices & OS versions. This allows us to pretty much cover all the automation test scenarios we need. We've also integrated them in our internal SDKs and are using their platform on a daily-basis to test the quality of our clients' apps and websites.
The number of the real devices they offer is also very important to our business, because most clients want to support multiple devices & OS versions. We have clients who want us to run automation tests on Android 7, 8, 9, 10, 11 and sometimes even on Beta versions. Sauce Labs offers all of that, enabling us to support all of these versions, which is really nice. The same goes for their desktop browsers, although most of our clients usually use either the latest version or the one before that. I haven't seen a client who wants to use and old browser version like Firefox 50, but it's a great feature that Sauce Labs offers, in case it is needed for any reason.
In terms of delivering software faster by getting feedback with every commit, it depends on the client and the system under test (SUT); some of our clients are testing manually and that would take them days, others have a CI/CD pipeline and run some smoke tests after each commit. Part of our job at Applause is to enhance the overall testing process of our clients & automate it as much as possible. Instead of taking days to do a full regression, it might take a couple of hours to get the same results.
Because we're working with Sauce Labs, and we're doing automation for our clients, they get a pretty fast cycle of feedback that allows them to make the right decisions at the right time.
Without Sauce Labs, it would be really hard to achieve the software quality standards or the automation coverage that our clients need.
What is most valuable?
Towards the beginning of this year, we had to constantly switch between Sauce Labs and their legacy solution for real device testing, TestObject. This was causing a lot of issues or confusion for our engineers & clients. Now, with the new unified platform, we get everything we need in one single place and it is so intuitive and straightforward. You can easily do manual testing, view running automation scripts or switch between data centers from one single place.
Overall, we're very happy with the combinations of browsers and devices they offer, especially because they always have multiple instances of the same device. If you want an iPhone 12 for your tests, you can get five of them for example and run your test scripts in parallel, which reduces the total execution time.
Sauce Labs also allows specifying the browser version as "latest", which means that they automatically pick the latest version they support and run the tests against it. Previously, we had to manually change the browser version in our scripts whenever a new one becomes available and it was wasting a lot of time (as it needed to be done for a lot of projects).
Their platform also offers some cool features if you're working with apps. You can just pass them a URL to an app build that is stored somewhere on your servers or on AWS, and they will install that on the phone/tablet and run tests against it on the spot. This has allowed for an easier integration with our SDK, as opposed to competitors who don't offer such feature.
What needs improvement?
The market is changing and we're seeing a lot of shift towards testing on OTT devices, like Apple TV, Fire TV Sticks, Chromecast and Roku. This is really an emerging market and has the potential to grow exponentially. We've been getting a lot of clients asking us to test on OTT devices, either manually or via automation. We have been able to automate testing on OTT devices, but it is done with local devices hosted on our premises, so we can't scale that fast or cover all of the huge demand we are seeing. We're looking for the ability to do this in the cloud, so if Sauce Labs offered such a feature, that would be really great. Working on OTT devices usually includes testing on desktop browsers or phones/tablets and having all of those supported in one platform would be ideal.
While Sauce Labs pretty much covers everything we need, another exception is HarmonyOS. I know it's a new thing and it's not supported by a lot of providers, but it is something that clients have started asking for. That is something we'd be really interested in seeing, for both manual and automated testing.
Also, latency, due to Sauce Labs being a cloud-based solution, has been a concern. We work in different continents and countries, but last time I checked, Sauce Labs was only offering two data centers, one in the EU and another in the US. If you're not in either of those two places, you would have latency and issues running your test scripts. However, I'm working with clients in different countries and we've seen some latency issues, depending on the country, although it's not huge. In comparison to other providers who have data centers in different places, there is some noticeable latency.
For how long have I used the solution?
I've been using Sauce Labs for over five years.
What do I think about the stability of the solution?
Sauce Labs is pretty stable in comparison to other providers. We still get some issues every now and then or random failures, especially when there is a new OS or browser version, but it's not a big deal and we can easily contact Sauce Labs' support to get things fixed.
What do I think about the scalability of the solution?
One of Sauce Labs' competitors does not force you to select the data center you want to run against. They just figure out which data center is closest to where you are executing your test scripts from. They offer a single, unified endpoint or URLs when it comes to automation and handle everything else internally. When we're writing automation scripts, we don't need to specify, "I want to run on the European data center," or "I want to run on the American data center." We can just say, "I want to run a test case," and depending on the location you come from, the platform is smart enough to direct it to the nearest data center to reduce latency.
This may not be directly related to scalability, but this kind of capability would make it easier for us to build our SDKs faster and focus on other features, which in return would allow us to scale faster as a company. The learning curve for newcomers would also be easier, because they wouldn't need to worry about figuring out which data center to run against.
While I love the fact that we can specify the data center ourselves, because it gives us more freedom, I would love to see more data centers in different places around the world, to reduce latency, and the selection of the data center implicitly done by the platform, so we don't need to worry about it.
How are customer service and support?
We open support tickets, and they're pretty responsive, they get back to us on time. They're really good about fixing things and making their platform more stable.
I've run into a couple of situations where tickets have been left for weeks without feedback, but that has not happened often. When it does happen, I can go to our customer success manager and say, "We need some attention to these tickets." She can take care of it and that makes the support pretty good.
How would you rate customer service and support?
What other advice do I have?
Pay attention to selecting the correct data center, otherwise you would run into some latency-related issues.
Although we are not using visual regression testing, that is something that would be really interesting, because a lot of our media clients want to test visual validations on their apps. If Sauce Labs is offering that out-of-the-box, it would be really interesting for us.
Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor. The reviewer's company has a business relationship with this vendor other than being a customer: Partner