Sauce Labs Reviews

We use it for automated testing. For our company, we do web testing automation. We run our test suites containing Java-based scripts and automate the entire mobile or web application. We've 500 to 600 scripts, and we execute them on Sauce Labs. Sauce Labs provides different browser versions and mobile devices where we can execute our tests and get the results. That is how we use this solution.

Running 500 to 600 scripts in our local infrastructure takes 8 to 10 hours for the cycle to complete and get results, wherein when we use Sauce Labs and get parallel instances, we can run our test in parallel or concurrently. The execution time reduces from 8 hours to 2 or 3 hours based on the parallel instances procured from Sauce Labs. That's really good. The other thing is that it's very stable. We might face some network issues when we run tests in our local infrastructure, but I have never seen any such issues with Sauce Labs. Their infrastructure is good.

Sauce Labs is optimized for automation and integration with the major CI/CD platforms and developer tools. We've integrated it with Jenkins. There is a Jenkins plugin for Sauce Labs where you can just configure your job in such a way that it can automatically trigger on Sauce Labs. No manual intervention is required. They also support integration with Jira for defect tracking. They have a solution for raising a defect with one click, which is pretty cool. If you feel something has to be reported on Jira for developers to fix, you can just click on that, and it is reported directly on their Jira project. We've also integrated Sauce Labs results with Slack. Once the execution is completed, it notifies us on Slack that the execution is done, and we can also get our results over there quickly.

The testing environments we've used include Windows with Chrome, Firefox, and Edge. We've used Mac with Safari. On the mobile part, we've used Android and iOS. We've also used iPad on iOS. It's important to have multiple types of testing environments available in a single platform. We don't have to look at different solutions. Sauce Labs is a complete package. There are clients who request to execute their applications in N number of devices covering all the operating systems and mobile devices, and having such a solution to look at everything in a single package is very helpful.

Sauce Labs affects the execution cycle because the time taken for us to run the tests in a local setup is a lot more. Sauce Labs helps us with that part. Also, when the developers post any new code, we don't have to be there to run it manually. With the CI/CD and Sauce Labs integration, it automatically happens, and we can just go and get our results, which is very helpful.

The ability to test on additional testing environments has enabled us to make our product available to a larger user base. We also do functional testing for applications. In our inventory, we might not have the latest devices. For example, the device that we have with us might be of a certain version, and it might not extend beyond that, whereas Sauce Lab immediately launches the new version devices in the market, so we can quickly go, test, and report any defects in those versions. The coverage of testing has increased because of the availability of new versions, which has had an impact on the business. We can test and push the code to production after testing on new versions. Later on, if the users with the latest versions of devices report an issue, it will make sense to us.

Sauce Labs can run with every code commit and provide developers with immediate feedback. We have to arrange our architecture in such a way that it does that. If the architecture is right and you have the code in the right place, it definitely provides what you want. Previously, it used to take a lot of our time. We are an India-based team. If the deployment happened at midnight or 12:00 AM, when we logged in by 10:00 AM, we could see that there was a deployment. We would then do the testing, run the automation tests, and give the results back to the developer by 2:00 PM. By integrating CI/CD with Sauce Labs, the moment the development team commits the code for a different set of devices and browsers, it immediately triggers the Sauce Labs pipeline, and we can get the results at the same time when testers and developers log in.

The immediate feedback with every code commit affects the development processes. It depends on the file or script size that we have. For a small number of scripts, it takes one or two hours.

The immediate feedback helps to deliver software faster. When we do manual regression, it takes 40 hours of time. With Sauce Labs, the duration reduces to 20 hours, so the cycle of delivering the code is much faster.

We run our tests in parallel on Sauce Labs. When you have 50 scripts and you run them in series, if each script takes 10 minutes, then your execution time would be 500 minutes with one machine or one instance of a browser. If you have Sauce Labs with five parallel instances, you can run five tests at a time, which will save a lot of time.

When I started with my current company we had a fairly lean tech department and Sauce Labs was originally implemented for a piece of software we had just written that helped implement web testing.

As things got bigger, and we got sucked into and were merged with another company, the number of teams and how they use Sauce Labs fundamentally changed. It went from everything running through just this one piece of software, and it managed all connections, to the point where we let everybody run tests now against the tunnel, if they're inside the network. Every use case for every team inside the company is going to be somewhat different. The majority of our approximately 85 teams that run things through Sauce Labs are web-based, and use it for tests that are run almost 24/7. We have engineers on those teams who will push changes to some type of front-end. There's a grouping of tests, depending on the team, and those tests are set via Jenkins and CI/CD to execute constantly.

The cool thing about Sauce Labs is their love of metrics. I don't like keeping track of metrics. My management is constantly screaming about metrics, but when you finally give them metrics they don't check them. We have built out individual solutions to track the metrics for various vendors' solutions, including who's using what, what time they are using it, and how long they are using it. We might spend two months building something like that and giving it to management, and it then gets ignored. Sauce Labs Insights provides an easy platform where I can see that there are 85 teams running tests; 23 teams running tests right now; most of their tests are passing; and they're across these browsers.

The ease of access to that data is incredibly important because it means I don't have to build more metrics solutions that just don't get used. The fact Sauce Labs offers that kind of feature helps me in my job when it comes to responding to upper management and the requests that they sometimes make.

Also, the solution definitely helps to save on manpower. I'm a part of a team that has been managing a few Selenium bridges, smaller scale, on-premises "Sauce Labs," to help with smoke testing, and managing that is a living, breathing nightmare. Overall, before I joined the company, everything was done by manual testing. One team would have one web app, a development team, and anywhere from 50 to 100 manual testers who just sat and clicked on a website all day long. When automation came in, my wonderful job was essentially getting a whole bunch of people fired or having them moved over to automation. Based on my experience with managing a few different Selenium bridges, the amount of man-hours I've spent just debugging that, I could easily have a team of five to 10 people and still probably be running a bit behind in terms of managing that grid. The fact that I don't have to do that at all saves us nine people right away. It's a solution that has really helped, but I balk at giving a more exact number because it's hard to calculate.

The cost savings with Sauce Labs, on an annual basis are anywhere from $200,000 to $400,000, just for base level salaries for a small group of interns to provide the manpower it would take for us to manage something like Sauce Labs at a company level.

There are also savings on testing time. A lot of that comes from the ability to get teams on board fairly quickly. With the Sauce Labs documentation, everything is made so that it's very easy for a team to just jump in and start running tests. That saves us a ton of time.

We have different content spread across different regions and Sauce Labs also helps in enabling the testing that is required by that situation. The things that Sauce Labs helps with are things I don't want to do. I don't want to manage a giant stable of virtual machines. I also don't want to—and God bless them—but they've been managing Internet Explorer browsers. They're nightmares. The problems I have just trying to get an IE container or browser up and running, so a test can be automated against it, is insane. The fact that Sauce Labs has it down and working fairly well saves us a lot of work and headaches.

We primarily use Sauce Labs to test browser compatibility. It's mostly functional rather than performance testing. We use a combination of tools, but Sauce Labs is mainly for compatibility testing. Selenium is our backend, and it also has compatibility testing, but we're not making use of that feature. Selenium is for capturing. 

Our custom framework for testers combines Selenium and third-party vendors to do some of those performance metrics. At the same time, we use Sauce Labs to test cross-browser compatibility for the top five browsers that the government requires us to support. 

We automate tests of our on-premise solution with Sauce Labs via tunnels. Sauce Labs allows individual testers to log in and test whatever they need, but we don't do it that way. Instead, we use the automation features through tunnels, and our CI/CD process will run tests for us through Sauce Labs. It returns metrics on compatibility and usage for us to review.

Our two major platforms are Windows and Mac. We don't run tests on Linux. Even though we build everything on Linux, we don't support that for our end-users. We test our applications on the two main operating systems and variations of Safari. OS X testing is the main reason we started using Sauce Labs because we needed to test our applications on Safari, which isn't available on Windows. Initially, we purchased some Macs to do compatibility testing, but that didn't prove helpful at all. 

We also need to test on all Chrome variations because there are multiple versions we need to support. When we launched, Microsoft had just released Edge, so very few of our users had it, but just about everyone has migrated from IE to Edge by now. Testing on variations of Firefox, Chrome, IE, Edge, and Safari is our essential requirement. 

It's easy to set all that testing up on Sauce Labs. We could use a virtual machine to run applications on all the browser variations, but you need to get people in there to connect to it, and a homebrew solution is way too complex. With Sauce Labs, it's all already there. We just spin it up, specify the version we need, and we're done.

Sauce Labs doesn't give us immediate feedback on every code commit. That's not how we have it set up. We've got a multistage process, so it goes through a code review for quality when we do the commit. We have unit tests that happen along the way, but when we do a full-blown merge and are ready for a release, that's when we actually launch our tests, and the tests run overnight. There are thousands of tests, which is why we don't do it on every code commit, but we do it every night. When a nightly job runs, we run a full regression test on that to get the results the following day.

From an automation standpoint, Sauce Labs enables us to have a fully functional CI/CD process while saving time and cutting down on the training required for individual testers. We can easily automate tests without training a whole bunch of people on Sauce Labs. There's a whole slew of quality and security testing tools that a tester typically needs to know. Automating and integrating with Sauce Labs reduced the number of things a tester needs to be trained on, and they don't need to use Sauce Labs daily. They just see the results.

It's one less aspect the testers need to worry about. They come in from time to time to review the results, but it's automated, so they don't need to connect to Sauce Labs, run the test, and get the results back. They don't even have to know that it's Sauce Labs behind the scenes. We take care of that for them, so it saves us time.

Automation allowed us to reduce the size of our team. When we initially got Sauce Labs, we had a full core development team with a lot of testers. There were 50 to 70 testers working 100 hours a week. That's around two hours each day per tester on the low end, so 50 multiplied by two is 100 hours per week. Multiply that by six to eight months of testing.

We create banking software and we use this product for testing. We have different business units and we set up an enterprise license, so everybody feeds into it from each of our business units.

We have about 750 websites and approximately 50 mobile apps, and we will test the different types of browsers against our automation. A good section of the work we do is running automation against different combinations, and that will expand into mobile devices once we kick off the new year.

For the most part, it's heavy automation, but there is also testing that is manual, where they can log in and pick their devices or browsers.

Our environment includes VMs on the cloud, as well as public and private devices. we have the CrossBrowser and we have the private and public cloud.

Sauce Labs allows us to expand our coverage without having to run them separately against different browsers on your local machine. You can just spin up an instance and run against whatever browser and operating systems combinations you need.

Especially on the mobile side, things are improving. We had an on-premises lab, where we were managing the physical devices on a device cart. That's going away, as we are now able to use the public cloud offering, which has 2,000 plus devices.

For locking down devices that we prefer to use without registering with other clients, we use the private Cloud. We split up the distribution 50/50 between private and public.

Sauce Labs is what enables our different testing combinations. We run our different suites that include smoke tests, regression tests, and functionality testing. All of these types of testing need to be verified against different browsers and mobiles, and this is our solution to that problem.

This product helps us to increase test coverage by testing both functional and visual aspects of the UI. This is very important to us because we have different types of customers. Some are end consumers, whereas others are banking customers. We won't release software without validating all of our use cases.

Sauce Labs helps us to deliver new features and products to the market. It's part of the use cases that we have to validate, so we need tools to complement our test cases. We're trying to eliminate manual testing or how we kick off scripts. We want to be able to automate testing of our suite of use cases against our devices and operating systems.

My company is quite large. My team supports the various different IT teams around the business who then work with their business partners. For example, if you think about our research and development division, our clinical trials division, and our manufacturing division, each one of those business groups has their own IT teams who build tools that the business team needs to do its jobs. My team provides the tools that the IT teams would need to build the tools for their teams. So, we are two or three layers removed from the broad side of the business.

We are the team who provides this solution to other groups. However, in terms of our usage, because the company has done things by paper signatures, official paper, and hand documentation for a long time, there has not been a whole lot of progress yet on the automated testing sector. Therefore, our usage is very small compared to other businesses of our size. 

We have 25 concurrent VM licenses for Sauce Labs today. That provision amount has not changed in my time here because it has never exceeded that capacity. So, it is a slow, upward trend, but it is very slow right now.

In January of 2021, there were 515 automated tests that ran through Sauce Labs in the company. By October, we were just shy of 3,000 tests. That is a 600% increase.

In general, I know that there are different pockets of the business that build apps for different cases. For example, we have an iOS dev team in one part of the business. We have a Windows team in a different part of the business, and they are able to use the same platform without us having to try to support multiple different things. In that sense, the ability to test on additional browser, OS, and mobile device combinations has been a great help. 

All our automated tests run in parallel. This helps us to demonstrate value to other parts of the business who have not yet adopted the solution. For example, when there are groups who use Sauce Labs and we try to share learnings, that team presents, and says, "I have saved this many specific hours. Or, I can run 225 tests (or however many tests) in parallel. It saves us this much time and this many dollars to have our application go out that much faster."

We use Sauce Labs in our CI/CD pipeline for our mobile app.

I work on the team that develops the mobile application for Discover, and we run about a couple of thousand tests per day on Sauce Labs devices. Whenever our developers are contributing code to our application, we'll run a bunch of tests on Sauce Labs devices, and every night we run a full suite of 1,000-plus tests on those Sauce Labs devices.

It saves us a lot of time because Sauce Labs provides us with plenty of devices, enabling us to run 1,000-plus tests in parallel. As a result, we can get it done pretty quickly. Typically, testing would take six days, and we got it down to one day. That's how much we're saving.

Also, because Sauce Labs provides tons of different devices, instead of relying on a small number of devices on-premises, we can rely on Sauce Labs' cloud and run tests on all sorts of devices. That can include things like an old Galaxy S7 or a newer Galaxy S20, as well as different operating system versions. Sauce Labs has all of that and we can test all sorts of different combinations of devices and operating systems.

We have been using Sauce Labs to test various browsers and OS combinations as well as test our applications. Our existing automation scripts are written in various technologies, which could be Java, JavaScript, Selenium, Cypress, etc. Jenkins is the tool that we use to typically run our jobs. Through Jenkins, they get scheduled and run in Sauce Labs. This is where we choose to run them, through various browsers and OS combinations.

We use Sauce Labs core and whatever services that we choose to go with, like browsers and mobiles. From there, we can choose the browser and OS versions, etc.

Without Sauce Labs, we wouldn't be able to test as quickly or simultaneously on various browsers. That is the capability Sauce Labs has brought to us.

The automated functional testing, visual regression testing are pretty important to us, as those are the use cases for which we use Sauce Labs. Specifically because it is hosted in the cloud, we don't have to use as much capacity from our own servers. We don't have to maintain and install various versions of browsers nor do we have to maintain them. It is a good solution and works well for us.

Most of our teams who have solutions to test on mobiles are using the solution's mobile emulators and simulators. Having that availability is very important for a complete testing experience.

We use the solution's emulators, simulators, and different browser version combinations for all our testing. For every release, we validate across various browsers. If it is a mobile application, then we need to validate across various Android and iOS devices as well as the previous few versions of each of the operating systems.

The customer base that we typically serve is vast, diverse, and varied. This makes it very common to have our applications used across various applications, systems, and screen sizes. To test across all that, we needed a system in place. That is why we are using Sauce Labs. Without it, we would either develop something in-house or rely on possibly testing in only one particular system. For example, with Android, our entire customer base who is not Android wouldn't be able to validate and reproduce the user experience.

When we are doing releases, the releases happen across all teams and various services. This means all of them need to test for a particular release deployment in whatever environment simultaneously. Based on the number of concurrent VMs that Sauce Labs provides, we can add or reduce VMs as needed. The amount of concurrency that we have purchased is specifically to support our various teams to do deployments simultaneously. Doing these simultaneous tests has reduced our test execution time.

We have two kinds of applications using Sauce Labs in our company. One is the website, and we're using it to test across browsers, such as Chrome, Safari, Firefox, Edge, and mobile, to see that all the components we have in a web page work. 

The second product we use the solution for is our driver application. We make and deliver pizza. Every time an order is received, we dispatch a driver to bring it to the customer. We have an application for driver dispatch on iOS and Android, and we run test cases for those as well.

I cannot imagine our company without Sauce Labs. We are using it a lot. It gives us a provider in the cloud for running test cases, scenarios, and validation, and that is really important nowadays. You could always run testing locally or have your own environment, but it's difficult to scale and difficult to maintain. You need to dedicate a lot of resources to do that.

We ran, for example, 500 test cases in the last week for one specific team, and those test cases were running at 2:00 AM or 3:00 AM, for some specific scenarios. We could not do that without Sauce Labs. We are integrated for continuous integration using Jenkins and we're triggering all these tests nightly for different teams on different schedules. It would definitely not be possible to run that manually or in a different way. The cloud is the right environment to run it in, and for that, Sauce Labs is the best option in the market.

Also, the Failure Analysis helps us prioritize remediation. If we have a very high number of test cases failing for a given feature, we can analyze them, see the root cause, see the code in our repository, and create a Jira ticket to address those failures. That feature is very helpful for us. It's pretty straightforward and doesn't consume too much time. I generally spend about 15 minutes analyzing all the failures across the teams to see which three test cases are failing most.

Another benefit is that Sauce Labs saves us on manpower. We have manual testers, but it's not possible for them to cover all the different scenarios and all the different features before release. That means we need to rely on the automated test cases, and to do that, we need to run those test cases on a cloud platform. Sauce Labs saves us a lot of time doing those validations. It probably saves us from having to hire hundreds of people. Every team within our organization has one or two manual testers, but for every team we have hundreds of automated test cases that we run before release. It's good enough, for our operations, to have one or two manual testers. We are seeing good results with that, but to run hundreds of different scenarios I don't know how many people we would have to hire.