Sauce Labs Reviews

My primary use case is for cross browsing testing of the largest eCommerce site in Latin America.

Sauce Labs gives us the availability to use any browser and OS combination whenever we want, with only a few clicks. This is very useful for our IT department. It is also used by our customer experience and marketing departments to reproduce some issues detected by our customers.

• Real devices (Test Object): It is very important to test everything on real devices, not only in simulators.
• Analytics: It is important to know what, when, and where tests are falling, then improve them and the application.

An image comparison would be a nice feature to include in the Sauce Labs product. 

As we began architecting a CI/CD pipeline in our SDLC, a need quickly arose for left-shifted, continuous testing of our web applications in real browsers. Specifically, a service that could integrate seamlessly with our Jenkins CI servers and executing tests continuously against our Dockerized applications and firewalled environments. Furthermore, our company does not have the resources to implement and maintain a proprietary Selenium Grid for test distribution. Sauce Labs addresses all of these concerns, and furthermore offers a platform for manually debugging applications in any combination of platforms, browsers, and devices needed, either from an automated test script or from a manual test session. The largest selling point for our needs is its relatively seamless integration with our Jenkins CI servers, including detailed playback and test reports per build job, as well as easy configuration of the proxy tunnel used to access our firewalled environments and desired browsers-under-test, right from the Jenkins UI via their Sauce OnDemand Jenkins Plugin.

Prior to moving to a continuous delivery model, we were on a classic two week regression testing cycle, mostly conducted via manual testing and manually-executed automated tests scripts in a large integrated testing environment. Upon shifting our SDLC to be more continuous, and rearchitecting our delivery pipeline, we were able to very quickly shift that long testing cycle to be integrated into the delivery pipeline itself by left-shifting these deep functional/regression tests to be executed during build time on our CI servers, and the implementation of these automated tests scripts being integrated into the development effort itself. Sauce Labs allowed all of this to happen by supplying the service needed to execute automated browser tests in from a CI server without having to implement or maintain our own proprietary grid of virtual machines.

Of all the services explored, Sauce Labs offers the single best solution for integrating deep automated browser testing in a CI/CD pipeline, and integrates best with Jenkins. The Jenkins UI plugin offers a way to easily configure browser settings for tests being executed continuously. The Sauce Connect Proxy is a necessary technology for companies who have firewalled testing environments, as well as localhost testing. Test reports (that also integrate seamlessly with Jenkins jobs) are also detailed and incredibly useful. Browser support is quick to be updated as new versions of browsers are released. The custom capabilities that can be provided to Sauce Labs VMs during automated testing sessions are a valuable option for experimental or niche testing.

The Jenkins Sauce OnDemand plugin could have more options available to tap into more of the custom capabilities Sauce Labs actually supports. Currently, in order to tap into these capabilities, a programmatic solution is required from the test code rather than being able to configure them via the Jenkins Plugin UI, e.g., desktop resolution, browser automation binary versions, remote Selenium version, and browser console logs. These are all things it would be nice to be able to customize straight from the Jenkins Plugin UI. 

In our experience, we have also had some issues with the Sauce Connect Proxy on our Jenkins servers failing to start, which makes the optimal CI/CD pipeline come to a halt. 

User account management needs an overhauls, allowing for user groups rather than just a hierarchy structure. Test job queuing is a first-come-first-serve, meaning there is no inbuilt way to queue batches of jobs from different Jenkins servers/accounts.

Occasionally there are issues with the stability of the Sauce Connect Proxy during Jenkins builds, which can cause builds to fail seemingly inexplicably, since determining the root cause is oftentimes difficult.

The only issue is cost. Each new unit of concurrency you add to your enterprise license represents a relatively linear increase in cost.

When you reach the Enterprise licensing tier, base level being with 10 concurrent test sessions, pricing is essentially per-unit-of-concurrency thereafter with a relatively linear increase and not much benefit for "bulk".

BrowserStack was also explored, but did not integrate with out intended CI/CD model as extensively or seamlessly as Sauce Labs was able to.

• Running tests in parallel
• Sauce Connect Proxy tunnel
• JIRA/Bamboo plugins

Enables faster execution of regression tests without having to support/maintain the software needed to validate regressions against different devices, OS, and browsers.

Speeds up the time it takes to run end-to-end user interface (UI) tests inside their virtual machines (VMs).

Less than a year.

None yet.

Not yet.

Not yet.

Excellent, so far.

Excellent, so far.

Yes. We built and managed all of our testing infrastructure in-house.

Pretty straightforward, except when you begin working within an enterprise organization that operates nearly all of its web applications behind a firewall. To get our automation running inside Sauce Labs's infrastructure, we first have to install and configure the Sauce Connect Proxy server. The complexity of setting up the tunnel comes mainly from the protocols our organization has for getting the approval to implement Sauce Labs behind our firewall. Explaining the details of the tunnel, why we need Sauce Labs for regression testing, and facilitating the connection between our Server's team with a Sauce Labs technical resource were all fairly complex.

In-house.

It is difficult to estimate at this time.

Start small, even if you are a large-to-enterprise level organization. The setup cost, if you're an enterprise-sized organization, is negligible. Establish a few quick and easy "wins" for one or two product teams. Hold a post-Sauce review with all the stakeholders within each team with the goal of reaching a consensus on whether the product is a good long-term fit. If the answer is yes, then calculate ROI, and if it makes sense, scale up and spend the money so every product that should use Sauce has access and the correct bandwidth.

SmartBear, Perfecto, and Ranorex.

The on-demand provisioning, the multiple operating system and browser choices (including multiple versions), as well as the really nice screenshots and videos, as well, are key.

Having full logs, with videos and screenshots, has really helped us work more closely within our own team, as well as, more importantly, being able to show developers (automatically!) what the precise problem is.

Since around August of 2011

Integration with Sauce has been a breeze for us here at Mozilla.

At various times, there are high-latency launches, or outright timeouts, but the Sauce Ops team has been pretty good about both 1) acknowledging the issue/investigation 2) working steadily to resolve it.

At times, the brief outages or slow service responses have temporarily hampered our fairly large (in terms of concurrency) test-automation runs, but that's not very often.

Sauce support has been very helpful -- always friendly, pretty prompt, and thorough, too -- even when it's not always a direct problem with their service or our usage of it.

For our needs, the customer service and technical support come through the same channel, and we've been very happy with the level and quality of responsive support we receive.

Being an open-source organization, we wrote our own custom plugin to handle integration of both our in-house Selenium Grid as well as Sauce Labs (https://github.com/mozilla/pytest-mozwebqa), and although I didn't personally do any of that coding work, can attest that the APIs Sauce provides have made it pretty easy for us to do that integration and setup work.

We are lucky to enjoy Sauce Labs's "Open Sauce" program, as an open-source, not-for-profit organization: https://saucelabs.com/opensauce

We did look (back in 2001) at other vendors, but found the business side and open-source/project relationships fit well with us.

As a disclaimer, I have done a number of Selenium Meetups with Sauce Labs, as well as a few blog posts which have covered our usage of the service, as well as, finally, authored a Featured Customer case study, in which I expound on nearly all the points I've made here:

https://saucelabs.com/customers

• They have a video recording feature which I find tremendously helpful. It helps me debug and find defects quicker. When I see something fails, it shows me the video of the failed part, the video playback feature is amazing.
• Easy to setup.
• Very responsive. Get back within the same day – overall great company.

My company was doing manual testing, I came in and built a framework using Sauce Labs; now it takes less than an hour for regression testing. Easy integration, increased the amount of codes that go out. I used to have to write the tests and perform manual tests; Sauce Labs, has changed all that.

Documentation is pretty extensive, but at an advanced level could be a bit better. But they are always helpful. Besides that everything is great, they help out right away, so nothing is a big deal.

About three and a half years.

No issues with deployment.

No issues at all with the product's stability.

Very good, highly scalable product.

They are amazing. Always get back to me, follow up, everything is amazing.

They are amazing. Always get back to me, follow up, everything is amazing.

Previously I used my own virtual environment – had to create and manage it on my own, now I don’t have to worry, infrastructure, setup, anything, Sauce Labs does it all for me.

Straightforward for me.

I implemented in-house.

In terms of value, we increased our release cycle to twice a week as opposed to once every two weeks – so its doubling productivity.

I did. I looked at BrowserStack. Did not have video-playback, which is tremendous for finding bugs and code.

I think its excellent and everyone should get it – I recommend it to everyone. No brainer. Nothing else like it out there right now.

I am part of the QA team. I am implementing this solution. Right now, we are also trying to implement this solution in order to gather results in the testing process.

We have several stores around the world, more specifically around Latin America. We are trying to automate many tasks for the mobile applications that we are building. We are also trying to automate many web tasks in order to upload items to the tool.

The device level of the platform is the most valuable aspect for our team.

So far, the stability has proven to be quite good.

The solution seems to have interesting automation capabilities that we still need to understand. 

The testing process is difficult. I need to prove the complete competency of the tool, and I am finding that challenging. 

Right now, we are in the initial phases when it comes to actually using the product. We are testing some tools in order to implement them in the future. We are in a very early stage.

While we are still in the early testing phase, from what I have seen so far, I can say the solution is quite stable. There are no bugs or glitches. It doesn't crash or freeze. The performance appears to be very good.

That said, we still need more time to really dig into its capabilities. 

We've been in touch with technical support as we've run into some issues and we want to better understand how the product will work in practice. We've been in touch to try to get a hold of someone that can walk us through a customer journey so that we have the knowledge we need to use the full potential of the product. We're still working on making this happen.

The solution has not been implemented fully yet. We are still trying to uncover if this is the right solution for our company. We seem to need assistance with support and with the setup process before it will actually go live. We're not far enough into the process to really discuss the deployment and if it is difficult, or complex, or very straightforward. Those details are still to be determined. 

I'd rate the solution at an eigh tout of ten overall. 

We had an issue in the past and we're looking further into the issue.  

Ability to run many different browsers and tests in parallel. This has allowed us to keep our total test times under 25 minutes (even as we've continually added hundreds more tests and up to 10 browsers) and run tests more often—decreasing the time from bug introduction to detection.

Being able to test mobile devices (iOS, Android, and different browsers on each device) has helped us catch tricky mobile-specific bugs in the past.

We've also found the web-based review features (video playback, step-by-step screenshots, "click to manually take over test", full logs) and API handy for diagnosing failing tests and looking back at the result and timing history.

We also found Sauce Labs to be more flexible than alternatives during the setup phase—if you go just one parallel instance over the limit, for example, I noticed they won't instantly fail your tests like another major provider does, which saved a lot of time and frustration when getting parallel tests set up to start.

We now find most browser-specific breakages before they hit production. This lets us push code out knowing it won't break in specific browsers, letting us be bolder in making larger changes across our codebase, frontend to backend.

We are now getting our Sauce Labs tests integrated with our CircleCI tests using Sauce Connect, which should dramatically decrease the time-to-issue discovery and save us time tracking down bugs once our tests catch them.

Finding ways to debug our flakier tests is an ongoing pain for us (we've fought with our own tests' flakiness across all different providers, often e.g. due to some test steps using implicit rather than explicit waits). We could always use more insights in to the causes of different failures, common advice when a failure type is detected, or more detailed guidance on best practices for test runner setup.