We primarily use Sauce Labs to test browser compatibility. It's mostly functional rather than performance testing. We use a combination of tools, but Sauce Labs is mainly for compatibility testing. Selenium is our backend, and it also has compatibility testing, but we're not making use of that feature. Selenium is for capturing.
Our custom framework for testers combines Selenium and third-party vendors to do some of those performance metrics. At the same time, we use Sauce Labs to test cross-browser compatibility for the top five browsers that the government requires us to support.
We automate tests of our on-premise solution with Sauce Labs via tunnels. Sauce Labs allows individual testers to log in and test whatever they need, but we don't do it that way. Instead, we use the automation features through tunnels, and our CI/CD process will run tests for us through Sauce Labs. It returns metrics on compatibility and usage for us to review.
Our two major platforms are Windows and Mac. We don't run tests on Linux. Even though we build everything on Linux, we don't support that for our end-users. We test our applications on the two main operating systems and variations of Safari. OS X testing is the main reason we started using Sauce Labs because we needed to test our applications on Safari, which isn't available on Windows. Initially, we purchased some Macs to do compatibility testing, but that didn't prove helpful at all.
We also need to test on all Chrome variations because there are multiple versions we need to support. When we launched, Microsoft had just released Edge, so very few of our users had it, but just about everyone has migrated from IE to Edge by now. Testing on variations of Firefox, Chrome, IE, Edge, and Safari is our essential requirement.
It's easy to set all that testing up on Sauce Labs. We could use a virtual machine to run applications on all the browser variations, but you need to get people in there to connect to it, and a homebrew solution is way too complex. With Sauce Labs, it's all already there. We just spin it up, specify the version we need, and we're done.
Sauce Labs doesn't give us immediate feedback on every code commit. That's not how we have it set up. We've got a multistage process, so it goes through a code review for quality when we do the commit. We have unit tests that happen along the way, but when we do a full-blown merge and are ready for a release, that's when we actually launch our tests, and the tests run overnight. There are thousands of tests, which is why we don't do it on every code commit, but we do it every night. When a nightly job runs, we run a full regression test on that to get the results the following day.
From an automation standpoint, Sauce Labs enables us to have a fully functional CI/CD process while saving time and cutting down on the training required for individual testers. We can easily automate tests without training a whole bunch of people on Sauce Labs. There's a whole slew of quality and security testing tools that a tester typically needs to know. Automating and integrating with Sauce Labs reduced the number of things a tester needs to be trained on, and they don't need to use Sauce Labs daily. They just see the results.
It's one less aspect the testers need to worry about. They come in from time to time to review the results, but it's automated, so they don't need to connect to Sauce Labs, run the test, and get the results back. They don't even have to know that it's Sauce Labs behind the scenes. We take care of that for them, so it saves us time.
Automation allowed us to reduce the size of our team. When we initially got Sauce Labs, we had a full core development team with a lot of testers. There were 50 to 70 testers working 100 hours a week. That's around two hours each day per tester on the low end, so 50 multiplied by two is 100 hours per week. Multiply that by six to eight months of testing.