What is our primary use case?
Following Microsoft’s Visual Studio Web & Performance testing tools being deprecated, StresStimulus had been adopted to continue performance testing work of an enterprise scale application with web-based user interfaces for customer interaction & back-office processing.
StresStimulus is being used to validate the non-functional requirement commitments made to clients, ensuring the product satisfies the volumes of concurrent users specified in contractual documentation and helping to identify system performance bottlenecks that can then be investigated and better optimized by development teams.
How has it helped my organization?
StresStimulus is being used as an alternative to Microsoft’s Visual Studio Web & Performance testing tools used previously, as the Microsoft tools have been deprecated. StresStimulus has been beneficial as it is a product in active development with a responsive support team, whereas, in contrast, the Microsoft tools are no longer supported or developed.
For small simple test cases and scenarios, StresStimulus offers similar functionality to the Visual Studio Web & Performance testing tools. However, for larger, more complex tasks, we’re finding it necessary to reduce the scope and scale of aspects of test scenarios that were previously possible with Visual Studio – either capturing fewer PerfMon metrics, reducing the monitoring sample rate, or reducing the concurrent usage profiles & test durations.
What is most valuable?
Technical support is great. StresStimulus’ support team have been excellent. An example (for which this review is being provided in the interests of reciprocity) was their response to our request for enhancements to the way tests were organized and executed.
We required a way to start a collection of tests running which emulate behaviour related to contractual Non-Functional performance requirements of the system, then in addition to that, subsequently also ramp-up another collection of tests to stress the system and identify the point at which bottlenecks, errors & unacceptable levels of performance degradation occur.
A Webex call with their technical staff was arranged immediately to discuss the issues & requirements. A solution was proposed to introduce a concept of ‘Tiers’ to enable sets of tests to be ramped up in a pre-defined order. The functionality was developed within the space of a week and delivered to us as a bespoke build prior to it being merged into the next version of StresStimulus made available to the public shortly thereafter.
What needs improvement?
Source control integration and versioning could be better. Currently, StresStimulus does not natively support any form of change control management.
Result analysis (snap-out tabs for side-by-side comparisons) need improvement. Multiple sets of analysis results can be opened in the user interface at the same time, and comparison reports can be generated to compare some aspects of pairs of test runs. However, the interface that displays the multiple sets of results only allows a single result set to be viewed at any one time & uses tabs to switch between the individual result sets. It would be useful to have the ability to view multiple tabs at once, either side by side in the interface or by snapping tabs out of the interface into their own windows to be viewed together with others.
Graphing (collect PerfMon data as part of the tests and then have the ability to create different graphs with that data after the test completes to help with analysis) could be better. Graphs and their contents have to be defined before tests are run. Often, as a result of running tests and viewing the results collected, it is necessary to have graphs showing/comparing different metrics collections than those specified before the run. Other tools collect metrics that are not tied to pre-defined graphs & allow the metrics to be used to construct graphs as required by subsequent analysis & investigation.
Nested tests/test case groups/test solution files (ability to build a hierarchy of test cases/test case groups/solution files) are required. Within a StresStimulus, tests are organized in just two levels, test cases, and test case groups. As is possible with other tools, it would be helpful to have a hierarchy with multiple levels of nested test cases/test case groups (or even the solution config files themselves). It would make it easier to manage multiple sets, combinations of tests, load profiles & test mixes if a high-level solution file (or test case group) could run a collection of lower-level ones, which in turn end up executing multiple different base-level test case groups and individual test cases.
For how long have I used the solution?
I've used the solution for five months.
What do I think about the stability of the solution?
The solution is stable with frequent version updates.
What do I think about the scalability of the solution?
The most effective way we’ve found so far is to create and manage several sets of different load profiles (number of VUs, ramp-up, test length, etc) and test mixes (which tests are being exercised & how the VUs should be distributed over those tests) is through multiple separate .sslnconfig test solution files. The number of these .sslnconfig files & related folders quickly add up, making the management and maintenance of large numbers of test cases/scenarios/test suites a challenge.
Analyzing results from test runs is proving to be challenging. For example, test runs that take 3 hours and capture PerfMon metrics from 14 ‘System Under Test’ and three ‘Test Rig’ machines – we’ve seen it take two hours to load test result data for analysis after the test completes, as well as experiencing repeated timeouts, errors & analysis reports indicating the tests were aborted.
StresStimulus does not integrate with any source control systems natively – versioning has to be managed by hand with external source control tools making it hard to work as a team on a single performance test project.
StresStimulus currently doesn’t effectively support multiple users working on the duplicate solution files which are being managed using a source control system. .sslnconfig test solution files currently use absolute file paths to locate resources in the solution, these paths differ between multiple users working with .sslnconfig files. This causes merge conflicts & test errors every time different users commit/get-latest the versions to/from source control.
How are customer service and support?
The product has excellent technical support where queries are responded to and actioned quickly.
How would you rate customer service and support?
Which solution did I use previously and why did I switch?
StresStimulus is being used as an alternative to Microsoft’s Visual Studio Web & Performance testing tools used previously, as the Microsoft tools have been deprecated.
How was the initial setup?
The initial setup was simple.
What about the implementation team?
We did an in-house implementation.
What's my experience with pricing, setup cost, and licensing?
Pricing plans are dependent on number of licences and virtual users required for testing.
Which other solutions did I evaluate?
We evaluated Loadrunner and Apache JMeter.
Which deployment model are you using for this solution?
On-premises
*Disclosure: I am a real user, and this review is based on my own experience and opinions.
