OpenText LoadRunner Cloud Reviews

We use a warehouse management system, my company is like a Dutch Amazon. We use LoadRunner to performance test the warehouse management system for peak load. So if Valentine's Day, Christmas, or whatever comes up, we try to run all functional loads through LoadRunner Cloud and make sure the peak count can be handled by the WMS. We use it almost daily. We have peak periods where the testing will increase, but nowadays it's so busy that throughout the year, it's almost the same. It's the same high load every day and we're testing continuously.

The amount of load on the system wasn't that large a couple of years ago for us to test. So the business case for performance testing wasn't pointing to investing in it, but now we have such huge, increasing numbers and increasing load that it's worthwhile setting up a structured and automated performance test and to invest in something like LoadRunner Cloud.

We use the cloud. The controllers and the dashboard servers are in the Micro Focus cloud. But we use our own load generators, which are on-premise. So we use a hybrid setup.

Before LoadRunner, we weren't even testing performance and now we're performance testing. So we now do performance testing as part of the delivery of a new release, which is an improvement compared to a couple of years ago. 

The TruClient feature is the most valuable for us. An application with testing can only be scripted using TruClient, so it's part web-based, but it also has its own protocol combined with HTTP and HTML. So many other tools do not recognize this specific proprietary protocol. Using TruClient, we can still create scripts that cover everything that we need to cover.

It supports multiple protocols, which is important to us. We also use JMeter open source combined in LoadRunner Cloud. In our company, we also use open-source performance testing tools like JMeter, Gatling, and Locust. It's important to be able to embed those scripts as well into a LoadRunner-based performance test. So it would really help us to have a single place for all kinds of test scripts and the open-source ones.

It's also very important to us that LoadRunner allows full integration into the ACI pipeline. We were focusing on a shift left on not only functional testing but also performance testing. And we're trying to integrate the performance testing in our CI/CD pipelines currently. So the better it supports this, the better and easier it is for us.

Reporting and analysis need improvement. Compared to the old school LoadRunner Windows application, the reporting and analysis are mediocre in LoadRunner Cloud.

I've been using LoadRunner since I started at this company one and a half years ago but the company has been using it for longer. 

We don't have any issues with stability. 

When it comes to efficient and cost-effective scalability, it can be expensive. It is efficient, but also expensive. So the balance would then be just okay.

It's used intensively by the three performance engineers, daily. We have plans to have other teams also use LoadRunner Cloud. So we hope it will increase.

We do our own maintenance on the on-premise part and Micro Focus does the maintenance on the cloud part.

We don't call customer support but we do online requests. We have put in service requests. They're helpful and the replies are quick and valuable. It helps us out.

We make use of virtual user hours. We buy time in the LoadRunner Cloud. It costs around $80,000.

I don't think there are any additional costs to standard licensing.

We tried Neoload and looked at JMeter. We chose LoadRunner because that was the only performance testing tool that had something like TruClient.

We're thinking about using LoadRunner Developer integration. We want to hand over the performance testing responsibility from our performance testing guilds to the actual feature teams, to the scrum teams, so they can do it themselves. The developers do their own performance testing in an early stage, and maybe that is a solution for that.

My advice would be to look at whether you need on-premise load generators or fully on the cloud. Invest in training. It can be complex. You need some training to get started.

I would rate LoadRunner an eight out of ten. 

• Load generator setup is not required so we can be running high concurrency tests in few minutes.
• Scenario setup for running tests was very easy.
• Distribution of the load from different locations was very easy.
• SiteScope and New Relic monitoring tools can be easily integrated with StormRunner Load.
• SLA configuration was available to monitor response times.
• Navigation breakdown report - to create a client navigation breakdown report.
• Real time graphs for monitoring results.
• Email reports.
• Benchmark- comparing previous test results with current tests.
• Anomalies detection.

Being able to setup and run 300,000 load tests using LoadRunner Performance Center takes around 1 to 2 hours when creating four scenarios of 75,000 each, including allocating/blocking load generators required for testing, initializing the fusers and running the test.

Running a 300,000 test was very easy using StormRunner. We just had to upload the script and create scenarios for 300,000 and run the test in 20 minutes.

Running back to back high concurrency test was very easy using StormRunner.

• Improved PDF reports and creating custom reports
• Real world- custom scenario setup
• Update existing scripts online without uploading them again
• Run time settings update in scenario

I've used it for five months.

None - there's no deployment required for running a load test.

During the initial days of high concurrency testing (300,000 and 600,000) we had issues with the StormRunner infrastructure e.g. graphs “disappeared” in the middle of a test, and HP had to give an online fix. Also, there were cases where tests appeared as “initializing” even though it actually started and HP again had to give an online fix. The StormRunner R&D team was very proactive in fixing the issues in very small turnaround time and supporting us running the high concurrency tests.

Scalability issues were seen during initial days of testing but it was resolved by StormRunner R&D team.

8/10.

8/10 - their team was very proactive in resolving the issues and provided hotfixes to support us running performance tests whenever required.

We were using Performance Center to run performance tests, so we had to switch to StormRunner to run high concurrency tests.

As StormRunner Load is a web-based application, it's simple setup, just registering users through an HP site.

In-house implementation and evaluation.

There were no other solutions evaluated.

StormRunner is one of the fastest-growing cloud based performance testing tools for running high concurrency tests very quickly.

We use the solution primarily for executing the load tests.

We are able to see the analysis component, and it automatically generates a report from the tool itself after completing the load test. I like that feature. It has great automated reporting features where the report will be generated from the tool itself.

It's user-friendly and it supports a lot of protocols.

The solution can scale.

The stability is good. 

While executing the test it automatically refreshes and I need to see the downwards component, however, it's not allowing me to see that. since it's automatically refreshing, I'm unable to see all the lines in that tool that we are seeing in the editor window.

We did have some challenges with the initial implementation.

I've used the solution for about one year.

I like the product. It's very stable. There are no bugs or glitches. It doesn't crash or freeze. It is reliable.

We can scale the product as needed. It's very easy to do. 

It's also good for any size of the company, from small to enterprise. 

Right now, I'm the only person that uses the solution in our organization. However, we use it to execute the tests with 800 users.

I use it daily. It's used for a quarterly test. For every quarter we'll be executing the testing required.

Technical support has been good. We find them to be quite helpful and responsive. 

Positive

We initially had some problems with the setup as we had an issue while configuring load generators with the tool. In the end, everything went well. We raised it with the Micro Focus team, and they resolved our issue.

I'd rate the initial setup a four out of five in terms of the ease of implementation. 

We handled the initial setup in-house. We didn't need any outside assistance. 

I'm not aware of the cost. I don't handle that aspect of the product.

My understanding, however, is that it is a little bit pricey. 

We have looked into other tools and find that this tool, although expensive, has many useful features. 

We are just a customer and end-user of the product. 

Even though the cost is expensive, there are good features when compared to all other tools. I'd recommend it to others as something they should use. Of course, it does depend on the applications a company uses. 

I'd rate the solution nine out of ten. 

The solution is used for load performance testing activities.

The product supports a wide variety of technology compared to any other tool.

The product must improve support. The support team provides delayed responses. Many times, there are a lot of conversations between the support team and the technical team without any conclusions.

I have been using the solution for 14 to 15 years.

The tool is not perfectly stable.

Around 30 to 40 people use the tool in our organization.

The installation is not difficult, but it is a pretty lengthy process.

The product must improve its price. The solution’s price is considerably high. It is the highest in the market.

My company decided to use the product due to the technology support it provided. I would recommend the solution to someone with technology that other tools couldn’t support. Overall, I rate the solution a six out of ten.

We use it to get response times around the world for external applications. We use it on top of our own virtual users for remote site testing. We include it in our scenarios in our performance center and use the results in the LoadRunner analysis tool.

We can determine response times for our external applications, from most places around the world, for a small fee.

The most valuable feature is having load generators in countries where we don’t have access to them.

An area for improvement is analytics on why response times are slow from certain countries. Slowness can be caused by the laptop, the connection, the VPN, the internal network, the country's internet, etc.

It has been stable every time I have used it. We don’t put much stress on it since we have our own performance center and load generators.

In terms of integration and configuration of the product in our AWS environment, it is the easiest Micro Focus tool I have ever used.

The pricing is very reasonable and the licensing is straightforward.

You decide on the number of virtual users you will need to support multiple runs or for a year and purchase that quality of virtual-user hours. We chose to procure this solution via the AWS Marketplace because of remote site testing and low cost.

I rate this solution a ten out of ten. It addresses a need in performance testing tools and is easy to purchase and use.

StormRunner Load reduces the overall time for performance testing engagement and is best suited for the products in agile process. Easy to automate and execute the tests, StormRunner Load provides a feature to compare the current test results with the previous results to see if the current build is better than the previous or not.

With the ability to integrate with monitoring tools like SiteScope and APM tools like New Relic, performance bottlenecks in the applications can be identified very quickly and easily.

Protocol Support

StormRunner Load supports Web, Web Services, SAP Web and JMeter scripts

Easy Automation

StormRunner Load uses Truclient and TruAPI (which uses node.js) which enables developers and testers to quickly automate the user actions at the browser level, which also helps measuring the end user experience by measuring the browser rendering times as well. Performance testers can upload their LoadRunner scripts that are developed using LR 12.XX to StormRunner Load directly.

High Scalability

StormRunner Load is highly scalable and supports upto 1 million concurrent users.

On Premise Load Generators

StormRunner Load introduces the concept of On Premise Load Generators to compare the application performance within the network to various other geographical locations.

Configure SLAs

SLAs can be configured in StormRunner Load during the test execution. The tool highlights all the transactions that did not meet the SLAs and further drill down can help the users understand the geographical locations and times where the SLAs are not met.

Real time anomaly detection

During the test execution, StormRunner Load detects any anomalies where the test results are deviated for a specific transaction / geographical location / script / counter to identify the issues real time and further drill down provides more details about the anomalies.

Integration with Monitoring Tools

StormRunner Load integrates with monitoring tools like SiteScope to provide the hardware and application level metrics during the performance tests. StormRunner Load can also integrate with New Relic and other similar APM tools that helps identifying the server side bottlenecks so that organization can reuse investments

HP StormRunner Load has provided us a) flexibility for on demand load generation b) reduced over all TCO on our load testing efforts c) Reduced the skill level to develop load test scripts.

Custom load patterns for the performance tests: Currently StormRunner Load supports only linear ramp up of the user load and doesn’t provide the user load to follow a step incremental load pattern or any other pattern that a performance tester wants to.

Load distribution among on premise LGs: StormRunner Load currently supports load generation from only one of the On premise load generators and the user load cannot be distributed to multiple on premise LGs.

Simulation of latencies: There is no provision to simulate network latencies to simulate realistic geographical locations and mobile networks.

Ability to create custom reports: StormRunner Load provides only the default reports and there is no provision available currently to generate the custom reports

Backward compatibility: StormRunner Load expects the LoadRunner scripts that are generated from LR 12.00 and above. Any scripts developed using the older versions are not supported.The work around to overcome this problem is to open the scripts in LR 12.00 or higher and save it again before uploading to StormRunner Load..

We have used the product for more than 14 weeks now and tested on some of our internal applications in the organization that would require performance testing.

HP StormRunner Load doesn’t need any installation or deployment unless you wanted to configure few on premise load generators. Configuring on premise load generators is pretty straight forward. All that you need is to have an account with HP SaaS to start using StormRunner Load.

HP SaaS web application is pretty stable and no issues were seen during the evaluation process.

There were no issues with the scalability, however, the time taken to initialize the test is directly dependent on the number of users to be simulated.

5/5.

The support provided to us very good which helped us to evaluate the solution for our context.

We are in the process of adopting a SaaS based option, this led us to evaluate a bunch of tools.

It is simple - just registering with HP SaaS platform called pronq.com.

We evaluated through our in-house resources.

Our estimated ROI is positive we need a couple of months to gain insight into actual ROI.

HP StormRunner load is SAAS offerings so the initial setup costs are very minimal when compared to the traditional approach (no or minimal capex ), and the day to day costs would be purely driven by the duration and number of virtual users required to simulate the traffic.

We have evaluated HP StormRunner Load against other market competitors that include Blazemeter, SOASTA’s Cloud Test, NeoLOAD, VSTS, Load Complete and StressStimulus.

HPStormRunner load is an apt product a) if your application stack has Web 2.0 traffic b) your requirement is for a service that allows for pay as you go model c) proven and reliable load testing solution with superior support.

This review was co-authored by Sravanthi Naga & Ramanjaneyulu Narra, Performance Architects at Gallop Solutions. Gallop is a US based Co-located Independent Testing Services operating since 2003 with offices in Philadelphia & California. World's largest and leading organizations have relied on Gallop's specialist independent software testing services for more than a decade and have achieved significant market acceleration, returns on investments in their software quality initiatives.

I have been using the solution for testing Salesforce and a few other applications.

The solution has improved the performance of the software.

The solution is intuitive, easy to use and navigate.

They should minimize the use of coding for the solution. Also, they should include features to import other scripts without rewriting them.

I have been using the solution since 2019.

It is a stable solution.

The solution is scalable. I rate its scalability between a seven and nine out of ten.

The solution's technical support has been good.

Positive

We have used StormRunner and JMeter before. Also, we are doing a proof-of-concept for NeoLoad.

The solution's setup process was easy. Although, we encountered a few issues sometimes.

The solution is expensive. There should be a version that optimizes the price-to-performance ratio for the service package.

The solution is costly. I rate it a nine out of ten.

The reason we chose it and the most valuable feature is the amount of virtual users we can run in a test. We have run up to 900,000 users.

We do online testing for the States as we have contracts with them. We have to be able to support a certain number of students, so this helps us confirm and validate that we can support the number of students that are taking online testing at one time.

I think the reporting could be better. I think you could have a little more variety of reports. Right now, you get an email saying, "Here's your report". You can't put in the analysis tool. You can't export it to Excel. Those are the things that would make it a little better. That's probably my biggest pain point, it is the reporting.

I think it's fairly stable. There's some features that I think could be made better but, for what they have, I think what they have works well.

I think it scales really well. I like that you can add users on the fly or remove users on the fly. That's a great feature.

I have used the support. We were one of the first users for Beta, so we have a relationship with the developers, so we tend to just go right to them when we have questions or problems.

They've been great. A lot of the times, it's not really a problem with StormRunner, it's just our understanding of what it's supposed to do, so they are very great about explaining the reasoning and what will be coming in the future. If we don't think that's really what we want, they'll explain how it's in the roadmap and how it's going to be made better.

We were using another HPE product, Performance Center. We haven't totally switched, but StormRunner Load is our option for the test where we need a lot of virtual users where we can run it in the cloud. Whereas, on Performance Center, you have to have load generators in-house, or what HPE provides. That's why we use it. It's for when we want to use a lot and perform a big test.

I wasn't involved in the deployment.

Not really because we're mostly an HPE shop and this was working with HPE they to meet our needs, so they provided this and it's worked out great.

I don't like their reporting. Other than that, I think it's great. It does what it's supposed to do and I know it's just the beginning, so there's going to be a lot more features added.

You should look into it. I know there are other solutions that have more bells whistles, but,when it actually comes down to function, I think StormRunner does the job.