What is our primary use case?
The primary use case is for web applications and also SAP GUI—not SAP Applications but the UI version of SAP. For both these areas, it was working fine from the NeoLoad side.
The solution has been moved to the cloud version.
How has it helped my organization?
The first benefit is that it's quite cheaper. In the market there are mainly three tools which dominate: JMeter, LoadRunner, and NeoLoad. JMeter is open source. LoadRunner is quite expensive but has quite good features. NeoLoad is competitively quite cheaper compared to LoadRunner, and it provides great features, except in some areas. For instance, it doesn't allow some SAP applications. There are some areas where it's a little bit tough but other than that, it's quite good.
What is most valuable?
I feel that the codeless part, the dynamic value capture part is quite easy in NeoLoad compared to other tools. JMeter is a little bit tough, but LoadRunner gives you many features. For any codeless value, it is easy to detect from the NeoLoad side.
What needs improvement?
The SAP area could be improved. Not the GUI applications, SAP log-on, or something like that. We can see features and use NeoLoad properly in all the normal SAP log-on areas. I haven't tried it, but I have heard that Citrix-based applications don't work properly in the SAP application. In those types of different areas, there could be more protocol flexibility.
If there is a high number of users on NeoLoad—like around 5,000 or 6,000—it sometimes feels like that might be a bottleneck for the performance. That could be improved. With more users, there shouldn't be any issues from the tool side.
With scripting in a different browser, sometimes there are compatibility issues and it doesn't generate the script properly. Those types of crashes could be removed.
For how long have I used the solution?
I was using NeoLoad six months ago because of some performance testing issues. Previous to using NeoLoad, we used LoadRunner.
What do I think about the stability of the solution?
The stability is good. It's quite reliable. Compared with a different browser, it sometimes creates problems and bugs. With the number of users increasing, sometimes the CPU and system where it was connected increases because of trade concurrency or something like that. That can be a little bit problematic. Other than that, things are fine.
Ongoing maintenance is not required. Whenever we have deployed the tools, if any version upgrade happens, we just check what the upgrade is exactly and someone on the deployment team connects the documents and everything. They check the document accordingly and update the version. It's not complicated.
We have roughly eight to nine performance testers, including the performance lead, the performance engineers, and the performance testers. Everyone will be running different projects. We'll be using the tool and running the performance testing based on the license and everything. The user count license will be ramping up the load. There are instances where multiple teams are using NeoLoad at the same time with different licenses. There has been no conflict error with each other, and we haven't seen a problem with different license things. Because we have two licenses, two tests can be executed parallel to each other.
What do I think about the scalability of the solution?
The solution is scalable. After the number of users increases to 5,000 or 6,000, we have occasionally found problems. Below that, things are fine and the response times are good.
How are customer service and support?
Tech support is okay. When we had 50 users, we reached out to the tech support team and within two to three days we received a response. The time took longer because it was a complicated problem. Response time is perfect.
Which solution did I use previously and why did I switch?
I previously used LoadRunner. One of the reasons we had to switch to Neotys was because it was quite cheaper, but I still prefer LoadRunner to NeoLoad.
How was the initial setup?
The deployment process is very straightforward. It's not complex. Only one associate deployed it, so not much effort was required.
What was our ROI?
There was no impact on the work that we used to do with performance testing when we moved to NeoLoad. Our normal performance testing was able to continue and it was going perfectly well. Return on investment was better because our deliverables were not getting impacted and it was cheaper, so return on investment increased a lot compared to LoadRunner.
What's my experience with pricing, setup cost, and licensing?
NeoLoad is cheaper compared to other solutions. There are no additional licensing fees.
What other advice do I have?
I would rate this solution 9 out of 10. I wouldn't give it a 10 because there are some issues that need to be resolved from the tool side.
Before using the tool, you need to understand the product you are going to design, the different types of performance applications that you are going to do performance testing with, and whether or not NeoLoad supports it. If yes, then go ahead. Otherwise, I would say go back to LoadRunner because whatever the application is, LoadRunner will support it.
Which deployment model are you using for this solution?
Public Cloud
Disclosure: My company does not have a business relationship with this vendor other than being a customer.