It gives us the ability to quickly script a load test scenario and to develop complex testing scenarios. If you ever want to know more about your application, just recording a session can be invaluable.
Vice President of Technology at a retailer with 1,001-5,000 employees
It provides data that provides executive-level comfort during peak periods, driving direction for future performance-oriented projects and goals. However, more community collaboration would be good.
What is most valuable?
How has it helped my organization?
Outside of the obvious benefits of load testing, the data we get back and are able to provide at the executive level provide a level comfort in the given application at peak periods, and drives direction for future performance-oriented projects and goals.
What needs improvement?
They have done well with patching and enhancements in each version. More of a “community” of shared experiences, help, and collaboration would be a good enhancement to their site.
For how long have I used the solution?
We've been using it since 2011.
Buyer's Guide
Tricentis NeoLoad
December 2024
Learn what your peers think about Tricentis NeoLoad. Get advice and tips from experienced pros sharing their opinions. Updated: December 2024.
831,265 professionals have used our research since 2012.
What was my experience with deployment of the solution?
No issues encountered.
What do I think about the stability of the solution?
At times we have had issues with load generator agent stability, but tuning the heap and documented network hacks greatly reduced this.
What do I think about the scalability of the solution?
No, it's very easy and quick to scale.
How are customer service and support?
Customer Service:
7/10
Technical Support:7/10
Which solution did I use previously and why did I switch?
No, not previously but we have demonstrated or performed proof of concepts of other products.
How was the initial setup?
The infrastructure setup was very straightforward, and getting started with our first test was relatively easy with online videos and resources.
What about the implementation team?
We did it in-house.
What was our ROI?
Difficult to really estimate, however we do know that by load testing we have found countless issues over the years that would have affected our e-commerce revenue had these issues made it to our production environment.
What's my experience with pricing, setup cost, and licensing?
We evaluated other products of similar nature as well other team member’s prior experiences with similar products.
Which other solutions did I evaluate?
We evaluated other products of similar nature as well other team member’s prior experiences with similar products.
What other advice do I have?
Start with some small, simple scenarios and expand complexity from there. Your project will take shape quickly, and something that seems overwhelming and complex will all of a sudden will not be so hard. Use the online resources and videos.
Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
Penetration and Neoload Tester at a university with 501-1,000 employees
Vendor
Apr 13, 2014
LoadRunner vs NeoLoad
The six phases of an IT project:
- Enthusiasm
- Disillusionment
- Panic
- Search for the guilty
- Punishment of the innocent (the performance tester)
- Praise and rewards for the incompetent non-participants
This article has been put together as part of an evaluation of the performance test tools NeoLoad and LoadRunner. I consulted a variety of sources, including user groups, discussions with colleagues, telephone chats with HP (the vendors of LoadRunner) and Neotys (the vendors of NeoLoad), and of course the Oracle – Google. The opinions in this article are my own and do not represent the views of any particular company, the software vendors or any organisation (and I’ve assumed that readers have some knowledge of web application performance testing).
The versions discussed in this article as of April 2014 are:
LoadRunner 11.52
NeoLoad 4.2.2
I hope this evaluation and comparison will assist you with your own evaluation of LoadRunner and NeoLoad.
Introduction
LoadRunner and NeoLoad are considered the top two best performance testing tools on the market. Comparing the two tools is just like comparing a Mercedes Benz and a BMW: both are high performance, perfectly engineered machines that also have an associated prestige.
Performance testing is a critical component of the software testing process. It determines the actual operational boundaries that will simulate the real world use of an application. Performance testing is load testing, stress testing and scalability testing.
Why are we here?
The philosophy behind performance testing is we don’t want any of our IT systems to crash – it’s terribly embarrassing and can be very damaging, never mind expensive, when a crash happens.
A load test determines how a system behaves under a particular workload. The main objective is to see how various components react to a gradual increased ramp up of workload. The usual outcome of load testing is throughput, response times, CPU load and memory usage.
Stress testing determines the breaking point or the unacceptable performance point in a system. It reveals the maximum service level the system service can sustain.
Scalability testing evaluates the effects of adding additional hardware, virtual or physical, in order to distribute work among system components.
The following evaluation considerations have been taken into account for LoadRunner and NeoLoad:
- Record and playback
- Recognizing dynamic web components known as ‘correlation’
- Data functions – the ability to use data from a text file to populate web forms resulting in realistic test data
- Randomising – picking random datasets in rows, check boxes and radio buttons
- Putting it all together and orchestrating the load simulation and analysing the system response
LoadRunner is written in C++, and NeoLoad is written in Java. LoadRunner scripts after recording can be manipulated and debugged using C. NeoLoad is a scriptless tool – instead, it involves dragging and dropping functions into a script. Also IF statements and loops can be dragged and dropped. Even exception handlers are controlled the same way within NeoLoad. There are some good and some cumbersome qualities attached to both – the NeoLoad feature is less powerful than the LoadRunner vugen feature, albeit LoadRunner takes more hours to script record, parameterise and debug.
Recording scenarios
If you’ve ever used a proxy debugger like fiddler, you’ll see what’s happening behind the scenes when surfing the internet. There are send requests and responses from your computer’s browser to the website’s web server. Recording load testing scenarios works just like a proxy: when the record button is pressed, it actives a proxy that acts like a ‘man in the middle’ capturing http requests and responses from the system’s web servers.
LoadRunner architecture
LoadRunner, which has a trial version, consists of three components: the virtual user generator, also known as vugen (script recorder and editor), the controller where all the vugen scripts are controlled into a scenario, and the Analysis, the results component. These enable the user to keep scripting in vugen while a scenario is running in the controller component, which usually happens because the load tester is hired at the end of the System Development Life Cycle (SDLC) and is under a huge amount of pressure.
NeoLoad architecture
NeoLoad consists of one component encompassing all-in-one software, which means that if a load scenario is running you cannot create and edit scripts.
Recording web with LoadRunner
Nowadays Chrome and Firefox work perfectly as recording browsers in LoadRunner’s vugen. Once upon a time, just Internet Explorer worked. Other browsers made vugen crash when the recording button was pressed, so LoadRunner consultants were limited to the browser people use to download other browsers.
If you’d like to get the time it takes for a user to login to the system under test, press a simple ‘Start Transaction’ button during recording – you can name the transaction appropriately, log the user in, then end the transaction when the logged in page has downloaded.
The transactions look like this for a simple login script:
Recording web with NeoLoad
NeoLoad records scripts in a ‘tree-view’ like structure. A wizard pops up after recording to automatically correlate the recorded script for you. Start and end transactions are not like those in LoadRunner – they are called ‘containers’ in NeoLoad and there is no end to the container as the ‘end’ of the container is actually the beginning of the new one, like this:
Correlating
Correlating is extracting dynamic values and placing each dynamic value into a variable from GET requests and substituting it into the POST response, usually in the request after the GET. This is the most time-consuming task in load testing – just ask any performance test analyst.
Correlating with LoadRunner
Design Studio is the tool LoadRunner’s vugen uses to try to correlate dynamic values in a newly recorded script.
‘No correlation parameters found’: I glanced at the recorded script below and found many dynamic parameters.
So why didn't the automatic correlation utility work for LoadRunner? This is a good question considering some of the other protocols that LoadRunner uses correlate really well. Like me, many of my colleagues have become used to manual correlation when using LoadRunner for http/web.
In fairness to LoadRunner, it can find one or two dynamic parameters, depending on the size of the recorded script. It looks like this:
It works (sometimes).
Manually correlating with LoadRunner involves placing this statement to grab a parameter from every dynamic GET request, which you’ll have to then place before a POST request if required.
Correlating with NeoLoad
Correlating is definitely where NeoLoad has an edge over LoadRunner. NeoLoad takes hours of painstaking manual correlation work out of the equation, and that saves bucket loads of cash in consultant fees and maintains deadlines. Any IT manager will know how much a LoadRunner contractor costs daily. If that figure is going to be potentially halved (or otherwise cut down by the speed NeoLoad can do the job), then maybe the next word that enters the manager’s head is ‘bonus’ or even ‘gratitude’, for finishing the project on time and within allocated budget parameters. Gratitude goes a long way.
There is some manual correlation in NeoLoad that involves a not-so-very-cumbersome two clicks of a mouse.
NeoLoad is so intuitive that if the same extracted value exists in subsequent requests within the scenario, this pops up (music to my ears):
And if you’d like to replace all, then:
The music keeps on playing – click, click, click ... and done.
Playing back debugging/recorded script with LoadRunner
Debugging is instantly more informative in LoadRunner than in NeoLoad as you can see the script executing and outputting to the log (circled below). It’s easy to watch variables and check for errors in real-time script compilation and execution.
Playing back/debugging recorded script with NeoLoad
One quick and easy way to check and debug a script in NeoLoad is to use the virtual user checking option by right clicking on your script. This runs the script and shows errors in requests that need to be addressed.
In NeoLoad, a more involved debugging analysis (after checking using one method) is to use JavaScript. Do this by dragging the JavaScript function into a script and coding the appropriate write to a text file. All variables can then be sent to a text file and used as a watch list.
NeoLoad will also check the JavaScript syntax for you.
Parameterising
Parameterising is adding a text or csv file to a load testing script. For example, when a script is recorded, there is a hard-coded user id and password in the script. Creating a file of five user ids and passwords and thus adding to the script while replacing the hard coded values is parameterising. You can control this, for example, using the five user names – the script can randomly pick the user id and password or it can pick them one line at a time incrementally.
Parameterising with LoadRunner
I’ve always found parameterising with LoadRunner straightforward, as long as you’re aware that creating a new parameter creates a new .dat file as well. It’s easy to have a user name and password in the same file (like above) instead of in two separate files. You can also edit the text file in Notepad, which makes pasting large amounts of data relatively easy.
Variables in LoadRunner
Where variables need to be declared as well as the variable type, like char or int etc., and values need to be added to them during the script execution, then do the following:
Parameterising with NeoLoad
Adding datasets to NeoLoad isn’t too dissimilar to LoadRunner. Also you can import spreadsheets into the variable manager or just create datasets manually.
Variables in NeoLoad
NeoLoad has an edge over LoadRunner, as none of the above coding is necessary. Variable types are added using the variable manager as in the screenshot below:
NeoLoad has an added SQL variables feature that queries the backend database for test data.
Randomising user input/actions
A very important aspect of a realistic load test is to have as many virtual users doing as many different things as possible to the system under test. The more random actions the better, such as if there are 10 links for two vusers to click, then there should be a one-in-forty-five chance that the two users click the same link (if you don’t know what I mean, check out permutations on Google).
Randomising with LoadRunner
Picking a random row from a result set you’ve just recorded with LoadRunner involves doing something like the code below. First declare the variables, then using the web_reg_save_param_ex() built in LoadRunner function, grab the number of rows, place the number of rows into a random number function between 1 and 10 (if there are 10 rows), and finally get the script to click on any of the rows – this should be different for each iteration in your script, depending on the test you’d like to conduct.
Randomising with NeoLoad
Randomising with NeoLoad is very easy, but to randomise you have to record every request and place them in a container. To give you an example, imagine you are filling out an online form to register for an online shopping site (see the screen below). Just before you submit the form you have the option of subscribing to the store’s newsletter, and also entering into a draw if a radio button is checked. Now that’s four combinations:
- 1.Submit form
- 2.Submit form + tick newsletter
- 3.Submit form + tick prize draw
- 4.Submit form + tick newsletter + tick prize draw
In NeoLoad you have to record this randomising scenario four times and place each into a container. It’s then possible to control the randomise settings for a realistic and valid load test.
The fun part – performance testing
After all the hard work is done – the correlation, the parameterisation and getting admin access to every single pre-production server in the system – it’s time for the real fun. Every performance tester loves this part because it’s the breaking stuff part: Lego for grown-ups.
Other forms of load testing include what hackers do – distributed denial of service (DDOS) attacks. Well, that’s more like a stress test.
IP spoofing can be implemented with both LoadRunner and NeoLoad – this means that an even more realistic performance test can take place. You don’t want a router in the network to reroute the web requests to a honeypot if there’s a rule in place that redirects heavy traffic away from the system under test.
Load Testing using the LoadRunner controller
All servers in the system under test can of course be added into the load scenario, so that real-time monitoring of server resources can be directly correlated to the number of vusers being ramped up, to a peak hour load test. So for the layman, let’s see what maxes out first if 1,000 users login to your website all at once, or all in the space of five minutes – things will start to break depending on a lot of factors, too many to discuss in this article.
Server statistic graphs can be added to the load scenario, and all the gathered stats as well as transaction time taken can be collated in LoadRunner’s Analysis component. A performance Test Summary Report (TSR) uses a template that can be modified to your liking and then presented to your IT Test Manager. Make sure you let him or her know that you typed the report from scratch – work is all about perception after all.
The client controller component of LoadRunner has all the bells and whistles needed for a realistic load test. There’s only one drawback: there’s a limit on the number of performance counters that can be added to a scenario. If there is a requirement for more performance counters you have to install and configure monitoring software called SiteScope. All servers in the system have to then be added to SiteScope after SiteScope and the SiteScope license server are installed.
Load testing using NeoLoad
The ‘runtime’ component of NeoLoad and the LoadRunner controller are on a par in terms of the capabilities they have for you to add the appropriate graphs.
LoadRunner vs NeoLoad – the cost
Here’s what every IT manager wants to know: Which tool is cheaper? Is it value for money? Now this is a tricky one: How much does LoadRunner cost?
Performance testers have always known that when it comes to price, LoadRunner is not a BMW or Mercedes – it’s a Ferrari. I wasn’t very successful in my endeavours to obtain a quote for LoadRunner. I contacted HP in London, and they referred me one of their local partners here in Australia. My enquiry was pretty simple: I wanted a quote for a 5,000 user Web vusers license, but HP’s partners said via email that they needed more information.
Using Google to ascertain the cost of a LoadRunner license had mixed results. I came across an old article mentioning that a 5,000 vuser perpetual web license with support for three years was about a million dollars. That article was written in 2005.
When I contacted Neotys in France, they were a little more helpful with a licensing quote for NeoLand, and there was a lot of room for negotiation as well, including email support, monitoring for Oracle and a 5,000 vuser license that never expires.
Based on my very ball-park figures, there’s an approximate cost correlation between NeoLoad and LoadRunner. Imagine that LoadRunner costs $100:
LoadRunner: $100
NeoLoad: $10
NeoLoad is about 10% of the cost of LoadRunner, and Neoload includes technical support as well as perpetual licensing.
There are also several product packages associated with both HP and Neotys that can be negotiated as per your allocated budget, so keep that in mind. All you have to do is pick up the phone and talk to them.
So there you go – my take on the top two best performance tools on the planet. Taking into account industry experience, time frames scripting, hours spent completing the job (usually after working hours) and downright value for money, the winner has to go to NeoLoad. It just works.
If your company has spent its money on LoadRunner and highly paid consultants developing test scenarios for your application, then the money has not gone to waste. If your company wants a cheaper product that does the same job as LoadRunner, then evaluate NeoLoad and ask an experienced performance tester’s opinion on the tool. The downside to this option is finding experienced NeoLoad consultants.
LoadRunner is the preferred tool for large organisations, such as banks and financial institutions. Right now, NeoLoad is the preferred performance testing tool for universities and smaller organisations.
Some final questions
My employer has been using LoadRunner for performance testing for many years. How expensive would it be to hire people to convert these scripts to NeoLoad?
Weighted Rankings 1 – 5 (5 is excellent)
Can LoadRunner scripts be migrated to NeoLoad?
Yes, and vice versa. It doesn’t take as much time or resources as you think. Companies don’t need to hire specialists to re-code LoadRunner scripts into NeoLoad scenarios.
Disclosure: I am a real user, and this review is based on my own experience and opinions.
Buyer's Guide
Tricentis NeoLoad
December 2024
Learn what your peers think about Tricentis NeoLoad. Get advice and tips from experienced pros sharing their opinions. Updated: December 2024.
831,265 professionals have used our research since 2012.
Test Analyst at a tech services company with 10,001+ employees
Recording a .NET application worked well.
Performance Architect at a tech services company with 10,001+ employees
I like the integration with various monitoring tools.
Digital Performance Test Consultant at a tech services company with 10,001+ employees
When it comes to mobile HTTP load testing, this helps to script quicker. I would like to see reporting improved.
Easy to build test cases, good stability and good scalability
Pros and Cons
- "The test cases are quite easy to build and to maintain. This is the most valuable aspect of the solution for us. It's the reason why they changed from JMeter to NeoLoad."
- "Sometimes it's complicated to maintain the test cases. It's much easier than in JMeter, however. I'm not sure if this depends so much on NeoLoad, or is more based on the environment that we are testing."
Performance Test Architect and HP ALM Expert at a tech services company with 10,001+ employees
I was able to use this product in the past for simulating mobile performance after recording a script over the LAN. I ran into installation issues when the procedure kept hanging.
Buyer's Guide
Download our free Tricentis NeoLoad Report and get advice and tips from experienced pros
sharing their opinions.
Updated: December 2024
Popular Comparisons
Apache JMeter
Perfecto
OpenText LoadRunner Professional
BlazeMeter
OpenText LoadRunner Cloud
OpenText LoadRunner Enterprise
ReadyAPI
Oracle Application Testing Suite
Eggplant Performance
RadView WebLOAD
Akamai CloudTest
SmartBear LoadNinja
ReadyAPI Performance
StresStimulus
Buyer's Guide
Download our free Tricentis NeoLoad Report and get advice and tips from experienced pros
sharing their opinions.
Quick Links
Learn More: Questions:
- Experiences with NeoLoad on Mac OSX?
- Do you recommend Tricentis NeoLoad?
- When evaluating Load Testing Tools, what aspect do you think is the most important to look for?
- SOAtest vs. SoapUI NG Pro?
- Does Compuware have a manual testing solution? Which manual testing solutions should we be considering?
- What are the top performance tools available to load test web applications?
- What is the best tool for mobile native performance testing on real devices?
- When evaluating Performance Testing Tools, what aspect do you think is the most important to look for?
- Cost of TOSCA Testsuite?
- Do you have an RFP template for Testing Tools which you can share?
Hi Good article, can u suggest how to migrate neoload script to vugen one..