The six phases of an IT project:
- Enthusiasm
- Disillusionment
- Panic
- Search for the guilty
- Punishment of the innocent (the performance tester)
- Praise and rewards for the incompetent non-participants
This article has been put together as part of an evaluation of the performance test tools NeoLoad and LoadRunner. I consulted a variety of sources, including user groups, discussions with colleagues, telephone chats with HP (the vendors of LoadRunner) and Neotys (the vendors of NeoLoad), and of course the Oracle – Google. The opinions in this article are my own and do not represent the views of any particular company, the software vendors or any organisation (and I’ve assumed that readers have some knowledge of web application performance testing).
The versions discussed in this article as of April 2014 are:
LoadRunner 11.52
NeoLoad 4.2.2
I hope this evaluation and comparison will assist you with your own evaluation of LoadRunner and NeoLoad.
Introduction
LoadRunner and NeoLoad are considered the top two best performance testing tools on the market. Comparing the two tools is just like comparing a Mercedes Benz and a BMW: both are high performance, perfectly engineered machines that also have an associated prestige.
Performance testing is a critical component of the software testing process. It determines the actual operational boundaries that will simulate the real world use of an application. Performance testing is load testing, stress testing and scalability testing.
Why are we here?
The philosophy behind performance testing is we don’t want any of our IT systems to crash – it’s terribly embarrassing and can be very damaging, never mind expensive, when a crash happens.
A load test determines how a system behaves under a particular workload. The main objective is to see how various components react to a gradual increased ramp up of workload. The usual outcome of load testing is throughput, response times, CPU load and memory usage.
Stress testing determines the breaking point or the unacceptable performance point in a system. It reveals the maximum service level the system service can sustain.
Scalability testing evaluates the effects of adding additional hardware, virtual or physical, in order to distribute work among system components.
The following evaluation considerations have been taken into account for LoadRunner and NeoLoad:
- Record and playback
- Recognizing dynamic web components known as ‘correlation’
- Data functions – the ability to use data from a text file to populate web forms resulting in realistic test data
- Randomising – picking random datasets in rows, check boxes and radio buttons
- Putting it all together and orchestrating the load simulation and analysing the system response
LoadRunner is written in C++, and NeoLoad is written in Java. LoadRunner scripts after recording can be manipulated and debugged using C. NeoLoad is a scriptless tool – instead, it involves dragging and dropping functions into a script. Also IF statements and loops can be dragged and dropped. Even exception handlers are controlled the same way within NeoLoad. There are some good and some cumbersome qualities attached to both – the NeoLoad feature is less powerful than the LoadRunner vugen feature, albeit LoadRunner takes more hours to script record, parameterise and debug.
Recording scenarios
If you’ve ever used a proxy debugger like fiddler, you’ll see what’s happening behind the scenes when surfing the internet. There are send requests and responses from your computer’s browser to the website’s web server. Recording load testing scenarios works just like a proxy: when the record button is pressed, it actives a proxy that acts like a ‘man in the middle’ capturing http requests and responses from the system’s web servers.
LoadRunner architecture
LoadRunner, which has a trial version, consists of three components: the virtual user generator, also known as vugen (script recorder and editor), the controller where all the vugen scripts are controlled into a scenario, and the Analysis, the results component. These enable the user to keep scripting in vugen while a scenario is running in the controller component, which usually happens because the load tester is hired at the end of the System Development Life Cycle (SDLC) and is under a huge amount of pressure.
NeoLoad architecture
NeoLoad consists of one component encompassing all-in-one software, which means that if a load scenario is running you cannot create and edit scripts.
Recording web with LoadRunner
Nowadays Chrome and Firefox work perfectly as recording browsers in LoadRunner’s vugen. Once upon a time, just Internet Explorer worked. Other browsers made vugen crash when the recording button was pressed, so LoadRunner consultants were limited to the browser people use to download other browsers.
If you’d like to get the time it takes for a user to login to the system under test, press a simple ‘Start Transaction’ button during recording – you can name the transaction appropriately, log the user in, then end the transaction when the logged in page has downloaded.
The transactions look like this for a simple login script:
Recording web with NeoLoad
NeoLoad records scripts in a ‘tree-view’ like structure. A wizard pops up after recording to automatically correlate the recorded script for you. Start and end transactions are not like those in LoadRunner – they are called ‘containers’ in NeoLoad and there is no end to the container as the ‘end’ of the container is actually the beginning of the new one, like this:
Correlating
Correlating is extracting dynamic values and placing each dynamic value into a variable from GET requests and substituting it into the POST response, usually in the request after the GET. This is the most time-consuming task in load testing – just ask any performance test analyst.
Correlating with LoadRunner
Design Studio is the tool LoadRunner’s vugen uses to try to correlate dynamic values in a newly recorded script.
‘No correlation parameters found’: I glanced at the recorded script below and found many dynamic parameters.
So why didn't the automatic correlation utility work for LoadRunner? This is a good question considering some of the other protocols that LoadRunner uses correlate really well. Like me, many of my colleagues have become used to manual correlation when using LoadRunner for http/web.
In fairness to LoadRunner, it can find one or two dynamic parameters, depending on the size of the recorded script. It looks like this:
It works (sometimes).
Manually correlating with LoadRunner involves placing this statement to grab a parameter from every dynamic GET request, which you’ll have to then place before a POST request if required.
Correlating with NeoLoad
Correlating is definitely where NeoLoad has an edge over LoadRunner. NeoLoad takes hours of painstaking manual correlation work out of the equation, and that saves bucket loads of cash in consultant fees and maintains deadlines. Any IT manager will know how much a LoadRunner contractor costs daily. If that figure is going to be potentially halved (or otherwise cut down by the speed NeoLoad can do the job), then maybe the next word that enters the manager’s head is ‘bonus’ or even ‘gratitude’, for finishing the project on time and within allocated budget parameters. Gratitude goes a long way.
There is some manual correlation in NeoLoad that involves a not-so-very-cumbersome two clicks of a mouse.
NeoLoad is so intuitive that if the same extracted value exists in subsequent requests within the scenario, this pops up (music to my ears):
And if you’d like to replace all, then:
The music keeps on playing – click, click, click ... and done.
Playing back debugging/recorded script with LoadRunner
Debugging is instantly more informative in LoadRunner than in NeoLoad as you can see the script executing and outputting to the log (circled below). It’s easy to watch variables and check for errors in real-time script compilation and execution.
Playing back/debugging recorded script with NeoLoad
One quick and easy way to check and debug a script in NeoLoad is to use the virtual user checking option by right clicking on your script. This runs the script and shows errors in requests that need to be addressed.
In NeoLoad, a more involved debugging analysis (after checking using one method) is to use JavaScript. Do this by dragging the JavaScript function into a script and coding the appropriate write to a text file. All variables can then be sent to a text file and used as a watch list.
NeoLoad will also check the JavaScript syntax for you.
Parameterising
Parameterising is adding a text or csv file to a load testing script. For example, when a script is recorded, there is a hard-coded user id and password in the script. Creating a file of five user ids and passwords and thus adding to the script while replacing the hard coded values is parameterising. You can control this, for example, using the five user names – the script can randomly pick the user id and password or it can pick them one line at a time incrementally.
Parameterising with LoadRunner
I’ve always found parameterising with LoadRunner straightforward, as long as you’re aware that creating a new parameter creates a new .dat file as well. It’s easy to have a user name and password in the same file (like above) instead of in two separate files. You can also edit the text file in Notepad, which makes pasting large amounts of data relatively easy.
Variables in LoadRunner
Where variables need to be declared as well as the variable type, like char or int etc., and values need to be added to them during the script execution, then do the following:
Parameterising with NeoLoad
Adding datasets to NeoLoad isn’t too dissimilar to LoadRunner. Also you can import spreadsheets into the variable manager or just create datasets manually.
Variables in NeoLoad
NeoLoad has an edge over LoadRunner, as none of the above coding is necessary. Variable types are added using the variable manager as in the screenshot below:
NeoLoad has an added SQL variables feature that queries the backend database for test data.
Randomising user input/actions
A very important aspect of a realistic load test is to have as many virtual users doing as many different things as possible to the system under test. The more random actions the better, such as if there are 10 links for two vusers to click, then there should be a one-in-forty-five chance that the two users click the same link (if you don’t know what I mean, check out permutations on Google).
Randomising with LoadRunner
Picking a random row from a result set you’ve just recorded with LoadRunner involves doing something like the code below. First declare the variables, then using the web_reg_save_param_ex() built in LoadRunner function, grab the number of rows, place the number of rows into a random number function between 1 and 10 (if there are 10 rows), and finally get the script to click on any of the rows – this should be different for each iteration in your script, depending on the test you’d like to conduct.
Randomising with NeoLoad
Randomising with NeoLoad is very easy, but to randomise you have to record every request and place them in a container. To give you an example, imagine you are filling out an online form to register for an online shopping site (see the screen below). Just before you submit the form you have the option of subscribing to the store’s newsletter, and also entering into a draw if a radio button is checked. Now that’s four combinations:
- 1.Submit form
- 2.Submit form + tick newsletter
- 3.Submit form + tick prize draw
- 4.Submit form + tick newsletter + tick prize draw
In NeoLoad you have to record this randomising scenario four times and place each into a container. It’s then possible to control the randomise settings for a realistic and valid load test.
The fun part – performance testing
After all the hard work is done – the correlation, the parameterisation and getting admin access to every single pre-production server in the system – it’s time for the real fun. Every performance tester loves this part because it’s the breaking stuff part: Lego for grown-ups.
Other forms of load testing include what hackers do – distributed denial of service (DDOS) attacks. Well, that’s more like a stress test.
IP spoofing can be implemented with both LoadRunner and NeoLoad – this means that an even more realistic performance test can take place. You don’t want a router in the network to reroute the web requests to a honeypot if there’s a rule in place that redirects heavy traffic away from the system under test.
Load Testing using the LoadRunner controller
All servers in the system under test can of course be added into the load scenario, so that real-time monitoring of server resources can be directly correlated to the number of vusers being ramped up, to a peak hour load test. So for the layman, let’s see what maxes out first if 1,000 users login to your website all at once, or all in the space of five minutes – things will start to break depending on a lot of factors, too many to discuss in this article.
Server statistic graphs can be added to the load scenario, and all the gathered stats as well as transaction time taken can be collated in LoadRunner’s Analysis component. A performance Test Summary Report (TSR) uses a template that can be modified to your liking and then presented to your IT Test Manager. Make sure you let him or her know that you typed the report from scratch – work is all about perception after all.
The client controller component of LoadRunner has all the bells and whistles needed for a realistic load test. There’s only one drawback: there’s a limit on the number of performance counters that can be added to a scenario. If there is a requirement for more performance counters you have to install and configure monitoring software called SiteScope. All servers in the system have to then be added to SiteScope after SiteScope and the SiteScope license server are installed.
Load testing using NeoLoad
The ‘runtime’ component of NeoLoad and the LoadRunner controller are on a par in terms of the capabilities they have for you to add the appropriate graphs.
LoadRunner vs NeoLoad – the cost
Here’s what every IT manager wants to know: Which tool is cheaper? Is it value for money? Now this is a tricky one: How much does LoadRunner cost?
Performance testers have always known that when it comes to price, LoadRunner is not a BMW or Mercedes – it’s a Ferrari. I wasn’t very successful in my endeavours to obtain a quote for LoadRunner. I contacted HP in London, and they referred me one of their local partners here in Australia. My enquiry was pretty simple: I wanted a quote for a 5,000 user Web vusers license, but HP’s partners said via email that they needed more information.
Using Google to ascertain the cost of a LoadRunner license had mixed results. I came across an old article mentioning that a 5,000 vuser perpetual web license with support for three years was about a million dollars. That article was written in 2005.
When I contacted Neotys in France, they were a little more helpful with a licensing quote for NeoLand, and there was a lot of room for negotiation as well, including email support, monitoring for Oracle and a 5,000 vuser license that never expires.
Based on my very ball-park figures, there’s an approximate cost correlation between NeoLoad and LoadRunner. Imagine that LoadRunner costs $100:
LoadRunner: $100
NeoLoad: $10
NeoLoad is about 10% of the cost of LoadRunner, and Neoload includes technical support as well as perpetual licensing.
There are also several product packages associated with both HP and Neotys that can be negotiated as per your allocated budget, so keep that in mind. All you have to do is pick up the phone and talk to them.
So there you go – my take on the top two best performance tools on the planet. Taking into account industry experience, time frames scripting, hours spent completing the job (usually after working hours) and downright value for money, the winner has to go to NeoLoad. It just works.
If your company has spent its money on LoadRunner and highly paid consultants developing test scenarios for your application, then the money has not gone to waste. If your company wants a cheaper product that does the same job as LoadRunner, then evaluate NeoLoad and ask an experienced performance tester’s opinion on the tool. The downside to this option is finding experienced NeoLoad consultants.
LoadRunner is the preferred tool for large organisations, such as banks and financial institutions. Right now, NeoLoad is the preferred performance testing tool for universities and smaller organisations.
Some final questions
My employer has been using LoadRunner for performance testing for many years. How expensive would it be to hire people to convert these scripts to NeoLoad?
Weighted Rankings 1 – 5 (5 is excellent)
Can LoadRunner scripts be migrated to NeoLoad?
Yes, and vice versa. It doesn’t take as much time or resources as you think. Companies don’t need to hire specialists to re-code LoadRunner scripts into NeoLoad scenarios.
Disclosure: I am a real user, and this review is based on my own experience and opinions.
Hi Good article, can u suggest how to migrate neoload script to vugen one..