What's the best way to trial backup and recovery software?

Question

Hi community, 
We all know it's important to conduct a trial or do a proof of concept as part of the buying process. 
Do you have any advice for our community about the best way to conduct a trial or PoC? 
How would you conduct a trial effectively? Are there any mistakes which should be avoided?

MichaelWeimann · Accepted Answer

Was going to write a lengthy response but yours is spot on Gary.  I will only add that the front end and back end of every SMART goal is to be Specific and Timely.  Document what is important to test and what the criteria for passing are BEFORE you ever take delivery.  Then put an expected time for this POC to complete and what would be a successful test.

The only other thing I would add is if the vendor is not providing technical resources to drive and/or assist during the POC...don't waste your time.  But, if you expect the vendor to devote the resources, you can also expect the vendor to hold you to a purchasing decision when/if everything passes with flying colors.

Gary-Cook · Answer

I am not sure if this question comes from a vendor or customer so the response is somewhat generic.  If you are the technical customer or end user,  try to be involved in the process start to end.  If possible, be the hands on the keyboard.  No better way to understand the solution if you are going to be the user of it in the future.  If you are the vendor promoting ease of use, there is no better way to sell your product to the technical team.

I have managed a lot of data replication, protection, and archiving POCs.  Two requirements always stand out.  Success criteria and POC type.  As a vendor delivering the POC, you will fail 90% of the time without clearly defining these up front.  As a customer, you should have a clear idea about why you are investing your time in POC and what you expect to gain from it.

POCs should not be a training exercise.  They are a path to purchase a solution for a budgeted project.  If you are just kicking the tires, consider the free or self-paced options provided by many vendors.  These include on-line labs and downloadable virtual machines or trial software.  These cannot be considered a POC in my book.

Now the two key components for a successful POC.

#1 - Define as a Functional or Performance POC

Decide whether you are running a functional or performance-based POC.  If you are the vendor, make sure the customer is aware of the limitation of a functional POC in a limited resource environment. Don't allow a Functional POC to become a Performance POC.  Been there.  Done that.  It's never a success.

Functional testing is easier.  There is no requirement for measured performance so sizing the environment is a minor issue.  Just has to be "fast enough" to keep your attention. They usually cover base installation, backup target configuration, agent configuration, test backups and restores, reporting, alerting, etc.  Data sets are generally small.  It can be executed in a limited environment with virtual machines.  Sometimes the vendor can supply access to a remote lab environment such as the VMware vSAN lab.  Sometimes it can be delivered as a preconfigured VM downloaded from the vendor.

Performance testing is complicated.  Speeds and feeds matter.  You will not be able to backup your entire live environment so you have to build a test environment to mimic it as close as possible if you are looking for GB/sec measurements.  Success Criteria become golden in performance tests.  You will be following the recommended hardware configuration supplied by the vendor.

#2 - Success Criteria

Define clear success criteria and stay with the plan.  This will avoid scope creep where testing has no endpoint.

A test plan can be extremely difficult to create from scratch.  Take the time because it is key to a fair and complete test.  It will make you think about the purpose of the test.  Most vendors have boilerplate POC documents.  They are a good starting point but they almost always focus on the strength of the product.  If you are the customer performing comparison testing, blend them into a single document.

Some or all of the success criteria should meet the "must have" requirements of a published RFP if it exists.

Test criteria should not be too detailed, especially to favor a particular solution UNLESS that is a pass/fail test.

Define a start and end date based on the testing requirements.  Testing should be sequenced.  Test backup of app A, app B, os C..  Don't jump back and forth between Oracle and Sharepoint for example.  Complete one, deal with any issues, check the boxes, and move on.

DR, Performance, and SLA testing absolutely require detailed planning.  Too much to detail in this short response.  Imagine a POC where you are faced with "I need to recover my 50 TB Oracle server off-site with an RPO of 5 seconds and an RTO of 5 minutes".

In a large POC, you might have regularly scheduled meetings or conference calls for updates on the progress and to deal with issues.

Include a site survey covering security and the network configuration,  Prepare to deal with fixed IPs, firewalls, ports, Active Directory, etc.  Nothing like a backup solution to break a network and bring the testing to a standstill.  Make sure you have a clear understanding of the environment.  I once had a POC where they were migrating some AD domains that were part of the test infrastructure.  Unknown to me.  Needless to say, we faced constant failures.

Define the hardware and configuration requirements on a per server basis.  OS, partition sizes, network, etc.  This applies to the backup infrastructure servers and the servers that will be the source of the backup data.

Include all the key contacts with access information to servers.

Make sure you have ALL the required resources (human and compute) resources available on the start date.  For example, you might need help from an Oracle DBA or SME on day 2 to continue the installation.

Define a process to modify the plan.  I've seen cases where another department sees the shiny new object and wants to jump into testing their app after the plan was approved and tests begin.  Plan to deal with this exception in the testing procedure but not deviate from accomplishing the original success criteria.  It should be approved by management.

Define what is considered critical to the success of the test, what is a nice to have feature, and optionally, what doesn't matter at all.   Be specific.  Include application versions if it matters.  You might judge the test completion as pass / partial pass / no pass or a percentage of how it meets the criteria.  Don't use subjective rankings.  Add a column next to the test for comments for subjective comments.

If you are comparison testing two or more solutions, make sure you can test "apples to apples" across the POC candidates.  All vendors should be tested to the same standard.  It can be difficult to compare an appliance to an enterprise software solution.  The appliance will win the easy to install checkbox but might fail in the ease of expansion category because it requires a new, larger box.

Consider the future in a POC, not just how it functions today.  For example, you should think about the process to add additional capacity locally or bring on new sites/servers.

NOTE:  Content here subject to updates if I think of something new or helpful.

Fred Kovacs · Answer

I know this is a simple answer but research companies that offer this service and use their free software trial versions to see if you like them or not. Research is the answer.

Dominik-Oppitz · Answer

1 - Build up a dedicated environment for evaluation. In this, you can control and monitor all aspects (performance impact on primary storage, restore times, etc) very granularly without jeopardizing your production infrastructure. Hardware vendors are more than willing to help out as often a new software solution comes hand in hand with a new backup solution.
2 - A man (or woman) with a plan is a man (or woman) having success. Work out an agenda for the evaluation, starting at the business needs (SLO/SLAs, etc). Define the necessary processes with the vendor - this is a great test for how supportive they are and will be.
3 - Document the outcome person by person! Everyone looks at a vendor differently, so you need multiple-vector information as a foundation for your decision. BTW, this is a great tool to motivate your staff and to a vendor´s pricetag where you need it!
4 - Stick to the plan but be open to expanding. Never go back from the initially defined scenario. It was based on business needs, and these needs do not disappear - but boy do they come up during these evaluations. Keep them tracked and manage them accordingly. Not every input needs to be tested, but it needs to be ticked off and to be addressed.
5 - Whatever solution you look at: form follows function follows usability follows security
6 - Squeeze whatever you can learn out of these scenarios. You never know when you need it again.
7 - Play fair. Vendors invest a lot of their time in these PoCs. So, if they do not fit your need is to tell them. Give them the chance to bring up another solution or to withdraw. But again: Never go back from your agenda. Your business defined the needs.

it_user897210 · Answer

Hello

Resilience is the keyword of any Backup & Restore software. Whatever it’s named. I can see 3 major mistakes while people are driving analysis & test of their Backup & Restore solution.

1 – Definition: Before even starting, RTO/RPO have to be clearly defined, this will be very helpful to determine your B&R tools and architecture.

-        RTO/RPO, do I need all my data backup or only critical one and to what timeframe?

-        Size, how much active data are we speaking about? Should I keep all data indefinitely or should I put strong Data policy management (in my environment, the standard is 30 days and all above have to be justified either by Business or Legal requirement)

As well, if I put deduplication in place, what is the impact?

-        Availability, should I make my B&R infra high available? What is the outage I can consider/live without my B&R software

2 – Environment: What is the scope of your Backup & Storage software ? We you be able to use as 1 tool, or many spread over your coverage (datacenter, workstation…) and at what scale?

3 – Testing, what is the purpose of performing the 100GB test if I’m covering 100TB? The test must be driven in “prod likd” situation.

-        Sizing of all daily backup

-        Restore in production while the backup is running in parallel

o   This will confirm my RTO/RPO or will show what gap I do have to answer.

-        My B&R “admin” task

o   Do I use deduplication? If Yes, what is the capability of my hardware to dedup daily backups (10, 20, 100TB/24h ?)

o   Do I use Disaster Recovery capability?

o   Recycling of tapes (considering smart environment)

If to protect my data in the best manner, the time all those tasks are taking in my B&R resource must be known.

-        Restore my B&R software.

This is quick head-up on the hottest topic too much often forgotten when driving study on B&R software.

it_user871440 · Answer

From my experience following aspects should be considered to avoid potential problems:
1. Choose only “known vendors” in the market for POC especially if the data to be secured its worth
2. Check for a single product which can fulfill “all data management” requirements out of the box
3. Conduct a “real life” POC which includes all required scenarios (backup & restore)
4. Don’t forget about the performance of the Backup & Recovery solution especially the Restore Speed (RTO)
5. Ask Data Management Specialists early for advise e.g. Rules of thumb / Best Practices
6. Before deciding for a solution check for the total costs over a longer period (renewal, growth,..)
7.  Avoid vendor lock-in solutions (flexible components e.g. Server, Storage,…)

Raul Garcia · Answer

I agree with the previous recommendations that are exposed, which would add the following 4 aspects;

1) Involves all the supply of the complete solution: A proof of concept does not only involve the provider of the application. For example, you should also consider communications providers (carriers) in the event that your test is being done by supporting servers in geographically distant locations, or even in the same SITE, it may be involving other solutions such as virtualization and even the database of the main application that you want to support (considering the size of the logs or things of that nature)

2) Prepare your level of acceptance and rate the test: Go testing and qualifying (knowing and not necessarily learning) is very good, however what are we qualifying ?, As we know we are after the test comparing "pears" with "pears " Well for this I recommend that prior to the proof of concept have the SCRIP ready for the steps you want to test, the range over which will qualify the result, the findings and assumptions, as well as the ideal qualification on which you will observe that both distance themselves the solutions he tried. Even when you only try one. Let's imagine that you are qualifying 3 criteria with an 80, 95 and 90 as a suitable qualification and that the solution complied at 40, 55 and 50. Would you say that the proof of concept was successful? In my opinion, the test fulfilled half of the functionality or solution expected for each requirement, so the test could be considered as failed (we must find another solution)
 
3) Time required for the test (your time not the provider's): Another aspect to consider is the time in which the provider is willing to invest with you the use of your solution. Sometimes the best tests are those that simulate a natural period of your operation. It may be 24 hrs. as it could also be a full week (7 days) or even a full month with a change from one period to another.
In this case you should negotiate with the supplier the time that you require and of course if you do not want to invest adjust to what the provider proposes, but that is already part of the qualification of the possible results (your requirement vs the variable of times required to replicate a scenario "of the actual operation")

4) Test the Backup on the Recovery in the same POC: Por last, as we are talking about replicating information (application, a virtual server, etc.), always consider exposing in the scope of proof of concept both the backing of information and the return to normal operation from said backup. Otherwise your proof of concept would be incomplete.

JohannFLEURY · Answer

HelloWith all already been said, be sure to test all different technology this PoC will be used for and do not neglect end-user testing. They are the final step of a good PoC.
Do not rely on vendor performance story. They can be far from reality of your own environment so have already a baseline and set of performance tests to be sure it fits your need and know the limits. As example, do not buy a Dell EMC if you need IOs, this is not made for it but if you need archiving solution then it is becoming a good candidate.
And last, PoC is here so… test, test, test… and redo test, test, test so your teams will be confortable with it.

SrSysAdmin45860 · Answer

First thing is to create a backup and recovery scenario, which may include at least 5 times backing up the same data and trying to restore it. And also it is essential to do it in real data/servers for performance comparison.
Another thing is to compare it with the current/running software (backup times, deduplication and compression ratios, ease of use etc.)
One of the common mistakes is doing the PoC in an isolated network which will give results different than the real environment.

reviewer719127 · Answer

It depends on the company main business type, business model and focus.
The main concerns/pre-requisites to conduct a trial effectively as follows:-

1.       The backup software compatible with various kind of main Operating Systems, for example, Windows, Linux (SUSE, RHEL, CentOS), Unix (HP UX, AIX, Solaris) and etc.

2.       Make sure the Backup software able to support various kind of Databases.
Example: Microsoft SQL Server, DB2, MaxDB, MySQL, Sybase ASE, Sybase SQL Anywhere, SAP Hana, Oracle (brtools / RMAN) and etc.

3.       De-duplicate and encrypted function is a must for a backup software in transit and at rest.

4.       The backup landscape is scalable and able to grow according to business needs.

5.       Able to support snapshot backup for VM and Public Cloud Instance.

6.       Good backup throughput of the landscape with minimal bottleneck in network bandwidth.

7.       Backup Software able to meet complex backup strategy requirement. Like Long Retention and Offsite backup readiness.

8.       Lastly, Backup audit report function availability.

What's the best way to trial backup and recovery software?

16 Answers

Related categories

Related Q&As

Backup and Recovery experts