BI Tool Evaluation Criteria Rating Matrix -- anyone have one they've used in making a tool selection?

You can get a comprehensive comparison from Panorama website, we have used this, it is very informative

www.panorama.com

First of all the Gartner - well structured in-depth analysis, forecasting

KLBaxter: Partly because we were focused mostly on the dashboarding side of the equation, and also because we already utilize a combination of SSRS and SAP Business Objects for canned and ad hoc reporting respectively, with no plans to replace those products in the near term. Had we been looking to find a single solution to cover most, or all our reporting and analysis needs, I'd likely have explored some additional vendors, especially because canned & distributed reporting is not one of Qlikview's strong suits. However, as I'm sure you and many others here have also come to realize, finding one vendor that is a leader in all things BI is a very lofty ambition. The phrase "Jack of all trades, master of none" comes to mind. ;)

Jhornber - was there a reason Cognos wasn't evaluated? All my initial scoping keeps leading me to QlikView but I'm determined to give the others a fair trial!

Ultimately Qlikview was selected, as it seemed to best meet our particular needs.

Thanks for sharing Jhornber! Which BI tool did you end up selecting?

I've posted my BI tools selection criteria here:
www.itcentralstation.com

Let me know if you’d like to know anything else!

Jhornber - Can you share your criteria here with the community? I'm sure everyone will benefit from reading it.

Yes, I would be interested in seeing your criteria.

Joanne.morgan@canforpulp.com

Jhornber, it will be nice to share it.

Please share your thoughts...

I just completed an evaluation of (and ultimately a final selection from) several vendors and identified about a dozen high level evaluation criteria (using Gartner capabilities as a starting point, and adding those specific to our focus). We were primarily concerned with an efficient, competent dashboarding tool with a secondary focus on ad hoc analysis. If you're interested in seeing my criteria, let me know. Happy to share if it helps.

We have noted that one single BI does not meet the requirements of large organisation. I would suggest some changes in the template shared.

Brevity is rewarded - the operative word is "report" and not analysis visual or otherwise

Hi, before responding to this question let me be open and declare a bias towards certain types of BI tools because I work for Tibco Spotfire, a data discovery and visualisation vendor.

Leaving product aside therefore lets establish some principals before starting down the road of a matrix - firstly, what needs to be in that matrix?
What do the business (and technical) users want to be able to do? There is a wide variety of capabilities across the vendors and building a matrix on what every vendor does is going to be a time consuming exercise to both build it and then administer it, especially if it is not all relevant to what the business needs.

For instance, if the key requirement is to burst out a pixel perfect report every Monday morning to 300 recipients then tools like Qlikview, Tableau, and Spotfire are unlikely to be your best bet.

If on the other hand you want to empower users to explore data themselves (in order to minimise the IT overhead), mash-up data from disparate systems, follow a train of thought to identify new threats and opportunities, and collaborate across the enterprise with colleagues on the insights you are discovering, and quickly prototype new dashboards - then the new generation of data discovery tools is likely to be more useful. That said, according to Gartner, this is where all of the traditional BI vendors are trying to take their products. They just haven't got there yet. So, timescales are important.

The fact is that extensive matrices are a tick box exercise, perhaps okay for a first cut, but largely a waste of time. All vendors (or their marketing departments) will try to put as many ticks in the product boxes as possible. Just because a vendor says their product can do something does not mean they do it very well and there are only two ways to find out the truth of it - buy it and build it, or talk to someone who has.
If you are clear about the capabilities you need (and I emphasise 'capability' not product 'functionality') your short list comes together pretty quickly. At that point, you start on the detail, and you will find your matrix more focused and easier to manage and you will be able to get to a conclusion a lot more quickly.

If you do pull down a generic matrix, use it as a guideline to getting to something specific that will meet the needs of your business, not someone else's a few years ago.

Thanks for all the helpful comments. Please send me the full matrix from example 1. Thanks, Steve

Starting with customer acceptance criteria before going to a Evaluation scorecard would be my advise. Ask them what they want and go have a look on how they work. If you choose the "best" tool but the rampup for learning how to use it is too big, customer want simply reject. Usual they know what they want, you just Need to translate it into a Feature & Performance set.

Disclaimer: I work for Microsoft

Quite an interesting discussion and eye opener.

Very useful, thanks.

Very interesting discussion! We've added several examples of BI matrices submitted by the community. See them here: www.itcentralstation.com

Email matrix@itcentralstation to get your complete copy.

I would recommend Gartner's BI Magic Quadrant see www.gartner.com . Hope this helps.

The problem with tool selection criteria is the way they are applied and the lack of any environment-specific filters. Check out Gartner and then go with what they see as the future.
I personally advocate Cognos for technically competent organizations. Government agencies do not fit this criteria.

Almost without exception, organizations(especially government orgs) do not have the expertise to implement BI tools properly. Microsoft had a terrible false start with Performance Point Server a few years back. They actually had people lined up to buy their BI tools and had to back out of the market until they got their tools matured. BI is not trivial.

To wring performance from BI tools requires a few people with very solid data modeling abilities and exceptional data/db skills. These people are hard to find and hard to keep.
Another thing that causes havoc when buying BI tools is any organization's structure. If the org has an ETL group, are they going to do the BI ETL or is the BI group going to do that also? Will your organization let the BI group build report-specific databases? Who will maintain the reporting databases; The enterprise or the BI group or both?

BI was crippled at a large aerospace company because the organization considered all databases to be equal. This meant that an enterprise warehouse counted one and the tiny, by comparison, BI report-specific cubes and stars were also counted as one each. The company bought Cognos without the ETL/DB building tool "Decision Stream/Data Manager" and suffered because of it. Finally after many years of failure or slow progress, it was acknowledged that BI requires its own databases to gather and stage the data. You cannot ask a meta-data modeling tool such as "Framework Manager" to do everything and you cannot ignore ETL. A BI environment will have dozens or hundreds of stars and cubes for specific purposes and report requirements.

I work at a government agency right now and would rate them a 1 on a scale of 1 to 10 for technical ability. There are sharp, up-to-date people but they are the exception. There are maybe 5 out of 100 with current skill sets.

I have one I used, but frankly I found that the "can you do this" feature-specific type matrix doesn't get to the real crux of why one tool is better than another.

A good deal of tool selection has to be based on what will fit with your company: are you aggregating data from multiple sources? If your database organized such that building views will be fairly easy? Are you migrating an existing reporting system for which you already have all the SQL and you don't want to re-invent the wheel? Do you have the resources to spend 6 months getting an "end-to-end" tool implemented? (And there are some good ones out there).

What about in-memory technology and performance? Caching? How does the cache get refreshed? Can I use my native SQL database, or do I have to build a star schema?

I find feature lists don't uncover these kind of implementation/design criteria.

I don't have access to an evaluation matrix. I worked for a very large company that had many BI applications at my disposal. There are a kit of considerations to be made. The most important considerations surround cost and functionality. After the budget is identified and the right applications are identified the next steps would be to gauge the support available from the vendor and then identify internal resources available to help in that effort. Some application licenses are more expensive but they also provide great support.

I would suggest checking in with Cindy Howson at the BI Scorecard. She typically has the tool area covered nicely. She could have just the info you are looking for.

Yes. I can help out. You can send me a message.

When it comes to BI, unless the business sees the need, it will forever say no!

When it does come around to saying yes, to make sure the win is certain, and the picture is rosy clear, I suggest the following approach:

Clearly define your requirements - (every step of the way):

In a rather common sense approach, satisfy the following questions/aspects:

1. What's my target?
How can the business benefit? Is it just for a specific unit/department, or the entire business? Can I take a modular approach - add on over time?

2. Functionality:
Who are my target audience - endusers, SMEs, hybrid (mix of both), executive management?
What kind of training will they need?
How many users - now and in the future? - plan for growth
Is the tool for internale use only, or both internal and external? Will it be accessed outside the firewall?
Do executives or salespeople need mobile BI capabilities?
Will dashboards, data visualization tools, etc. be implemented as well?
How important is real-time BI for operational workers, like customer service staff or tech support?
What kinds of ad hoc queries must the tool be able to handle?
Browser-based? Specific Application?

3. Licensing:
What pricing/licensing options are available?
How are support and maintenance contracts set up?
How many users will be querying the system - concurrent users, named users?
What kind of licensing scheme?

4. Off the shelf vs. customization:
What can the product do?
How does it meet our requirements?

5. Data makeup and placement:
Will you need to build a new data warehouse?
Heterogeneous environment... or separate data marts?
Centralized data, or all over the place?
Can you identify you data sources?
What is the technical environment that this BI software will need to integrate into (source systems, databases, preferred hardware platforms, etc.)?
How will your BI software integrate into our current environment?

6. Data volume and growth:
What are my data volumes today? What will they be in the near and far future?
What are your software update cycles like? How easy is it to add on new functions/features?

7. Training:
How will BI users and administrators be trained?
All-inclusive training class?
Or will you train only administrators and have the administrators teach the others?
8. Testing:
Do you want to test the product in your environment? Can we get a proof of concept?
Do you want a Proof of concept -- in your environment with your data? Or use Vendor's test environment?

9. Time frame for implementation?
Corporate deadline?
How long will implementation take? Will you need consultants or vendor services? In-house team?
Who's going to be on my BI team?
Who's driving the process (evaluation & purchase) - IT? Or by the business? Or both?

If you've made it this far, you've answered the biggest question - Do we need BI?

Now you can finish with the last 2 questions:

1. Should we write a request for proposal (RFP)?

2. What's my budget?

3. Who can we talk to to compare LOE (level of effort)? Find someone in the same field/industry.

A review of the gartner magic quadrant is a good starting point. There also might be something for use on the TDWI website

Gartner probably does the best job of all the rating organizations, but Klas also has a pretty good reputation for healthcare products.

This is a pretty comprehensive overview with a Magic Quadrant chart.

www.gartner.com

Good luck.

Gartner BI-MQ-2014 can help

The best and most known evaluation Matrix is Gartner´s Matrix. It also explains all the criteria and the strengths of the most important BI tools in the market

www.gartner.com

Yes - but it addresses only data visualization and analysis vendors