Having just put the phone down after talking to a customer about a product evaluation process, I thought I'd share a few thoughts on BI Product Evaluations or "Shootouts". Many companies in New Zealand have dedicated a fair swag of resource to this process over the years, so here are our considered thoughts.

There are two things about BI Tool Evaluations we feel should be highlighted:

1) They shouldn’t take too much time and their outcomes shouldn’t be a deal breaker for the project.

2) They must be heavily oriented toward the end user community, as they are the ones who will be actually using the tools, not the IT people.

It amazes us when IT shops spend several man years evaluating BI tools on behalf of and in isolation from their end user community. If the entire solution was going to be developed in this tool, and the solution was only allowed to use features that were provided by this one tool for ever more, then fair enough. In reality though, with an open BI architecture in place, there is more than one way to skin the proverbial cat. No end to end solution should ever be entirely developed within one product’s development realm.

Here’s Stellar’s editorial comment on tools evaluations: Don't waste too much time on tool evaluations. While the arcane mechanics of each tool are being dissected by technical people in white coats, the end user business community is demanding a solution to their problems "right now" before their window of business opportunity closes. So whatever you do, make sure these guys get involved early and buy into the tools chosen.

Tools evaluations should be done once the BI technical architecture is established and understood. We have seen several examples of tools being selected to perform roles that they are not really capable of doing. If they were evaluated against a mature BI architecture backdrop, where these roles are clearly documented, these mistakes would not have happened.

The BI Tool segment of the IT industry is churning out some great products. In its proper role, virtually any BI Tool today is rock solid and can provide 80 to 90 percent of the functionality required by the average kiwi company. There are some “horses for courses” situations to be sure, where one tool stands out above the others in a particular role, but as a general rule, they are all pretty good these days.

Give us a holler if you need help – remember, we are 100% product independent!