It seems that the level of adoption of business intelligence tools as a percentage of users is much lower than typically thought. The data warehouse institute (TDWI) published a commentary on the latest business intelligence survey.

This survey, published by the business application research center, reports that although your BI vendor might be telling you that baseline adoption is 20%, it turns out it’s a lot less.

In any given BI-using organization, notes Nigel Pendse, a principal with BARC and the primary architect of BI Survey, just over 8 percent of employees are actually using BI tools. Even in industries that have aggressively adopted BI tools (e.g., wholesale, banking, and retail), usage barely exceeds 11 percent.

I have to admit that the around 10 percent didn’t surprise me overall, but I was expecting that in those particular industries they would be much higher- i.e. even higher than the 20% figure that big BI has been telling us for years. Not having worked in retail or banking directly, I just assumed that they were all glued to their reports and cubes.

The article goes on to say that some indicators show that adoption is actually falling slightly. All this while the mega vendors have been focusing their message (and supposedly the capabilities of their tools) on “BI for the masses” and “self serve business intelligence”.

So what does this mean? Well, maybe it means that all those people who are frustrated by their big business intelligence, are disappointed at how few people are actually using the reports, and are wondering how the other companies do it should know that, well, on the whole the other companies don’t do it either.

On the other hand, maybe the “right” number of people using business intelligence is 5%. Maybe in some industries it’s three people at head office and thats it. In other cases it might be that the target should be every single employee using BI every single day. I think in the end the ever useful “it depends” is in play here.

What I do believe, however, is that big business intelligence is broken (I’m talking about the mega vendors). If it’s not completely broken, then it looks broken and everyone is talking about how it’s broken. Which is just another form of being broken.

I wonder what would happen if we looked at the adoption rate for Microsoft Excel for data analysis? Going out on a limb here, but I’m betting it’s more than 8%.

People vote with their behavior. Obviously, its not all doom and gloom, and it is possible to have a highly successful business intelligence deployment. In a former life, I had issues making sure we had enough funding for Cognos licenses because certain cubes were so popular that we had more demand than supply. But I’ve also looked at log files for entire data marts that were flat out empty. Hundreds of reports built and maintained without users.

What makes BI successful

Generally, the key difference wasn’t the tool, or the features, it was the content. Data warehouses and data marts that had been driven by the business and were focused on what the business wanted to see, in the way they wanted to see it were successful. Projects that were technology focused, run by IT and did not have the needs of the end users anywhere on the map would end up doing nothing but consuming cash and data center space.

It simply does not matter if the report takes 2 seconds or 2 hours to generate if it is not important to the business and there is no decision to make after seeing it.

Big business intelligence seems to think that BI for the masses is a tool problem- something in how their portal works, or how many rows of data per second their appliance can process. Sure, if the tools are hard to use or learn, it’s a factor, but I think more often than not business intelligence isn’t used because it’s not providing what is required.

There are a lot of very talented business intelligence professionals that work very hard to deliver the goods, and there are lots of very successful business intelligence projects of all sizes that create real value. But the whole industry needs to take a hard, honest look at where we’ve come from, where we are going, and be honest about who is using what and why.

Often, people use excel because last week they didn’t know exactly what they needed, and it is a tool that lets them build it themselves this week when the boss wants the answer and there is a decision to make. With all its flaws, it’s still the most adopted Business Intelligence tool in the world.

Should everything be done at lightning speed? Is it really never possible to know what analysis is needed and use a traditional business intelligence approach to creating it? Of course not.

But are there some types of analysis that need to be done very quickly? Yes. Should we as an industry ignore these, or “leave that to the users in excel”. I don’t think so.

I’m spending all my time building a self service data transformation tool because I believe there is room in the overall architecture for less formal and more creative, ad-hoc tools. Excel is the ultimate in informal (often far too informal). At the other end of the spectrum, a carefully run, tightly managed data warehouse project is the ultimate in formal (and often is overkill, hopelessly expensive and too slow for a rapidly changing environment).

I’m looking at the middle ground where we meet the users part way, and nurture capabilities to do rapid prototypes, one-time analysis and user driven data transformation. Not as a replacement for big BI- but as another tool in the tool box.