I love the acronym FUD which stands for “Fear, uncertainty and doubt”. What I don’t love is the underhanded use of FUD to manipulate peoples behavior. Spreading FUD is not about creating something new, but destroying- destroying someones confidence in something, clouding the real issue, stopping a new or creative direction from being taken. FUD is often used to block reform and change because FUD can cause people to do nothing- and doing nothing is good for the incumbent.

In the data analysis realm, spreadsheet errors are often used to try to dissuade companies from letting their people “work with the data directly”. Software vendors of all sizes, but particularly the really big ones (those incumbants) spread FUD because if they can stop people from getting at the data themselves, it increases the chance of companies buying some more business intelligence suites.

The argument goes something like this:

Spreadsheets have been shown to be plagued with errors, many studies showing error rates above 90%. You need to reduce the risk that spreadsheets are creating in your organization by establishing formal, documented processes that are created an managed by professionals using sophisticated tools.

Then the usual nightmare scenarios are brought out, all involving rabid Auditors, Sarbane-Oxley, governance failures etc.

Now, don’t get me wrong, spreadsheet errors are a very real and serious problem, and there are all sorts of data applications that should never be done in Excel or other ad-hoc, user driven tools. Ever. Formal documented processes are critically important, and there are lots of places where you better be using the right tools and professionals.

I have seen the culture of the spreadsheet completely undermine initiatives that would have driven better data quality, data analysis and business processes. The spreadsheet certainly has its dark side.

But the problem is that FUD paints with a broad brush. People take it as “Spreadsheets with data in them? Bad news. Don’t do it. Individuals able to get at the data, and quickly transform it, analyze it? Who knows what they’ll do- shut them down!”

Sadly, from a data quality point of view, sometimes the spreadsheets have the BEST data quality- because people have fixed the issues they can’t fix in the transactional system due to constraints or IT department delays.

Encourage positive change with reasonable controls.

Intelligent, responsible people should be encouraged to use “informal” methods and tools to do data analysis.

These people will find things, learn things, and drive positive change (including change in those big formal professional systems).

They should do it with a reasonable understanding that doing things in an informal way, with spreadsheets or other tools does introduce errors, and should consider this when they recommend taking action based on the results.

Balance between two extremes

The totalitarian state: I don’t think there is an IT department in the world that is capable of stopping all unofficial data analysis. In fact, I would suggest that the moment such an IT department comes into existence, it would kill the host company, a harsh sort of self-regulation. People interested in data and thinking for themselves would just pack up and leave. So who would be left making the decisions and based on what?

The twisted web of spreadsheets: Companies that allow an anything goes, visual basic code, macros and manual cut and paste direct to the annual report environment are not going to be long for the world either. They populate the horror story pages on the spreadsheet risk websites.

The zone of win.

You want to be somewhere between insane spreadsheet addiction and strict formal big tool paralysis.

I submit that companies that balance risk while still encouraging their smart people to “play” with the data and do analysis in new and interesting ways with new tools are going to win.

Again, don’t let this process generate your profit and loss statement- understand where and what the informal discovery process is for- but do let it discover things. If it discovers something interesting you’ll have the chance to check for the errors. Make sure its part of the process to do so.

By letting the FUD get you down, you’ll never get that far and who knows what insights you might be giving up?

Of course, we believe you should go even further and give those intelligent, responsible people new tools that are less error prone than spreadsheets but still provide as much or even greater flexibility. That’s why we’re building Datamartist after all.

Openness, balance, and clear minded pragmatism will get you further than FUD every time.