While I don’t feel an urge to argue the “should we use data or use our gut to make decisions” question nearly as much as I used to, I am noticing that there is the “data is good, data is great, more must be better” camp that is starting to occasionally make me think. The pendulum is swinging- where is it in it’s arc is anyone’s guess, but I’m thinking we’re past center line.

“Don’t argue against over reliance on data! You sell a data tool, for goodness sake!”

Now, I’m a huge fan of data. I’ve been known to exclaim that more data is better and to store it all because storage is cheap you never know if you’ll need it.

I really do believe that you should data profile early and often.

But, as we get older (not old, mind you, just a bit older than before is all), we start to wonder if perhaps we’re suffering from a type of “irrational exuberance” in terms of our data.

Obviously, since I’m in the business of data tools, and managing lots of it, and doing interesting things with it, data is important and key and I always want lots around.

But too much of anything is too much. And sometimes piles of bad data are worse than no data at all. And unquestioned beliefs are the first step to blind spots, and after that it’s out of touch time.

Being too sure of your methods is often accompanied by being overly harsh on alternatives. “Going with our gut” is not really making decisions with our lower intestine at all. Even with the upper intestinal tract lending some opinions, we all know that the brain is involved in all those “gut” calls. When we make leaps in judgement that might not be completely supported by a MDM, ODS, DW, Data profiling approved method there is data involved. It’s been through a very advanced neural net that is the product of millions of years of evolution.

Even if we don’t know exactly WHY we feel a certain course is right, it is often a fact that our brains have sorted out patterns and trends in the data we’ve seen that might defy our formal systems, but in reality what’s often called “gut feel” or “intuition” it is fact based, just a bit more mystical.

So where am I going with all this?

I think in the end, there are no absolutes. Data is a critical thing. Nurture your processes, infrastructure and tools for collecting, storing and analyzing data.

But never forget that at the end of the day, a decision needs to be made. Not making it is a decision in itself, and if you wait long enough, if you wait for the data to support you completely, it’s possible the only decision you’ll make is not to decide, waiting for more analysis. Sometimes you need to leap. Leap based on Data- do fast ad-hoc analysis, use your Data Lake, don’t wait for the massive IT Data warehouse project to get done – get your Gut and Your Brain and at least some data- but leap.