In a recent interview, Reid Hoffman, co-founder and executive chairman of LinkedIn speaks about the nature of data and probability.
“We live in a probabilistic universe, and we tend to think in determinist ways. If A is data-driven and I think I have that data, how certain am I that I have that data? What could I discover that might actually tell me that that data is formulated wrongly? When you dig into it, most of your arguments are actually probabilistic. They’re not certain, even when you have data. You’re really trying to get a sense of whether you have a reasonable bet on the probability.”
This gets at a point that’s often lost in the enthusiasm for Big Data: data is not a substitute for judgement. In fact, the more data you have, the more judgement you need to exercise in selecting, refining and ultimately using it to make a better decision.