The Problem with More Data

Intuitively we think that we’ll be smarter and understand more with more data.  We’re awash in data and generating more every minute.  I love data.  The plural of anecdote is not data. There was a sign in the building where I did my Ph.D. work which said “In God we trust. All others need data.”

The challenge is interpreting data, understanding signal from noise.  Statistically, the bigger the pile of data you have, the more likely you’ll mistake noise for signal.  There are enough bits of data in the big pile that you can selectively pull some and create a story.  (Our species is tremendously good at creating complex stories from little threads of data.)  And others will generate a completely different story by extracting different bits from the same pile.  This is the modern version of the blind men touching the elephant story. 

Remember this when you hear anyone say, “The data clearly shows…” or “No one can disagree with this.”  An attribute of boldness and gentleness (strength under control) is treading cautiously and humbly. 

It’s also important to remember that information alone is not enough.  If all we needed was more information, every guy would be a multimillionaire with six-pack abs.