29
Jun
08

The Data Deluge Makes the Scientific Method Obsolete

Chris Anderson has written a great article in Wired on the data deluge and how it poses new challenges to the companies. He writes that the petabyte age that we live in information is not a matter of simple three- and four-dimensional taxonomy and order but of dimensionally agnostic statistics. For companies, that have or gather loads and loads of data, the implications are about how can they quickly sift thro’ this massive volumes of data and the successful ones will be the ones who can track and measure this with unprecedented precision and scale. Take a look:

Speaking at the O’Reilly Emerging Technology Conference this past March, Peter Norvig, Google’s research director, offered an update to George Box’s maxim: “All models are wrong, and increasingly you can succeed without them.”

This is a world where massive amounts of data and applied mathematics replace every other tool that might be brought to bear. Out with every theory of human behavior, from linguistics to sociology. Forget taxonomy, ontology, and psychology. Who knows why people do what they do? The point is they do it, and we can track and measure it with unprecedented fidelity. With enough data, the numbers speak for themselves.

The big target here isn’t advertising, though. It’s science. The scientific method is built around testable hypotheses. These models, for the most part, are systems visualized in the minds of scientists. The models are then tested, and experiments confirm or falsify theoretical models of how the world works. This is the way science has worked for hundreds of years.

Scientists are trained to recognize that correlation is not causation, that no conclusions should be drawn simply on the basis of correlation between X and Y (it could just be a coincidence). Instead, you must understand the underlying mechanisms that connect the two. Once you have a model, you can connect the data sets with confidence. Data without a model is just noise.

But faced with massive data, this approach to science — hypothesize, model, test — is becoming obsolete. Consider physics: Newtonian models were crude approximations of the truth (wrong at the atomic level, but still useful). A hundred years ago, statistically based quantum mechanics offered a better picture — but quantum mechanics is yet another model, and as such it, too, is flawed, no doubt a caricature of a more complex underlying reality. The reason physics has drifted into theoretical speculation about n-dimensional grand unified models over the past few decades (the “beautiful story” phase of a discipline starved of data) is that we don’t know how to run the experiments that would falsify the hypotheses — the energies are too high, the accelerators too expensive, and so on.

Read more

Advertisements

0 Responses to “The Data Deluge Makes the Scientific Method Obsolete”



  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


At Cequity, we believe customer intelligence will be the biggest competitive advantage enterprises will have in the next decade or two. Successful enterprises of tomorrow will be the ones who can organize and leverage this information at speed to optimize their marketing performance, increase accountability, improve profit and deliver growth. Cequity insights will bring to you trends and insights in this area and it’s our way of sharing best practices so as to help you accelerate this culture and thinking in your organization.
June 2008
M T W T F S S
    Jul »
 1
2345678
9101112131415
16171819202122
23242526272829
30  

Blog Stats

  • 15,133 hits

%d bloggers like this: