The Perfect and The Good

by Ben Perreira

Those who seek solutions to complex problems first encounter the most significant problem: what types of information to use in their quest. 

Alan Greenspan addressed this on The Daily Show a couple days ago. He said that growing up as a Wall Street analyst, he believed that the “screwiness” of people would ultimately be neutralized because people act rationally in the long term. He figured that if they had enough raw information they could create a forecast based on the assumption rationality. He learned in 2008 that this assumption was very wrong.

People are pretty screwy indeed. We are almost always reasonable but seldom rational. (Evidence of this is that marketing works.)

Greenspan and his cohorts were not wrong is how they performed calculations in the same way that much of the research we read is not wrong is how it uses statistics. The problem lies in how the data that is used as input for these models is generated and categorized. It is the classic “garbage in, garbage out” (GIGO) scenario.

Imagine you know you may have a GIGO problem, but at least the data is reliable or precise. Pre-Copernican astronomers were precise too; they were just wrong on a very consistent basis. This is a conundrum in Western business culture because he love numbers. We are more likely to believe something we can put on a graph. In the era of big data this is even worse. With so many ways to slice data, there is always an answer that could make sense and is represented in numerical form.

We can avoid these problems by establishing some bona fides:

– Many, if not most, things are inherently unpredictable and unknowable.

– A hypothesis is the product of contextualized intuition.

– Numbers are not inherently the most reliable representations of ideas.

A solution to this problem is to calculate everything we can down to the most precise level possible. Then, if we cannot remove the vast majority of systematic bias (e.g. that which assumed geocentricity), we must allow ourselves to minimize or exclude that information from our calculations. We can allow intuition and forward-thinking a seat at the table.

Regression analysis and generic questioning did not deliver society the car or the Smartphone. We have the option to stay informed while giving ourselves some credit for moving knowledge forward.