Mining Unconscious Wisdom

Polling crowds to gain insight about the future has become commonplace. The collective guess beats the informed individual almost every time—witness the Hollywood futures market and others like it. The true power of the crowd, however, isn’t in individuals’ consciously espoused knowledge and opinions. The essence is actually buried somewhere deep inside your company’s database. Tools for slicing and dicing customer stats are better these days because of innovative applications of research methods such as regression analysis and randomization. In addition, the technologies for storing, accessing, and distributing customer data are becoming cheaper and easier to use. This convergence of improvements has allowed some forward-thinking companies to finally take full advantage of the huge stores of information at their disposal. They’re no longer letting their tapes collect dust; instead they are digging for dollars and “sense” in their databases. And they are finding compelling stories about customer segmentation and service—“unconscious wisdom” that the crowd itself may never have thought to share. The dating service eHarmony, for instance, doesn’t solicit your or others’ opinions about your ideal mate; it tells you whom you will like based on your responses to a 436-question survey. The questions are geared toward figuring out your personality—are you an unconventional thinker, for instance, or a people pleaser? Using research data on successful marriages, eHarmony then suggests potential matches—sometimes pairing personality types that might, at first blush, seem incompatible. Similarly, sites like Pandora and Rhapsody can make fairly accurate inferences about the music a customer will buy based on her historical purchase data and on a computerized parsing of song attributes: You’re a fan of Arcade Fire? Here are some artists whose songs have the same characteristics as those in Arcade Fire’s catalog—the use of orchestral arrangements in rock music, for instance. The travel site Farecast mines terabytes of data not only to tell end users whether the time is right to buy a ticket for that flight to San Francisco—based on historical data about how fares behave—but also to gauge the precision of that advice. The site assesses the data and then offers its recommendations with, say, 85% confidence if the historical record is strong and, say, 60% confidence if the record is weaker. A 2007 external audit concluded that Farecast’s overall rate of accuracy in predicting price trends was 75%. Asking the crowd whether it thinks the price will go up or down wouldn’t be nearly as efficient or effective. There’s no doubting the critical role that crowd power has played in the evolution of markets. Still, we’re just skimming the surface. By trolling for the unconscious wisdom in consumer data, companies are able not only to uncover useful patterns, segments, and influences, but also to peek into consumers’ psyches.

Mining Unconscious Wisdom

Summary.

Partner Center