Keep Up with Your Quants

“I don’t know why we didn’t get the mortgages off our books,” a senior quantitative analyst at a large U.S. bank told me a few years ago. “I had a model strongly indicating that a lot of them wouldn’t be repaid, and I sent it to the head of our mortgage business.” When I asked the leader of the mortgage business why he’d ignored the advice, he said, “If the analyst showed me a model, it wasn’t in terms I could make sense of. I didn’t even know his group was working on repayment probabilities.” The bank ended up losing billions in bad loans. We live in an era of big data. Whether you work in financial services, consumer goods, travel and transportation, or industrial products, analytics are becoming a competitive necessity for your organization. But as the banking example shows, having big data—and even people who can manipulate it successfully—is not enough. Companies need general managers who can partner effectively with “quants” to ensure that their work yields better strategic and tactical decisions. The Explainer: Big Data & Analytics For people fluent in analytics—such as Gary Loveman of Caesars Entertainment (with a PhD from MIT), Jeff Bezos of Amazon (an electrical engineering and computer science major from Princeton), or Sergey Brin and Larry Page of Google (computer science PhD dropouts from Stanford)—there’s no problem. But if you’re a typical executive, your math and statistics background probably amounts to a college class or two. You might be adept at using spreadsheets and know your way around a bar graph or a pie chart, but when it comes to analytics, you often feel quantitatively challenged. So what does the shift toward data-driven decision making mean for you? How do you avoid the fate of the loss-making mortgage bank head and instead lead your company into the analytical revolution, or at least become a good foot soldier in it? This article—a primer for non-quants—is based on extensive interviews with executives, including some with whom I’ve worked as a teacher or a consultant. You, the Consumer Start by thinking of yourself as a consumer of analytics. The producers are the quants whose analyses and models you’ll integrate with your business experience and intuition as you make decisions. Producers are, of course, good at gathering the available data and making predictions about the future. But most lack sufficient knowledge to identify hypotheses and relevant variables and to know when the ground beneath an organization is shifting. Your job as a data consumer—to generate hypotheses and determine whether results and recommendations make sense in a changing business environment—is therefore critically important. That means accepting a few key responsibilities. Some require only changes in attitude and perspective; others demand a bit of study. Learn a little about analytics. If you remember the content of your college-level statistics course, you may be fine. If not, bone up on the basics of regression analysis, statistical inference, and experimental design. You need to understand the process for making analytical decisions, including when you should step in as a consumer, and you must recognize that every analytical model is built on assumptions that producers ought to explain and defend. (See the sidebar “Analytics-Based Decision Making—in Six Key Steps.”) As the famous statistician George Box noted, “All models are wrong, but some are useful.” In other words, models intentionally simplify our complex world. Analytics-Based Decision Making—in Six Key Steps When using big data to make big decisions, non-quants should focus on the first and the last steps of the process. The numbers people typically handle the details in the middle, but wise non-quants ask lots of questions along the way. 1. Recognize the problem or question. Frame the decision or business problem, and identify possible alternatives to the framing. 2. Review previous findings. Identify people who have tried to solve this problem or similar ones—and the approaches they used. 3. Model the solution and select the variables. Formulate a detailed hypothesis about how particular variables affect the outcome. 4. Collect the data. Gather primary and secondary data on the hypothesized variables. 5. Analyze the data. Run a statistical model, assess its appropriateness for the data, and repeat the process until a good fit is found. 6. Present and act on the results. Use the data to tell a story to decision makers and stakeholders so that they will take action. To become more data literate, enroll in an executive education program in statistics, take an online course, or learn from the quants in your organization by working closely with them on one or more projects. Jennifer Joy, the vice president of clinical operations at Cigna, took the third approach. Joy has a nursing degree and an MBA, but she wasn’t entirely comfortable with her analytical skills. She knew, however, that the voluminous reports she received about her call center operations weren’t telling her whether the coaching calls made to patients were actually helping to manage their diseases and to keep them out of the hospital. So Joy reached out to Cigna’s analytics group, in particular to the experts on experimental design—the only analytical approach that can potentially demonstrate cause and effect. She learned, for example, that she could conduct pilot studies to discover which segments of her targeted population benefit the most (and which the least) from her call center’s services. Specifically, she uses analytics to “prematch” pairs of patients and then to randomly assign one member of the pair to receive those services, while the other gets an alternative such as a mail-order or an online-support intervention. Each pilot lasts just a couple of months, and multiple studies are run simultaneously—so Joy now gets information about the effectiveness of her programs on a rolling basis. In the end, Joy and her quant partners learned that the coaching worked for people with certain diseases but not for other patients, and some call center staff members were redeployed as a result. Now her group regularly conducts 20 to 30 such tests a year to find out what really makes a difference for patients. She may not understand all the methodological details, but as Michael Cousins, the vice president of U.S. research and analytics at Cigna, attests, she’s learned to be “very analytically oriented.” Align yourself with the right kind of quant. Karl Kempf, a leader in Intel’s decision-engineering group, is known at the company as the “überquant” or “chief mathematician.” He often says that effective quantitative decisions “are not about the math; they’re about the relationships.” What he means is that quants and the consumers of their data get much better results if they form deep, trusting ties that allow them to exchange information and ideas freely. Of course, highly analytical people are not always known for their social skills, so this can be hard work. As one wag jokingly advised, “Look for the quants who stare at your shoes, instead of their own, when you engage them in conversation.” But it’s possible to find people who communicate well and have a passion for solving business—rather than mathematical—problems and, after you’ve established a relationship, to encourage frank dialogue and data-driven dissent between the two of you. Katy Knox, at Bank of America, has learned how to align with data producers. As the head of retail strategy and distribution for the bank’s consumer division, she oversees 5,400-plus branches serving more than 50 million consumers and small businesses. For several years she’s been pushing her direct reports to use analytics to make better decisions—for example, about which branches to open or close, how to reduce customer wait times, what incentives lead to multichannel interactions, and why some salespeople are more productive than others. Bank of America has hundreds of quants, but most of them were pooled in a group that managers could not easily access. Knox insisted on having her own analytics team, and she established a strong working relationship with its members through frequent meetings and project-reporting sessions. She worked especially closely with two team leaders, Justin Addis and Michael Hyzy, who have backgrounds in retail banking and Six Sigma, so they’re able to understand her unit’s business problems and communicate them to the hard-core quants they manage. After Knox set the precedent, Bank of America created a matrix structure for its analysts in the consumer bank, and most now report to both a business line and a centralized analytical group. Focus on the beginning and the end. Framing a problem—identifying it and understanding how others might have solved it in the past—is the most important stage of the analytical process for a consumer of big data. It’s where your business experience and intuition matter most. After all, a hypothesis is simply a hunch about how the world works. The difference with analytical thinking, of course, is that you use rigorous methods to test the hypothesis. For example, executives at the two corporate parent organizations of Transitions Optical believed that the photochromic lens company might not be investing in marketing at optimal levels, but no empirical data confirmed or refuted that idea. Grady Lenski, who headed the marketing division at the time, decided to hire analytics consultants to measure the effectiveness of different sales campaigns—a constructive framing that expanded on the simple binary question of whether or not costs were too high. If you’re a non-quant, you should also focus on the final step in the process—presenting and communicating results to other executives—because it’s one that many quants discount or overlook and that you’ll probably have to take on yourself at some point. If analytics is largely about “telling a story with data,” what type of story would you favor? What kind of language and tone would you use? Should the story be told in narrative or visual terms? What types of graphics do you like? No matter how sophisticated their analyses, quants should be encouraged to explain their results in a straightforward way so that everyone can understand—or you should do it for them. A statistical methods story (“first we ran a chi-square test, and then we converted the categorical data to ordinal, next we ran a logistic regression, and then we lagged the economic data by a year”) is rarely acceptable. Many businesspeople settle on an ROI story: How will the new decision-making model increase conversions, revenue, or profitability? For example, a Merck executive with responsibility for a global business unit has worked closely with the pharmaceutical company’s commercial analytics group for many years to answer a variety of questions, including what the ROIs of direct-to-consumer promotions are. Before an ROI analysis, he and the group discuss what actions they will take when they find out whether promotions are highly, marginally, or not successful—to make clear that the effort isn’t merely an academic exercise. After the analysis, the executive sits the analysts down at a table with his management team to present and debate the results. Ask lots of questions along the way. Former U.S. Treasury Secretary Larry Summers, who once served as an adviser to a quantitative hedge fund, told me that his primary responsibility in that job was to “look over shoulders”—that is, to ask the smart quants in the firm equally smart questions about their models and assumptions. Many of them hadn’t been pressed like that before; they needed an intelligent consumer of data to help them think through and improve their work. No matter how much you trust your quants, don’t stop asking them tough questions. Here are a few that almost always lead to more-rigorous, defensible analyses. (If you don’t understand a reply, ask for one that uses simpler language.) What was the source of your data? How well do the sample data represent the population? Does your data distribution include outliers? How did they affect the results? What assumptions are behind your analysis? Might certain conditions render your assumptions and your model invalid? Why did you decide on that particular analytical approach? What alternatives did you consider? How likely is it that the independent variables are actually causing the changes in the dependent variable? Might other analyses establish causality more clearly? Frank Friedman, the chief financial officer and managing partner for finance and administration of Deloitte’s U.S. business, is an inveterate questioner. He has assembled a group of data scientists and quantitative analysts to help him with several initiatives, including optimizing the pricing of services, developing models that predict employee performance, and identifying factors that drive receivables. “People who work with me know I question a lot—everything—always,” Friedman says. “After the questioning, they know they will have to go back and redo some of their analyses.” He also believes it’s vital to admit when you don’t understand something: “I know I am not the smartest person in the room in my meetings with these people. I’m always pushing for greater clarity [because] if I can’t articulate it, I can’t defend it to others.” Establish a culture of inquiry, not advocacy. We all know how easily “figures lie and liars figure.” Analytics consumers should never pressure their producers with comments like “See if you can find some evidence in the data to support my idea.” Instead, your explicit goal should be to find the truth. As the head of Merck’s commercial analytics group says, “Our management team wants us to be like Switzerland. We work only for the shareholders.” In fact, some senior executives push their analysts to play devil’s advocate. This sets the right cultural tone and helps to refine the models. “All organizations seek to please the leader,” explains Gary Loveman, of Caesars, “so it’s critical to cultivate an environment that views ideas as separate from people and insists on rigorous evidence to distinguish among those ideas.” Loveman encourages his subordinates to put forth data and analysis, rather than opinions, and reveals his own faulty hypotheses, conclusions, and decisions. That way managers and quants alike understand that his sometimes “lame and ill-considered views,” as he describes them, need as much objective, unbiased testing as anyone else’s. For example, he often says that his greatest mistake as a new CEO was choosing not to fire property managers who didn’t share his analytical orientation. He thought their experience would be enough. Loveman uses the example to show both that he’s fallible and that he insists on being a consumer of analytics. When It All Adds Up Warren Buffett once said, “Beware of geeks…bearing formulas.” But in today’s data-driven world, you can’t afford to do that. Instead you need to combine the science of analytics with the art of intuition. Be a manager who knows the geeks, understands their formulas, helps improve their analytic processes, effectively interprets and communicates the findings to others, and makes better decisions as a result. Contrast the bank mentioned at the beginning of this article with Toronto-Dominion Bank. TD’s CEO, Ed Clark, is quantitatively literate (with a PhD in economics), and he also insists that his managers understand the math behind any financial product the company depends on. As a result, TD knew to avoid the riskiest-structured products and get out of others before incurring major losses during the 2008–2009 financial crisis. TD’s emphasis on data and analytics affects other areas of the business as well. Compensation is closely tied to performance-management measures, for example. And TD’s branches stay open longer than most other banks’ because Tim Hockey, the former head of retail banking, insisted on systematically testing the effect of extended retail hours (with control groups) and found that they brought in more deposits. If anyone at a management meeting suggests a new direction, he or she is pressed for data and analysis to support it. TD is not perfect, Clark acknowledges, but “nobody ever accuses us of not running the numbers.” Your organization may not be as analytical as TD, and your CEO may not be like Ed Clark. But that doesn’t mean you can’t become a great consumer of analytics on your own—and set an example for the rest of your company.

Keep Up with Your Quants

Summary.

Partner Center