The Path to Good Decisions Starts with Data

Jules van Binsbergen: Welcome. My name is Jules van Binsbergen and I’m the Nippon Life Professor in Finance at the Wharton School at the University of Pennsylvania.

Jonathan Berk: And I’m Jonathan Berk, the A.P. Giannini Professor of Finance at the Graduate school of Business at Stanford University. Welcome to the first episode of our podcast series, All Else Equal. This series is about making good business decisions. We will talk about how to do it and why it’s so important. But of course, making good business decisions is very similar to making good decisions in general. And so, we hope the principles we learn on this podcast series will apply in your personal life as well.

Jules van Binsbergen: Making good decisions is a very old problem, and a major breakthrough on how to do it properly occurred during the Enlightenment with the discovery of the scientific method. The scientific method has become so engrained that it’s hard to imagine a world without it. Many people would regard it simply as obvious. Of course, major breakthroughs are often obvious once they’re made, but discovering them is an entirely different matter. The basis of this podcast series is that good business decision making is very much like applying the scientific method.

Jonathan Berk: So, what is the scientific method? In a nutshell, the scientific method is evidence-based decision making. And the main insight of the scientific method is that you need evidence to support your theory. In fact, unless a theory is supportable by evidence, what scientists call testable, it isn’t really a theory. Scientists don’t waste time and effort on theories that are untestable.

Jules van Binsbergen: One other thing that is important from the business perspective is that we would like to reject theories and we would like to reject ideas as quickly as possible. And the reason why it is important as a business decision to try to do this, that we don’t want to waste resources on things that don’t work. And so, it’s important that we organize our organizations in such a way that you give people the opportunity to go after things and try to reject them.

Now here is the issue. For most people, it feels much better to be proven right than it feels to be proven wrong. Now it’s hard to say why that is because you could say if you’re proven wrong, you’ve learned something new. And shouldn’t learning something new make you feel good? For some reason, for most people it doesn’t work that way. It feels that if you’re proven wrong that you’ve lost something.

And so, for that reason, we need to set up rules and we need to set up a system, which is really what the scientific method did, so that we can together try to overcome that flaw. So, in group decision making and making decisions together, we would like to set up things in such a way that this human flaw of not wanting to admit your mistakes and sticking to your guns is overcome so that we together can move forward.

Jonathan Berk: It reminds me of an anecdote attributed to Alfred Sloan, the legendary CEO of General Motors. He was chairing a board meeting where a major decision was being made. And he turned around and asked for dissent on the decision, and nobody dissented. When that happened, he decided to table the decision. And he said to everybody, “We cannot make a decision of this magnitude without dissent.” And the key there is if you don’t try, you can’t falsify.

So, a common mistake for people to make is they don’t try. This is a very common mistake in business organizations. As the famous physicist Richard Feynman once said, “The first principle is you must not fool yourself, and you are the easiest person to fool.”

Jules van Binsbergen: And in many organizations, it’s even worse than that. It is that the organization sometimes, or maybe even often, is set up that falsifying things is punished. If you prove something or somebody wrong, particularly somebody that’s higher up in the organization, it doesn’t often lead to big career successes. And so, one way of improving organizational structures is to make sure that if an important idea is convincingly falsified, that the people who falsify it are being rewarded for that.

Now one way to institutionalize things is to set up explicitly what’s called red teams and blue teams where you intentionally assign people on opposite sides of an argument so that you try to get all the pros and cons of the decision out in the open so that everybody can hear about them and trade them off accordingly.

Jonathan Berk: In science, one of the best examples of the cost of trying not to falsify, or worse, having an organization that punishes people for dissent, that punishes people for falsifying, is Galileo. Before Galileo’s time, the center of the Renaissance, the center of the Enlightenment, was in Italy. And then Galileo came around with hard evidence that the universe does not revolve around the earth.

Rather than look at that evidence, what the church did was shut him down. And in shutting him down and then punishing him — wasn’t just they shut him down; they punished him — they sent this clear message that they were not in the falsifiable business. And the net result was that the Enlightenment moved to Northern Europe, outside the purview of the Catholic Church. And so, all major breakthroughs previously was in Italy, and after that it was not in Italy again. So, the cost to the intellectual environment in Italy was enormous.

Jules van Binsbergen: And so, the Enlightenment values really stress to believe that science and logic give individuals more knowledge and understanding than tradition or feelings. And so, in that sense it’s closely related to rationalism.

So now if you wanted to get some sense of how successful the scientific method has been, and it is no coincidence that if you go back for as long as we have decent data and economic growth that, say, between 1300 and 1800, we had a 500-year spell of almost no economic growth. And then in 1800, suddenly there was this explosion of economic growth. This was almost like a miracle. And it’s not a coincidence that that moment of economic growth arising coincides with the introduction of the scientific method through Enlightenment values.

Jonathan Berk: You know, Jules, I think it’s hard not to argue that Google isn’t the most successful organization of our time. And I think looking from the outside that that success is very largely driven by the fact that they all got these principles, that they have a single focus on evidence-based decision making, and essentially applying the scientific method to making business decisions. So, we’re really happy to have on our show the Chief Financial Officer of Alphabet, the parent company of Google, Ruth Porat. Ruth, welcome to our show.

Ruth Porat: I am delighted to be with you. Thanks for having me.

Jules van Binsbergen: Ruth, we are so excited that you’re with us today. So, let me start by setting the stage a bit. Right now, your company stock market capitalization is close to $2 trillion with cash holdings close to $140 billion. Just in 2020, you made $12 billion in investments in plant, property, and equipment, and you’ve also done some major acquisitions in the past few years. One example is Fitbit which you acquired for over $2 billion. These types of investment decisions are only a small subset of the business decisions that you need to make on a regular basis.

So, that really brings us to the main theme of this podcast, which is how do you make good business decisions? And I don’t think we could think of a better guest to kick the podcast off with. So, let me pose this question directly to you. How does Google make major business decisions and what process do you follow? Maybe you could also comment a bit on what you would say is Google’s biggest strength in this.

Ruth Porat: Well, I would say one of the core processes at Google that I greatly admire is what we call the OKR process. On an annual basis, every product area, every function then rolling up at the corporate level, we set objectives, the O, and establish key results, the KR. And that becomes a heavily discussed, debated strategics that set the priorities for the upcoming year.

And the goal with OKR is, one, that they’re audacious enough so that by the end of the year, you achieve about two-thirds of them. The goal is not to achieve a hundred percent. The view is if you achieve a hundred percent, you have not set the goals high enough. And then have very specific metrics where you can measure your progress along the way. And there’s quite a bit of debate and discussion to set what is that strategic roadmap, and then to roll it up so that there’s a true north for the whole company and there’s a shared point of view about what we’re doing. That’s the starting point.

Jules van Binsbergen: It sounds to me that this process is also a very data analytical process. So, that raises the question, how important is data to you? And how do you deal with people that would say things like, “I think that decision making should be done at the gut level. If it feels right to me, I want to go for it regardless of what the data says”? How do you respond when people say that?

Ruth Porat: Well, you’ve hit a couple of points that I think are most important. I’ve said so often, anchor everything in data and the rest will follow. You absolutely need to have the truth that comes from the data and the analytics. I think the biggest mistake is when people think that because they’ve done it before and worked, they can operate on gut. And frankly, I would say if you’re concerned about taking the time to do the data and analytics, why? It should be reinforcing to a decision.

And in fact, through the OKR process — and frankly, everything that we do — it helps lay out what is that roadmap, what are the metrics and milestones along the way where you can gauge whether you’re making the progress that you expected. And if you aren’t, maybe the reason is you need to invest more. Maybe the reason is that the idea wasn’t as audacious or well thought out as needed. But data [unintelligible] one of the arbiter of truth. And so, A, critically important.

I would say the other elements of your question that’s really important is when you’re thinking about what are the metrics, what data do you want? That is not always as obvious as it seems. You need to make sure that the metrics that are being selected are ones that are driving the outcome that is desired. And I think we can all think about companies over history who’ve selected the wrong metrics and set the wrong north star. So, that data and the choice of metrics is key.

Jonathan Berk: So, Ruth, speaking as a professor, often people don’t like what the data says. They come in with emotions. They come in with a set way of doing things. The data says something different, and they ignore the data. How do you deal with that?

Ruth Porat: Well, again, I think one of the really important elements in data and analytics is to look at the trends that come though the data. I think that what you don’t want is a false precision. I often say I don’t like the numbers; I like the trends and I like the sensitivity analysis. Because I think it is too easy to anchor on just a set of numbers. That, in my view, is not helpful.

The most important is to look at what are the trends over time, and then to your question, break it down to the sensitivity analysis. What are the key variables that will drive behavior one way or another or outcomes? What must you believe? And when you lay out, in my view, the sensitivity analysis and you engage in a discussion about the variables that will drive it, it makes for a more constructive process. Coming with a just a flat set of data isn’t constructive because obviously the world in which we live is not static.

And so, there may be some validity in the point that there’s concern about what a flat set of data tells you. And hence the importance in sensitivity analysis and engaging your business partner, your colleagues, with what do you disagree and why, and then trying to break down those elements of it. But to me, the sensitivity analysis is one of the key tools that we all use.

Jules van Binsbergen: Very nice. So, one other big theme that we want to address in this podcast is what are the most common mistakes that people make when making decisions. What do you think those mistakes are and what can you do about them?

Ruth Porat: Well, you both already hit on the biggest mistake, and I most certainly saw it as a banker, and that’s when people decide that they can go on instinct or their gut as opposed to data and analytics. And as I’ve already said, that really does not happen with any frequency at Google. I think the other critical element is an over reliance on what I’ve described as flat data, not looking at the dynamic nature of the world in which we operate. And that’s the value of sensitivity analysis.

But it’s also the importance of understanding when you’re living in a very competitive market as we do that any action we take is going to lead to change in that overall ecosystem and trying to think through how the world will continue to evolve. And so, living in a silo, living in a cave, and not understanding the dynamic nature of what one does I think is a key area.

And then I think the third, and it goes to the importance of setting out goals up front and having clear metrics and milestones against which you measure your performance, you have to have a way to check along the way whether things are working, and you have to admit when it’s not. And I would say that if you don’t have things that fail, you’re either not reaching high enough or you’re not being honest enough with yourself that you’re not doing as well as you expected.

And so, there’s a whole protocol at Google around what’s called blameless postmortems. It’s the goal to actually do a root analysis at the end of something that hasn’t gone as well as you want, not blaming anyone assuming that process, actually, can lead to bad outcomes. People have good intent, but you need the right process. So, what is it that we can learn through a blameless postmortem such that next time you’re in a similar situation you actually have a better outcome?

And I’m very much of the view that one of the most valuable things each of us brings is pattern recognition from some of the toughest things that we’ve ever gone through. But you need to take the time to deconstruct what did I learn, what pattern recognition is important so that it sets you up better for the next time.

Jonathan Berk: This is an important theme in this podcast series. It’s very important that you admit your mistakes. And so, why is that so hard to do when obviously you learn the most from when you make a mistake? I mean, in principle, it should be easy to admit you made a mistake. So, in that vein, what mistakes have you made yourself that you particularly watch out for? What are you most careful about not doing?

Ruth Porat: I’d like to actually comment on the first part, and then I will come to the second part. You’re absolutely right that human nature is — it can be hard even when you’re trying to make it safe to say something went wrong. And probably one of the best approaches that I’ve seen is at X, our moonshot factory, where they actually celebrate when they’re deprecating something, because they want to make it safe to say, “I tried. I reached. It didn’t work.” And if you don’t celebrate and say, “I learned something from it,” it’s going to make it hard for people to raise their hands.

I created something in finance similarly — I call it the [unintelligible] — which is I want people to raise their hands early when something’s not working. Because my view is that when we bring people together, we can solve problems. And time is the one thing you can never get back. And so, you have to find ways to actually make it fun and safe to say something seems like it’s not going as well as expected.” So, I do think that’s a critical area.

To your question, where have I found that I keep coming back to it. I would say probably the most important thing as leaders is we need to have the right organizational structure and the right people.

Jules van Binsbergen: Yes, absolutely. And there was something earlier that you said that I also thought was very interesting. Because one thing that we wanted to push as an idea is that good business decision making is very much like the scientific method. So, quickly finding out what doesn’t work so that everybody can move forward is very valuable. We don’t want to spend time on hypotheses that have already been proven wrong, even when people just want to stick with them. It’s just a waste of time — exactly as you said — time that we will never get back.

And so, Ruth, I have one more question for you that I would like to have your thoughts on. So, sometimes people argue with important decisions that they want to have a red team and a blue team where you intentionally assign people to take opposing positions. Have you ever tried this? Do you think that this is something that would work well? Under what conditions do you think it will work well, and when do you think it would not work very well?

Ruth Porat: I actually use that quite often. I think it’s a great idea. I think that when business leaders are putting forth a proposal, they’re not putting it forth because they think it’s a bad idea; they think it’s a good idea. So, what I want to hear is the what can go wrong or why would I not do this. How much consensus was there on the idea? What were the voices that were actually questioning it? And I think there are multiple ways to get at that kind of debate. There are plenty of times where I’ll say, “I want somebody to take the alternate view. Walk me through it. Why would I not do this?”

But I would say that at an even more basic level, in meetings, I think it’s extremely valuable. I think a best practice is when you’re soliciting views around the room, first of all, have a diverse team because you’ll get a diversity of views. And it’s one of the most potent elements of looking at how to actually grow as a team.

Second is start with the most junior people in the room. You want to make sure you’re hearing the voices of the most junior people in the room. They’re probably the ones who are most granular, in the data, having done the analysis. And it can be hard for a junior person to say, “I actually am concerned about X, Y, or Z” when all the senior people have already said, “This is fantastic.” So, I think there’s a process about how you run the discussion.

And then I do think at times the red team/blue team, red team/white team, whatever you want to call it, is super helpful. So, there’s a elements and process that I think can enhance the quality of the debate and the insight.

Jonathan Berk: So, Ruth, what I’m hearing from you is that at Google, you selected for people who appreciate the importance of not being in a set mindset, understand that you can be wrong, and maybe you’ve done something one way and there might be a better way. That’s not the average person. How do you maintain this culture?

Ruth Porat: At Google, I believe there’s a culture of humility. People feel we’re never doing enough. And sometimes that can be wearing because there’s always a sense of you’re not doing enough. But in my career, what I have seen time and time again is humility drives people to actually raise the bar on themselves and on everything they’re doing because their concern is we’re not doing enough. And I saw it with the best bankers. I see it at Google all the time. We should be able to do more. We should explore more. There are different things that we can be doing.

I think the key in my career that I’ve seen is that the single best predictor for failure is hubris. And it goes to so many of the things that the two of you have been asking, which is if you want to default to gut, you want to default to instinct, you’re so smart you can throw out all the data, you don’t want to engage in the debate, that’s when you need to get concerned. Because if, in fact, you approach things with intellectual curiosity and humility, and it’s about learning so that you can do even better, you’re going to want every tool at your disposal, and you’re going to want to be surrounded by people who challenge you.

And so, I think at the core of it, it is about who are we individually and how do we as leaders pull together a group where you create the psychological safety to have the debate, admit you don’t know. But intellectual curiosity and humility to me are absolutely critical if you want to build a successful high-performance team.

Jonathan Berk: I couldn’t agree more. The idea of humility, understanding that you might not know the answer, that you always need to learn, is so important. It’s one of the reasons Jules and I were motivated to create this podcast series.

Ruth Porat: Yeah. Well, to be in a leadership position and say “I don’t know” for many people is hard. I actually think that level of openness and vulnerability actually builds stronger teams. Your teams want to know that their voice is wanted. I often tell people we don’t hire super talented people and not want to hear what they say. The reason you’re at the table is we want to know what you think. And so, I think that the leader who knows everything isn’t going to get the best out of their team and they’re not going to have the best results.

Jules van Binsbergen: Thank you so much, Ruth. This was just great. We really appreciate the opportunity to talk with you today.

Jonathan Berk: It’s been a pleasure talking to you. Thank you so much for finding the time and sharing your insights with us.

Ruth Porat: Thank you. It’s wonderful to see both of you. I look forward to staying in touch and seeing where you go with this. Take care. Bye-bye.

Jules van Binsbergen: Wow, that was such a great interview, and she had so many good points to make. Also related to the things that we talk about our students in class all the time. Ultimately, this is what this podcast is going to be all about: smart thinking, concrete examples, and great guests. But more particularly, it’s going to be about making mistakes, identifying those mistakes, and understanding why those mistakes were made to begin with, because that allows us to learn from those mistakes.

Jonathan Berk: But the key insight is, unless you’re willing to admit you’ve made a mistake in the first place, we can’t even begin the process of learning from our mistakes. And I think today we saw the importance of, A, admitting that we make mistakes, but B, understanding that that is a difficult process to do. And in going through that process — In fact, I would say it another way around. The only way of becoming a better business decision maker is to be willing to go through that process.

Jules van Binsbergen: So, in our next episode, we’re going to explore a very important mistake that many people make. And in the process, we will also explain why we decided to call this podcast All Else Equal.

Thanks for listening to the All Else Equal podcast. Please leave us a review at Apple Podcasts. We love to hear from our listeners. And be sure to catch our next episode by subscribing or following our show wherever you listen to your podcasts. For more information and episodes, visit allelseequalpodcast.com or follow us on LinkedIn. The All Else Equal podcast is a joint venture of Stanford University’s Graduate School of Business and the Wharton School at the University of Pennsylvania, and is produced by Podium Podcast Co.