How to Chat with Bots: The Secrets to Getting the Information You Need from AI

Note: Transcripts are generated by machine and lightly edited by humans. They may contain errors.

Matt Abrahams: The promise of AI is tremendous, yet most of us, if we’re using it at all, are using it incorrectly. It’s not about a transaction, it’s about a conversation and interaction.

I’m Matt Abrahams, and I teach strategic communication at Stanford Graduate School of Business. Welcome to Think Fast, Talk Smart, the podcast.

Today I am really excited to chat with Jeremy Utley and Kian Gohar. Jeremy is a repeat guest to Think Fast, Talk Smart. He’s an adjunct professor specializing in creativity and entrepreneurship at Stanford and the author of Ideaflow: The Only Business Metric That Matters.

Kian is founder of Geolab and former executive director at Singularity University. He’s the best-selling author of Competing in the New World of Work. Welcome, Jeremy and Kian. Thanks for being here.

Jeremy Utley: Thanks for having me back.

Kian Gohar: Such a joy to be here with you.

Matt Abrahams: All right. Shall we get going?

Jeremy Utley: Let’s do it.

Matt Abrahams: To get us started, Kian, I’m curious, what motivated you and Jeremy to look at the impact of AI on creativity and problem solving?

Kian Gohar: So in my business, we coach teams to achieve complex business goals through team transformation practices. Sometimes this involves tackling a difficult innovation project. Sometimes it’s struggling with change management. But it always boils down to this one issue, which is how can we solve this problem? And that usually entails human ideation and prioritization.

And I wanted to see if we can bring a new technology tool into the mix to get better ideas. And given that I’ve been teaching AI in Silicon Valley to executives for a long time, that was always my hunch. But it wasn’t until ChatGPT became publicly available that I had the aha. And I didn’t quite know how it could be useful to the issue of problem solving on teams, but I had a hunch.

And so we designed the study together, uh, to learn in from real world practice of how teams might use AI to get better ideas to solving problems. And what we did was we recruited hundreds of participants from many companies in Europe and the US and we asked them to identify a particular pain point within their organization or a problem that they needed to solve, or something that was real. And then we actually gave half of the participants in each of these problem-solving ideation workshops had access to ChatGPT, and the other half didn’t have access to it. So they were just thinking on their own as humans and then the other half had humans plus AI. And we went through a problem-solving ideation exercise.

And at the end of it, we asked the problem owners or the executives who shared with us the particular pain points to grade all the various ideas from A, being spectacular all the way to D, not worth pursuing further. And so we ran this blind study to understand how generative AI can facilitate problem solving, ideation and collaboration with real world examples.

Jeremy Utley: The other thing that we did was we asked participants before and after the session how they felt about collaboration and problem solving and ideation. Because we wanted to see what was the impact of just a standard brainstorming session on someone’s attitude towards problem-solving and collaboration and innovation. And what was the impact of a session that included generative AI? Did it have a differential impact on participant sentiment?

Matt Abrahams: And the research to me is fascinating. Jeremy, can you summarize the results of the research you and Kian did and relate it to the quality of idea generation and the feelings people have about the ideas that were created?

Jeremy Utley: Very broadly speaking, we found that teams, if they want to outperform using AI, they need to adhere to certain practices. And sadly, most teams do not. So despite the potential to dramatically outperform, most teams leave the vast majority of their innovation potential on the table when they use AI. When you approach AI like an oracle, or like a search engine just looking for the answer, it may feel like magic, but the data shows you end up underperforming.

If you want to get world class results and outperform, you can’t approach it like an oracle. You need to instead approach AI like a conversation partner. And that doesn’t feel nearly as magical. It feels more like work. And for that reason, the teams that outperform actually feel worse than the teams that underperform. It’s very counterintuitive.

Matt Abrahams: I find that absolutely fascinating. And I also find that interestingly, the ability to have a good conversation with AI is what makes the difference for creating valuable ideas and solving problems.

Kian, can you go into a little more depth with the counterintuitive findings that Jeremy just shared? Why do you think it is that AI assisted teams that delivered worse solutions actually felt better about their work and the AI assisted teams who delivered better outcomes actually felt worse related to their non-AI assisted counterparts?

Kian Gohar: Because rewiring our brains to act differently takes practice and we professionals have decades of experience or certain ways of doing things. And we’re asking them to work differently now with a different workflow, and that’s hard. It’s like going to the gym after not having exercised for a long time. It really sucks. And you don’t particularly enjoy it at first, but it has long term benefits. And so we found that teams that were able to use AI effectively, they actually had to rewire their workflow on how they went about to try to incorporate AI as a co-pilot on the team. And that’s different. And it was exhausting, and they felt like it was just a lot of work, but ultimately they had better responses.

Matt Abrahams: Would you add something to that, Jeremy?

Jeremy Utley: Part of the challenge is I almost picture a cartoon strip with a thought bubble that’s like, but I was promised superpowers. The premise and the promise of AI, I think it’s intimidating, and it maybe strikes fear into a lot of people’s hearts. That’s one thing I don’t think there’s a reason to be afraid. I think there’s a lot of reason for enthusiasm and fun and seeking fluency, obviously. But I think there’s a premise of this is going to be like magic. But when you start there and you think all I have to do is ask a question and then I get answers, well, guess what? You do.

You ask a question, and you get three pages of call it B plus documentation and teams go, it is kind of magical. Well, Let’s go get coffee. Like we’re kind of done, right? If you’re expecting magic, you will get it, but you won’t do that well. If you’ve gotten practice being a conversation partner to an AI co-pilot, what you realize is the first stuff that you get out isn’t great unless you’re really thoughtful about framing and providing context and digging deeper and pushing back and asking questions and treating it much more like a conversation. And then the more you treat it like a conversation, the less magical it feels, and the more it feels like AI is actually getting you to work and pulling your best thinking out of you.

And so I think part of it is just the mindset people have when they hear, oh, we get ChatGPT. The people who go, oh, magic are almost always the underperformers. The people who roll out their sleeves and go, oh, now we have work to do. It’s maybe a different kind of work to do, but I’m excited to do it. Those are the people who outperform.

Kian Gohar: I would also add that there is a reason why ChatGPT gives you generic answers. And it’s designed mathematically to give you the most likely answer that it thinks you want to hear. And so as a result, the first few suggestions it offers are supposed to be generic and average. And so unless you have this conversational back and forth with it, like you would with a colleague or a friend to push it and to give it context, you’re just going to get average answers and average ideas. And that’s really not good enough if you’re trying to solve complex problems.

Matt Abrahams: It comes down to mindset shift and work. There is no magic, it’s not for free. If you really want to solve problems well and get creative, you actually have to have a conversation. So Jeremy, can you share your FIXIT methodology and describe the different components and how they work?

Jeremy Utley: Absolutely. So FIXIT is a five-step methodology to turbocharge your collaboration. F stands for you have a focused problem, right? So you don’t want to boil the ocean, but you want to be very focused in the kind of challenge that you are bringing to ChatGPT. So in this example, instead of saying, what recommendations would you make for a human interfacing? I would say, I’m joining a podcast and I’d like to give concrete, tangible suggestions to professionals who haven’t had much experience with ChatGPT. If you had three or four, or maybe I’d start with ten suggestions for someone who’s unfamiliar but wants to be more familiar, start by asking me three questions to better understand the kind of professional I’m describing, right?

That’s what we mean by F, kind of focused, that’s a very focused prompt, right? I is ideate individually. So before coming to the team, because remember our study was conducted in the context of teams trying to solve problems. Before coming to your team, and even before coming to ChatGPT, think for yourself about what do you know and what’s your point of view.

Again, this kind of hones the context that you bring to the conversation, right? It’s critical. Research has shown that the best way to brainstorm is to alternate kind of individual ideation and group ideation. Well, the same is true actually in the context of collaborating with generative AI. You want to alternate between individual personal thinking and thinking assisted and amplified by AI, right? So that’s the I.

X is context. So providing sufficient background context. A lot of times we recommend that folks upload documents, right? When we were conducting our study, we would have a problem owner actually do a dossier that we would upload to ChatGPT to provide context. If you don’t know what your context is, here’s an amazing hack. Ask ChatGPT to ask you for the context.

So as an example, I’ve got a friend who’s negotiating a lease, trying to build a gym and he’s trying to come to an agreement with the property owner and there’s a gap between where they need to be. And I said, hey, why don’t you ask ChatGPT for help? He said, well, how would I do that? I said, well, have ChatGPT interview you about your objectives and then interview you about the counterparty’s objectives and then make some recommendations. Well, that interview effectively becomes a means by which you can provide context. So the X is make sure that you’re giving ChatGPT context on the problem.

The second I is interactive, iterative conversation, right? So you’re never just taking the first response that ChatGPT gives you. You want to be having a back and forth and a dialogue. So for example, if Matt and Kian and I are going to try to title this episode, we might say to ChatGPT, hey, we’d love ten titles for an episode. And then ChatGPT comes back with ten. Almost everyone’s default is to think, which of these ten do I like best? What we’d recommend is just immediately say, I’d like another ten, and then read all twenty and say, here are the ones that I like, would you give me ten more like these? Chances are you’re not going to get as good.

And then if you think, wait, why did I like number two? Oh, there was a funny alliteration. And number seven had a funny pun. And number nine made reference to, okay, use those as design principles for the next ten, right? That’s an iterative back and forth that yields, we’d probably find that the fourth tranche of ten title suggestions would be radically, exponentially better than the first ten, right?

But it’s by that going back and forth. F I X I T. T is for team incubation. So it’s important to then bring the ideas that you’ve generated individually and with ChatGPT or whatever LLM you’re using back to your team. And then importantly commission some experiments. So this is where this work dovetails with traditional innovation methodology, but you never just want to select one idea and move forward.

It’s impossible for an LLM or for a human to a priori know which solution is going to be the best fit. So as a team, you want to have a practice and a process around incubating or low resolution prototyping a handful of the high potential solutions that you’ve generated in order to determine which one actually solves the problem the best.

Matt Abrahams: This methodology, upon hearing it, makes a lot of intuitive sense. But I can definitely see the effort involved. I like that it involves individual work and collaborative work, not just with the ChatGPT LLM, but also with others on the team.

Kian, not surprisingly, I’d like to dive deeper into the fourth step that Jeremy introduced us to, interactive conversations. We’ve spent a lot of time on this podcast talking about how to have better conversations, but of course, focusing on humans. What specific advice can you provide to us for improving our conversations and our communication with LLMs to make sure we maximize the potential goodness that can come from collaborating with a tool like ChatGPT?

Kian Gohar: Yeah, so the first thing I’d say is to make sure you’ve downloaded a LLM app on your phone. And interact with an LLM on the phone instead of doing it on the web browser. If you don’t see an app on your phone, it’s very unlikely that you’ll use it on a consistent basis. And the more you use it, the more familiar you become with it, the more likely it’ll actually be part of your workflow, whether it’s individually or whether it’s as a team.

So that’s really the first step. Download one of these apps on your phone. And part of that is because we have historically been for the last twenty plus years in the internet era and the browser era, we see a text box and we know exactly what to do with it. We type in a particular word or particular question.

And now that we have, uh, large language models, the user interfaces look pretty much exactly the same, like the Google search engine box. And it’s oftentimes difficult to figure out the right kind of question to ask it or to word it in the right way to get the right answer to suggestions. And so we actually think it’s a lot better if you start interacting with these large language models through conversation, through spoken audio, rather than just trying to type it into a search engine box and trying to figure out the exact right prompt.

The second thing is that you should be uploading your prompts with audio messages, voice messages, and talk to it just like you would talk to a friend. So whether you’re talking to a friend about a particular problem or talking to a colleague about a particular problem, just record it literally on the phone on audio to text, and then the large language model will transcribe that and then offer out some suggestions.

And then the third thing I’d say is think about using different models. There are several different kinds of large language models that you can use from ChatGPT to Claude to Bing and to others. And they all have their different flavors and different kinds of personalities. And you might want to use one of those models for a particular exercise or activity. And then maybe you should go talk to another model, just like you would talk to two or three different friends and get their opinions or two or three different colleagues and get their opinions. Because they’re going to give you different kinds of nuance, different kinds of answers.

Matt Abrahams: I find that really helpful advice. I mean, we approach the communication with technology based on the interface that we have with it. And if we move to our phones, which we’re used to communicating with very differently than we are, let’s say a browser in a text box, it can really change it. And it strikes me that it’s not just the actual interaction interface, but it’s also the curiosity we bring to it.

When I go to enter information into a search engine, I’m not just I just want to get the answer. I don’t necessarily see it as a conversation and a dialogue. So that curiosity that I bring with the follow up questions or the doubting and the exploration and expansion I think is really important. And I like the advice to check with different LLMs, just like you would check with different friends. It gives you different ideas and inputs.

Jeremy Utley: You can even feed different LLMs answers into one another, right?

Matt Abrahams: Oh my goodness, you can facilitate and broker a conversation amongst LLMs.

Jeremy Utley: Absolutely. I mean, I plug in ChatGPT’s answer into Claude and say, what do you think of this all the time? And vice versa, take Claude’s answer and plug it into ChatGPT.

Ask ChatGPT what it thinks of it. So this is why Googling is such a bad metaphor for what we’re talking about here. Because you’d never query Google about a Google query to get meta for a second, right? But after a sales call, I hop on my ChatGPT voice app while I’m stretching for a run, instead of sitting at the screen. I’m stretching for a run. I say, hey, I just talked to Matt and Kian about our research and a couple of follow ups. Would you craft a quick memo that I could send to the team, just thanking them for the time today and how much I enjoyed the conversation, right? Well, it’s going to do it. Well, then my next thing is I just look at it and I say, if I were to upload this memo to you, and I were to ask you for advice on how to make sure that they read it and respond, what three changes would you recommend?

And then immediately, I had this happen the other day. I was doing a voice vomit on a post sales call and I had ChatGPT write a memo. And then I asked ChatGPT, how would you criticize this memo? And ChatGPT said, it’s far too long for today’s busy professionals. No one’s going to read this memo. And I said, would you please go ahead and shorten it to a point where you think that people actually read it? You can ask ChatGPT to evaluate its own work and it will do so dispassionately, right? And that’s the fun, but that requires iteration and a back and forth.

Matt Abrahams: In many ways, the advice that you all are giving are advice that we give to people when they are engaged in empathetic dialogue, and human to human interaction. It’s about curiosity, questioning, about challenging in certain ways. And it changes the metaphor completely, as you mentioned, Jeremy.

So before we end, I like to ask my guests a series of questions. Two are similar to everybody and one that’s very unique. So Jeremy, for those who haven’t started with ChatGPT and large language models, where should they start?

Jeremy Utley: I think for a lot of people, if they say, where should I start? We’ve talked to so many professionals who say, I’ve been meaning to try ChatGPT, I’ve been meaning to try generative AI. And that’s us as a shape every single listener to this podcast could have at least ten hours of ChatGPT under their belt.

And if you find yourself going, Oh man, I’m behind. Well, don’t worry. You can get up to speed quickly. Here’s a simple place to start. This is something that every single listener can do right now. Think of an emotional human decision you’re trying to make in your life. Something that you would ordinarily ask a partner or a friend or a spouse or a colleague about.

Okay. Think about what that is. Go to ChatGPT and say, hey, I’m trying to make this decision. Will you please ask me three or four questions before giving me your recommendation? If every single listener will do that one simple activity, they’re going to have what we call a personal epiphany that’s going to have cascading impact. All of a sudden, they’re going to start thinking, could ChatGPT do this? Could I ask ChatGPT about that? Right? But it must be emotional. It must be deeply personal. It must be the kind of thing that you would ordinarily ask another human being about. Get ChatGPT to ask you three or four questions about it before giving you advice and start the snowball rolling that way.

Matt Abrahams: Question number two and three, Kian, I’m going to ask you because Jeremy has previously answered these on an earlier episode, and I encourage everybody to listen to that episode where Jeremy and I talk about his book, Ideaflow. So Kian, who’s a communicator that you admire and why, and you can’t say ChatGPT?

Kian Gohar: I’m going to give you two answers. One is Peggy Noonan, who is a columnist for the Wall Street Journal, and I so deeply admire her elegant prose and her personal anecdotes that make complex topics readily understandable. She was a speechwriter for George Bush senior. And she developed the phrases that we became very familiar with during his presidency, like kindler, gentler nation, or read my lips, no new taxes.

And the politics and the policy aside, the ability to translate big ideas into a few short words is just so masterful that I always enjoy reading her articles in the Wall Street Journal regardless of the politics or the policy. And the second person I’d recommend is Sam Horn. She is an author of a book called Tongue Fu. And she is a complete master at teaching which words to lose and which words to use. To convey meaning without tripping over yourself and creating unintended arguments. And these two are role models of how I think about communication.

Matt Abrahams: Thank you. Both of those recommendations, I hear you talking about words, and the ideas that those words bring about. Both of those are individuals who have mastered that craft.

Kian Gohar: Because words are really conversations that are the seeds of innovation, and how we think about solving big problems. And if you don’t get the words then we won’t be able to actually get to the meaning and connection of what we’re trying to solve for.

Matt Abrahams: Final question for you, Kian. What are the first three ingredients that go into a successful communication recipe?

Kian Gohar: The first ingredient is something that might seem counterfactual when we’re talking about a communication recipe, and that is listening. Listening to the environments, listening to the context, and listening to what your counterparty might be thinking. And so trying to understand the context by listening is super important.

The second one is to know your audience. Who are you speaking to? Who’s on the other side of the table? Who’s in the room? What are their priorities and what are their goals? And so that you can think about crafting your message in a way that resonates with them.

And the third thing is to know your end goal. How do you want the audience of your intended communication to feel after the communication is over? Whether that’s in person, whether that’s in a meeting or whether that’s online. How do you want the audience receiving the communication to feel? And then you work backwards from that to structure the flow and the necessary ingredients to make your communication easily relatable, understandable, and to land.

Matt Abrahams: We have certainly heard the notion of knowing your audience and being critical. I really appreciate and like this idea of backward mapping. Start from the outcome and then we have to build, and do, and say an emote to get to that outcome. Thank you for that. And thank you both, Kian and Jeremy, for the conversation. I find your work fascinating and super helpful as generative AI becomes more and more commonplace.

Further, I love how the concepts of effective communication and conversation and critical thinking can help our partnering with AI to be better, more creative problem solvers. To learn more about the results from Kian and Jeremy’s work, go to howtofixit.ai and check out their article in HBR. Thank you so much.

Jeremy Utley: Thanks for having us.

Matt Abrahams: Thank you for joining us for another episode of Think Fast, Talk Smart, the podcast from Stanford GSB.

To learn more about creativity and AI, please listen to Episode 70 with Jeremy Utley, in Episode 77 where I have a conversation with ChatGPT. This episode was produced by Jenny Luna, Ryan Campos, and me, Matt Abrahams. Our music is from Floyd Wonder. Please find us on YouTube and wherever you get your podcasts. Be sure to subscribe and rate us. Also, follow us on LinkedIn and Instagram. And check out FasterSmarter.io for deep dive videos, English language learning content, and our newsletter.