Hello everyone and welcome to Better Done Than Perfect, a show for SaaS marketers and product people. Our awesome guest is Connor Joyce, Senior User Researcher at Microsoft and the author of Bringing Intention to Impact. And we're going to talk about the impact mindset today.
This show is brought to you by Uselist, an email automation platform for SaaS companies. On board, engage and nurture your customers as well as marketing leads. To follow the best practices, download our free printable email planning worksheets at uselist.com slash worksheets. Hey, Connor. Hey, Jane. Thanks for having me.
We're excited to learn from you. Before we dive in, tell us more about your book and your work and your background story. Yeah, absolutely. So I started my career really in my undergrad where I was a psychology and management major.
undergrad. And I liked that combination because it both had the business side, which is where I knew that I ultimately wanted to operate within, but it had the psychology side. And I've always really just enjoyed understanding people more. And that led me to management consulting at Deloitte, where I was doing large scale change management.
And what I found was change was trying to be done by having people do these large waterfall approaches. We're going to build a bunch of change. We're going to get everybody to change. Then we're going to freeze that change was the style. And at the same time, I was reading about behavioral science and this more iterative based approach to trying to create change.
It was more like agile product development where you build something small, you test it, you see what works, you scale what works, and then you reiterate what doesn't work. And I'm thinking to myself, why are we not trying to take this approach more? It seems like this is the one that's going to actually be more effective.
So I ultimately left and I joined a behavioral science master's. And there I really solidified my belief that good behavioral change is more rapid. It involves experimentation and it has to just be done with an open mindset of how to create that actual movement in change. As that happened, I realized the best way to do that is really through technology.
Technology is the way to take those micro interventions, the things that are actually driving behavioral change, and be able to scale them to large audiences. And so at that point, I said, okay, it's time for me to go into tech, thinking that when I would get here, that everyone would have this mindset.
Unfortunately, that wasn't the reality. I very soon found myself in a tech environment that is oversaturated with the belief that usage is the best metric. If people are using something, it must be working. That's what I heard. And that is what I've learned to be the biggest misconception in product development in our modern tech environment. And so since then, I have been on a mission to try to change that. And I do that through the work that I do, along with the book that I wrote.
Congrats on the book launch, by the way. Thank you. Thank you. What were the biggest discoveries before you started on the book and after you finished? What's the difference? What were some big mindset shifts that you have gone through?
Well, the real answer is I thought if I built it, they would come. And I think that it's true to a degree. It also, just like everything else in this world, requires marketing. And I am very proud that the people that do sit down and really understand the impact mindset and decide that they want to try to be the catalyst of change in their own work environment are
I hear great things from them. I have over 15 companies that I have had individuals go and run these types of projects where they come back and they say, thank you for this insight. It helped me move this forward or it helped me rethink this piece of whatever they were working on. But to get those insights,
I had to do a lot of marketing. I had to get out into the public and share this philosophy. I had to attend meetups, shake hands, jump on calls, which is great. And I am completely fine doing it. But I think I had the misconception before I wrote the book that if I wrote the book
It would just instantly turn into this doctrine like we see with just some of the canonical design thinkings and jobs to be done. And obviously those didn't happen overnight either. There's a part of me that thought that it could. And now that I'm a few months into having the book
live, I realized that the success of the book and the ideology behind it is really dependent on how much I can continue to get the word out there. And again, it's why I'm so happy to be on the show and talking to a new audience here. So the foundational concept in your book is the impact mindset. Could you walk us through that a little bit? And there are four components to it. Give us an overview of those four components.
Absolutely. Yeah. Yeah. Before this, I when you and I were talking, I really liked the framework that I actually kind of constructed. So I'm going to walk through that again and then I can dive more into each of the four components. So the impact mindset is a philosophy and it is a philosophy because it is something that a company adopts as part of their overall strategy for product development.
It starts and its core is a framework, what I call the user outcome connection. And the user outcome connection has three components. It says that specific behaviors drive user outcomes that ultimately create business impact. So if you're building a product that's supposed to help customers create an emergency savings account, that product outcome is the emergency savings account.
And to get someone to do that, you have to change certain behaviors, things like ensuring that they increase their contributions to that savings account, decrease their withdrawals from that savings account, and other behaviors that are going to lead to a higher and more sustained emergency savings. And in theory, and ideally what you want to validate is if you do that, people are going to stay on the platform.
I like to use the example of Acorns. So it's a pretty big app in America. And it is an app that helps people create emergency savings by auto rounding up every purchase that they make and contributing that money to that savings account. And that Acorns was ultimately purchased, I believe, by Intuit. So it shows that they did build a successful product.
That user outcome, again, that is the core framework of the impact mindset. And it says specific behaviors yield impacts to the user outcome and then ultimately drives business impact. So that's the core framework.
To be able to validate that that framework does exist, you use something called the feature impact analysis, which is a process. It's an experimentation and testing process that is the bulk of the book. And it goes step by step on how you actually validate if each of those connections exist within the user outcome connection. If it is true that changing a specific behavior does change that outcome, and if changing that user outcome does actually change that business impact.
So that's framework, user outcome connection. Process of validating it is the feature impact analysis. That's two of four components. The third is what I call the insights hub. And it's a place to store all of those definitions of features from the user outcome connection and all of the data that is collected from the feature impact analysis. It's what some people may call a research repository.
It's what others may call just a notion page that is filled with a bunch of different information about the product. It's up to the company to decide how they best do that. But the important piece is that there is a centralized location that all teams can go and visit to create alignment around what the features are and how they're contributing to the success of the product. And then the last piece, and it really is the underpinning of this whole
products or this whole framework process and centralized location is that to do this successfully, companies have to adopt an evidence-based decision-making culture. They have to be accepting of experimentation, which means sometimes you fail. They have to be willing to ask the question, why do we believe that this actually exists? What is the evidence behind the decision we're actually making?
And so adopting that culture enables the other three components to be able to effectively be deployed and create value for the company. The first question I'm eager to ask is how would you like an ideal outcome just exist in a vacuum pretty nicely, but in real life, if you're building a complex product and which has a lot of things it can do, a bunch of features, do we judge
that the ideal outcome is all features being used completely regularly and fully, or do we settle on like a median, you know, 50% there or 80, 20% there situation that would make users somewhat happy and stick with you? Because in reality, of course, we want to have full adoption, but in practice, complex products are being used in the multitude of ways.
It is a great question. And I not as much in the book. I do talk about some in the book, but I've really found this to be an important aspect when I'm in my talk track for some of the conferences I've been speaking at is products at the end of the day in a competitive market are pretty similar. In America, we've got Uber and Lyft. If you think about Uber and Lyft, they're not that different.
DoorDash and Grubhub, they're not that different. A few of them have just a couple of unique features that create their unique value proposition of how they ultimately compete in the market. And so for me, I think that that's the place to really start for a company is every Uber-like company, ride-hailing or ride-sharing company is going to have
The ability to see where your car is and see as it's making its way to you. That is a pretty standard feature. You got to have something like that. In an email marketing platform, every company is going to have some sort of templates. They're going to have some sort of way to try to make it easier for customers to be able to go in and develop templates.
whatever campaign they're building. But what are the unique value propositions that each company is trying to say, hey, we're different in this way and that's why you should choose us. Those are the features that I think a company needs to start with when they're using this type of ideology is choose the most important features and
Choose the ones that really do construct the environment that says we are the right product for this customer and make sure those are working. Because if those are working, then organically that will drive more adoption of those features because people are going to recommend it to others. If it again, if it's something that is working.
it's going to yield loyal customers who are more likely to go and share about that service to their friends, to their colleagues. And so if you start with building, ensuring that the features that are most important to your product are being developed and ultimately deployed in a way that they're going to drive user outcomes, then the rest of the other metrics that you care about, usage, usability, things like that, should also increase. If you're finding you built something that is working,
and it's still not being utilized, people still aren't using it at all, then that is a separate design-related question of saying, well, how do we make this more enjoyable to use? How do we increase the ease of use? How do we increase more of the usability metrics to actually see people engaging with this feature? If you build something that works, the people should ultimately want to use it once they find that it exists and find that it does work for them.
There are a lot of open questions about this because let's say, let's talk serums because we're not a serum. HubSpot, Pipedrive, name your own serums are just serums. All of them are pretty wildly successful businesses. I cannot say how HubSpot is different from Pipedrive in terms of
you know, certain features, even though HubSpot and Intercom are all-in-one tools, and that's probably one of their advantages. But other than that, Pipedrive is thriving. I don't recall any unique angle to them. I don't know them enough to tell you what their unique angle is, but I know HubSpot is a successful product. I also know it has a whole bunch of different components to it.
I've used HubSpot before, and it was hard for me in a very iterative and as a new user to quickly adopt the platform and be able to understand how I could integrate it into the workflow that I had already developed as I was scaling a business. I ultimately used another CRM that was a startup. I can't remember their name. I think it was Apollo.
But I can't remember exactly who it was. But what I will say is that company made it too easy to start tracking whatever I was doing within this. I think it was just a management of people through a website form. So what I would say is in that case, what I'm imagining is that since HubSpot
has become so big and it is so multifaceted that they probably have a collection of features that are very useful to users. They're satisfying those user outcomes. And for the people that find that they exist and find how effective they are at what they need, they love it and they retain it.
Yet there are a lot of users that get to that platform, are overwhelmed by all of the options, and as such, they churn and they go and find a simpler option like the, I think you called it pipeline, that I'm imagining is a smaller company with a more direct service that is able to actually just deliver on a very tightly scoped, here is the user outcome we satisfy. We do this very well. That's all we do. That would be my hypothesis, not knowing the companies.
Thanks for this detour, but it was essential for me to ask the question on that. Your second component is the most fascinating to me, where you're measuring the impact, trying to measure the impact of features using different mediums. And I know you have two approaches, a simple one and an advanced one, right? Tell us more.
Yeah, yeah. So before I jump into the simple in advance, what I'll first say is that I like to say that there's a five step of success metrics. So the first step is usage. And again, I'm never trying to say usage should never be used. Do I think it's overused? Yes. Have I sat in meetings where people said, but it's being used, so obviously it's working. And I thought to myself, you're wrong. Yes. But I'm not saying usage shouldn't be used.
It's just that it shouldn't be the sole metric that is used. The next level is very important, but also not enough, and that's usability. So if usage is, are people actually using it? Are they retaining the platform? Usability is, is it easy to use? Is it fun to use? Are users satisfied using it? Those sort of questions.
And again, those are really important because you could build the best solution in the world, but if it isn't used and people don't like to use it, it's not going to have that much impact. And so you need to build something that people actually enjoy interacting with. And that's the usability piece. Now, most companies are familiar with those two levels where my frameworks extend and to lead into your question of what you're actually building are three new success metrics for products.
The first is behaviors.
And that connects to that specific behavior piece of the user outcome connection. What behaviors are actually being changed and how can we measure that? So if it's an app that's supposed to help with fitness, is if the app is to help people lose, we'll say, yeah, lose weight, the behaviors would be increasing calories burned, decreasing calories consumed. So it's about how to build features that are going to be able to help with those. And then you can measure that.
On an average day, are we seeing a decrease in calories consumed? On an average day, are we seeing an increase on calories burned? Those would be the types of ways you can measure a third level of success metrics, behavioral metrics. The fourth are user outcome metrics. So this would be in that case, again, of the fitness app, is that person's weight actually changing?
Are they feeling more confident in their body over a length of time? Those would be examples of how you can measure the user outcome based on that user outcome connection framework.
And then the last one is business metrics. And the reality is, is that there are other teams outside of UX and generally and sometimes marketing, but even outside of marketing that handle the business metrics that are most important. And so that is a collaborative effort between the product team and the business to determine what are the right business metrics that these products should be driving toward. So.
Five levels of success metrics. Usage. I'd love an example of the business metrics. Thank you. Yeah. So in that case of the fitness example, if we built something that helps people lose weight by having them increase their calories burned and decrease their calories consumed, we would hope that they would stay loyal to that platform. So we would see higher levels of retention. We would see higher levels of organic word of mouth marketing.
We would see increased upsell opportunity. So maybe that person converts to a premium account or they buy extra tokens. Those would be some examples of all of the business metrics that would lead to the ultimate business metric, which is we are bringing in more revenue than we are losing in the costs that we have associated so we can make a profit. Let's talk about those simple advanced approaches to measuring feature impact.
We have this problem of over-focusing on usage in the tech space. And it's not because we are trying to be misled or we're trying to ultimately choose the wrong metrics. It is because we are choosing what is the most easy to gather.
And I get it. A lot of listeners here are going to say, well, yeah, I would be doing something behavioral if I could, but I don't collect that data or I don't know what to actually collect. So the simple route starts with thinking as an ethnographer. Literally, what are people doing today to try to change that outcome?
And you as any sort of product person can go and figure that out. All you have to do is recruit someone who's within your target audience and ask them or go and follow them if that's possible and see what are they doing today? If you want that person to, if someone is trying to go and lose weight, what are they doing today? What like do they believe is going to actually work for them? And you can go and see how they go about trying to do that.
If you, again, with the emergency savings, have somebody that is saying, I need to build emergency savings, well, what are they going to do to actually go and try to do that? And those are the specific behaviors that if you watch someone, if you're actually able to see and talk to them and take this approach of whether it be a user interview or just as somebody who is, again, like an ethnographer, which is like an anthropological approach.
approach of just being a shadow on the wall and watching somebody, you can begin to understand what they're trying to do. And then whether you can build that metric in your platform, awesome. And if you can't, then it's about finding a proxy metric of being able to
get as close to that specific behavior or as close to that outcome as possible with whatever data you have. And it is, it's not the easiest approach, but it is a much more effective approach for building successful product.
Now, on the flip side, that's the simple approach because it is simple in the sense that you don't need much technology. You just need users. And with users alone, you can begin to understand what is possible within your current data infrastructure for measuring those behaviors and user outcomes. On the advanced side, that does require technology, but it can also be really effective. And technology here is with generative AI.
Now, it is not a perfect simulation of getting in and really doing the testing with users themselves. So I'm not trying to say that this is intended to be a complete replacement. But if you're running on a low budget and you can't get in with actual users and do the proper testing and infrastructure development to capture things like event data, which is a great source of behavioral data, well,
Then go and ask ChatGPT, here is the outcome I'm driving for. What are some of the ways that people change their behavior to drive that outcome? So it's like the outcome might be, I want to increase my meditation.
I am building a solution that is going to help a user increase their level of meditation or frequency of meditation. What specific behaviors can be created that will yield a change to their level of meditation? Ask ChatGPT that. Use search on ChatGPT, and it will give you sourced ideas for how people do that.
One might be making a little sanctuary in their house, a place that they know, a cushion that they can go to. And now that could be a feature idea. That could be something that would help a user actually increase their meditation. Another would be, you know, same frequency within like a frequency of doing a meditation the same time every day. Again, that could be a feature idea. But the advanced method of trying to create these metrics is to have ChatGPT develop both
what behaviors could be driven to change an outcome,
Obviously, the more research you can do to validate them, the better. But then you just ask ChatGPT, now that I've defined these behaviors, what are all the ways I could collect metrics to see if people are actually doing this? And that is, it is actually simple in the sense that you're not doing the labor of having to go and watch somebody and ask them questions. But it is advanced in the sense that it does take a lot of iterations to really get this down. And it does still require some validation on the back end to make sure that
You didn't get misled by these generative AI systems. Do you personally believe that niche B2B insights, let's say people who do SaaS email marketing, there's like 500 of them in the world, probably 400 of them are silent, 100 of them are asking newbie questions. Just kidding. But like in general, do you think the foundation of their knowledge is somehow available to chat DPT to feed insights?
to you and similar to any B2B software building niches, et cetera? At this point, yes. I truly believe, and I have done a lot of work, not with the specific business side, but I think even more nuanced than the business side is personal development work. And I do on the side, I run a men's group and I do a few other things that just personal development work.
And I like to say to people, because I genuinely think it's true, the most incredible aspect to me about generative AI systems is that it has made me realize how similar we all are. Because whatever one is experiencing in their life, the reality is, is that someone probably experienced something pretty similar, wrote about it. And by this point, these systems have
actually taken in that information and it has changed their philosophy of how they create their outputs. And I say that if what I say you find skeptical, literally go into ChatGPT and describe a very personal situation that you're currently facing and see what it says. And it's probably going to give you pretty similar advice that you would hear from a parent or a best friend or a spouse. It does a really good job actually understanding people.
On that part, I do believe you, actually. There are way more, let's say, wives who don't like their husbands out there than they are niche email marketers. But I would say that with the niche marketers, so I think that you're right with that specific, but I think that there's only so many people who have experienced this.
I don't know. I don't want to get too niche here and call out any specific group, but we'll say, I mean, better yet, I'll say it for myself. I'm in the middle of a divorce. I'm 31. So there's not many men that are in the middle of a divorce when they're 31.
And yet, it gets me pretty well, and also highly educated. It actually understands my situation very well and has played a good role as a personal development coach for me, really meeting me where I'm at to the same level that when I was doing actual coaching, I felt was pretty equitable of the level of domain knowledge that the human had with the generative AI system.
And so I do believe that. Now, what I want to say, too, is the difference, I think, for your case, too, and especially in B2B spaces, is what are B2B spaces known for? Churning out SEO content, because that was a great strategy for a lot of the 2010s. And so there is a lot of great content that is on the internet that was created online.
for that specific purpose to explain why each product is the best product and how each person's competitive in the market, et cetera, et cetera, et cetera. So I actually believe that with that specific case, and really in most B2B spaces, there's a lot of content out there that these systems have ingested and will actually do a pretty good job creating that persona. Are there personas that these systems don't
Do well. Absolutely. At UPenn, where I'm a guest lecturer, I was talking with some students who are doing a project in Africa. It's Africa. It's about a hygiene-related thing. And we were doing some testing, and ChatGPT had no idea what to do. And so there are cases that don't. But as long as it falls within the five, we call it weird in psychology, weird psychology, Western, educated, industrial psychology,
rich and democratic, as long as it falls within those groups, there is probably a significant amount of data that these systems have ingested, integrated into their models, and then now have turned into can simulate those answers pretty well. Again, is it perfect? No. But does it get much closer than anything except for getting in front of those real people and asking them questions? I personally think so.
I did not expect going into this conversation that the biggest practical learning would be to use, you know, a chat GPT instance instead of a real human for research from a Microsoft researcher, just to, you know, confirm the credibility of this resource. Like I said, I'm not saying that in that specifically, I was not saying that it replaces the human.
I think that it is not even at parity at this point with a human, but I actually do think we're getting very close to it. And I also believe that there are situations where people will not tell you something, but that these systems will identify it. There's times when humans themselves will not actually share something because it's vulnerable. I mean, it's the classic, I know a researcher that's at large adult content website, we'll call it.
And they say that they struggle to recruit participants. And the only participants that they recruit, they're like super users. They're happy to talk about it because it's such a lifestyle for them. And so the rest of the people struggle to talk about it because it's kind of a taboo sort of thing. They don't really want to engage with that.
But you go into these systems, again, a lot of these people have written about it. Now, granted, it might be anonymously and things like that. But that content, people talk about what they like and what they're interested in sharing. They do that on the internet. It's been ingested. So that could actually provide a clearer path to answering certain questions than trying to get that out of a human.
As we're wrapping up today's conversation, what's the danger for the organization not to adopt the impact mindset? So the ultimate fourth component is to adopt that experimental mindset and evidence-based decisions and everything. What happens if you don't? What are the dangers? The danger is, and it's something that may be an experience that you as the listener have gone through,
and I've seen it too many times, is you're sitting with your team, you're looking at your dashboards, and things are trending red. And everybody is asking themselves, well, how did we get here? Everything was growing well. It was looking good. But now our growth has slowed and our retention is dropping. People are churning. What happened? And how do we stop this?
That's the worst time to ask the question, well, is our product working? Yet that's where the majority of the companies I work with begin to ask that question because they thought it was working because it was growing. They thought it was working because people said that they liked it. They never actually measured, was it working? And so the concern that teams face is if you don't start asking that question early, you're
By the time that you actually see the problem of building something that doesn't work and therefore has very low switching costs, so as soon as users realize that it's not doing what they hoped that it was for them, they're going to go and find the new greatest solution to try to do it. That is too late to then quickly turn around and build something that works.
If you begin to ask the question, how do we know this product is actually solving user outcomes from the outset during the design phase, when you're still prototyping, when you're still building something, you will ensure that what you create and send out into the market is actually effective at whatever it says that it is. And from there,
You will see success across the board when it comes to adoption, as long as people do actually want to use it, and then retention because people will be hooked onto that platform since it's actually doing what it's supposed to. And I'll give this quick example of just look at Duolingo. I think Duolingo is one of the greatest examples of this. And they have continued to grow and stay dominant even as this generative AI revolution has happened.
And you can do a decent amount of tutoring through ChatGPT now, but Duolingo still has a very strong hold on that market because they built something that works. There are academic studies that show if you use Duolingo for a certain period of time, it's equivalent to college-level courses. So it does what it intends to with the user outcome of learning a new language.
but they built it to be fun. It's filled with gamification. Now, do they go far at times and actually include some dark patterns and things like that? Yeah, they push the boundary, but they made a fun experience that keeps people on the platform so that they keep doing what they want to do, which is learning a new language and their business success shows. I mean, they continue to grow and have a really strong brand now. So I would give that as example of why
If they had built something that didn't work, they would never be where they are today. There's a lot of other failed language learning companies that ultimately were not able to drive success because they probably asked the question, is this actually working way too late in the game? Do you think Duolingo is going to adopt AI?
There you have, to my understanding, they're using it for some of their content creation. And I don't know anybody. So this is all public information and speculation. So just to be totally clear. I asked you for a speculative answer. Yeah, I believe they're using it for content creation. And I believe they either have a bot already launched or they're about to launch a bot. As we're wrapping up, one do and one don't. For our listeners, we want to adopt the impact mindset.
So do start small. You listening to this show, if this excites you, you can start this. It's as simple as that. You can be the catalyst of change. Start by just asking your team, why do we think this product works?
It's as simple as that. Just ask that question a couple of times. And if you don't get a solid answer, well, now you have an opportunity to go and try to create some evidence that shows that it does or does not work. You can be the catalyst of change.
That is as simple as a grassroots movement where one person begins to pull the threads, which causes more people to ask the questions, which causes more people to get involved, and it actually begins that cascade. So do start small. You can be the start of the change. Don't.
Don't think that usage is not important because it is. And that's the most common complaint that I hear from people is, are you, you're saying usage isn't important. And I'm not saying that, but don't over value usage because it is an important metric. It's almost like your speedometer. I like to think about it like that. Your speedometer is pretty useful. It tells you how fast you're driving. And that's a really important metric while you're driving.
But it's not the only thing. If you don't look at the gas gauge, you don't know if you're going to run out of gas. For a manual car, if you don't look at the RPMs, you might be blowing your engine out. There's a lot of other metrics that exist that tell you if the whole picture of the driving experience is actually working. And that's the same thing with product development. And so don't
Leave this conversation thinking usage is not important, but also don't continue to think that it is the only metric that matters. Thanks so much for sharing your wisdom with us today. Now tell us where can people get your book and where can they find you online? You can find the book Bridging Intention to Impact at Amazon. You can find it on the publisher's website, Peach Pit.
You can also find it at Barnes and Nobles. There's a couple of other booksellers out there too around the world. If you have an O'Reilly license too, it's available on O'Reilly. And then you can find me. I run a company called Desired Outcome Labs. So if you like what you hear and you want to have a discussion about how to actually get into the weeds and implement the stuff, feel free to reach out to me through Desired Outcome Labs.
I'm on LinkedIn a lot. That's another great way to connect with me. I write about this sort of stuff and post a lot of my talks in the store on there. And then in 2025, I'm launching a website, personal website. So if you just, by then, hopefully if you search Connor Joyce, you'll see my name and a website for him. You also mentioned that you'll be doing a bit of traveling in 2025. So if folks want to see you in person, they may as well scan some
European conferences, right? This is true. Yeah. Yeah. So on the America side, I'm going to be speaking at product world in February and doing a West coast tour. So if you're in America, I'll be in California, Washington, and potentially Oregon in February.
In Europe, I'm going to be speaking at a couple of conferences that will be on my speaker website and across the board as they get launched. But yeah, that's going to be in May of this year. I'll be doing a European tour. And I'll be at a collection of product tank meetups. So if you're part of the product tank network.
Hopefully we'll be seeing each other. And otherwise, if there are any gigs, I'm always open to meeting for meetups and stuff. So yeah, whether that be in America or Europe, feel free to reach out and would love to join a bunch of our collection of product minded folks. So always love those groups. Awesome. Thanks so much once again and have a wonderful rest of your week. Thanks you too, Jane. This is great.
Thanks for listening. You can find a written recap for this episode at usaless.com slash podcast. Please help us grow by leaving a review on iTunes.