Solving Real User Problems With Generative AI: Slack’s Jackie Rocca
33:22 Share

Today, we're airing an episode produced by our friends at the Modern CTO Podcast, who were kind enough to have me on recently as a guest. We talked about the rise of generative AI, what it means to be successful with technology, and some considerations for leaders to think about as they shepherd technology implementation efforts. Find the Modern CTO Podcast on Apple Podcast, Spotify, or wherever you get your podcast.

How can generative AI help us collaborate more effectively at work? Find out on today's episode. I'm Jackie Raka from Slack, and you're listening to Me, Myself, and AI. Welcome to Me, Myself, and AI, a podcast on artificial intelligence and business. Each episode, we introduce you to someone innovating with AI. I'm Sam Ransbotham, professor of analytics at Boston College. I'm also the AI and business strategy guest editor at MIT Sloan Management Review.

and I'm Sherwin Kodubande, senior partner with BCG and one of the leaders of our AI business. Together, MIT SMR and BCG have been researching and publishing on AI since 2017, interviewing hundreds of practitioners and surveying thousands of companies on what it takes to build and to deploy and scale AI capabilities and really transform the way organizations operate.

Hi, everyone. Today, Shervin and I are talking with Jackie Rocca, Vice President of Product at Slack. Jackie, thanks for taking the time to talk with us. Welcome. I'm so excited to be here. Let's get started. I think most of our listeners, many of our listeners likely use Slack, but let's fill everyone in. What is Slack?

Slack is a collaboration platform. So it started out as a really effective way to message and work with your colleagues. And it's really expanded over the years to include a lot of your work tools that are accessible from right within Slack.

So it's a great way to work with your team, with your colleagues. We actually have external functionality. You can work with customers or contractors. And we are really looking to make it the hub of all of your AI tools and bring in AI into that conversational interface. All right. You mentioned AI. Let's go. What's going to be AI related? I heard communication tool AI.

Where's AI in that? I lead our Slack AI product area. We are building native functionality. We are looking at ways that AI can be used to address top user problems that we've been facing for years. Whether you're overloaded with all the communication and tools that you have in Slack, or if you're just trying to find that knowledge across your enterprise, we are using AI to solve some of those top user pain points.

Because we are a collaboration platform, we're also a place that a lot of your AI tools are being built into Slack today. We actually have more than 10,000 AI-powered tools that are available through applications or third-party developers. All right, 10,000. Let's go through them all, right? Start at the top. A lot of them as well are private to specific organizations, but some of them are public, available to download today too.

That's quite interesting. Tell us more about generative AI. I have to imagine they're either doing a lot or thinking about doing a lot with generative AI. Can you tell us a little bit about that? Yeah, so we are very excited about what's happening in generative AI. First of all, we've been using AI and machine learning for many, many years. If you think about our search functionality or when you're recommended different channels to join. So that foundational element has been there for many, many years, like many products.

But as many of you know, the industry in the space with LLMs and generative AI has really taken off in the last few years. I was on parental leave when a lot of the things happened with ChatGPT, and I was so excited about how some of these capabilities could help us take new approaches to some of the user problems that we've been facing.

for such a long time. We're really taking the approach of starting with those user pain points and constantly trying to think through how does new advancements in technology help us take a fresh approach to solving those issues rather than starting with the technology itself and saying, oh, generative AI is out there. Let's throw the spaghetti at the wall and see what sticks. The features that we have launched and available today are around channel recaps,

thread summaries, and search answers. And let me talk through a few of those. So on channel recaps and thread summaries, this is really about getting up to speed quickly in channels. So often there's just so much conversation back and forth and whether you're away from Slack for an hour or a day, or you're coming back from vacation,

it can be overwhelming to catch up on that conversation. So what we are offering and what's available now for customers is the ability to summarize that content as you want it. Mm-hmm.

And then there's the ability to go back to the source as well to validate that or even dig in deeper if you want to get more information. So that's really around channel summaries and thread summaries as well. Another great use case that we're seeing is let's say there is some sort of technical incident. So maybe your service is down or having issues.

and every single moment really counts in those situations, maybe that subject matter expert just got paged in the middle of the night. Rather than have that person have to spend minutes and minutes catching up on everything that's happened so far, they can request a quick summary and jump into that conversation and contribute to the solution immediately. And also you can get a summary at the end and say this was the root cause and this is what the resolution was.

So it's been super valuable, so many different use cases. And then the second category is around search answers, especially for companies who've been on Slack for a long time. There's so much conversation in there. People talk about projects, about people, about topics, about company policies. If you think about all of that knowledge that really lives in Slack, how can we make it more accessible and easier to use?

So with Slack AI search answers, you can simply ask a question. You know, what is Project Gizmo? I haven't heard about that. Or maybe I don't want to ask my colleague or disrupt their work. You can just ask Slack AI to get up to speed on different topics.

You want to learn about company acronyms. So many of us have been in meetings at your company and somebody throws something out there. Or you're a new hire, and rather than ask your manager these 32 questions that you have, just ask Slack AI. It's sort of your trusted companion that you can rely on. Super valuable. Yes. The ones you mentioned are user-facing, as in I'm a Slack user.

I've been gone for some time or I, you know, on 50 different channels and a lot's going on and some are incidental to what I care about or some are really, really critical. And then somebody says, hey, why aren't you responding to my Slack? And I'm like, I don't know what channel are you or what are you talking about? And so I get that it's super valuable. Are there also use cases where the generative AI creates messages on my behalf?

Yeah, we are focused first on really keeping the human in the loop and making sure that you are always in control. We know that Slack can be a really sensitive place. There are a lot of private conversations. People are in a work context, so they really want to have control and it's really important how they show up.

Our experiences are very guided and situational, but we want to make sure that we are not creating and posting things on behalf of our users without them having really the agency to make that choice and send that message. Yeah, I love that. Well, you have so many users of your product. How do you go about introducing some features like this?

Yeah. And I'd love to talk through some of our core product principles because this has really guided us here. One of our product principles is don't make me think.

which in the generative AI space for us means Slack AI should not require our customers to be prompt engineers or experts in the field. It also means it shouldn't require training. So we really want to have situational and guided experiences. Let's surface some Slack AI answers for you when we think it's appropriate.

You know, this channel is really busy. Let's surface a summary that might be really valuable to you at that moment. We don't require you to figure out how to write that prompt so that you get that great answer. We're doing that work for you. A second principle that we've also really leaned on is prototype the path.

So, you know, there's a lot of potential with generative AI, but we know that accuracy and relevancy and frankly, latency as well are really important to the user experience. So we've been prototyping for the last several months so that we can, again, not make users think and we can do all that work for you to make sure that we are creating a great experience and we're shipping those experiences that we have high confidence in. And we're continuing to work on those things that maybe need a little bit more refinement.

How long does that take? Is that a short process, long process, 15 minutes, 20 minutes? More than 15 or 20 minutes. One of the big challenges as a product leader working in the generative AI space is balancing time to market with time.

getting that experience 100% right. We know that generative AI is not 100% right. So we need to kind of find that middle ground between, we know that there's a ton of customer value here and we feel great about the experience we're shipping, but we're going to continue to iterate and make it better over time. So for me personally, that's been a challenge because I always want to have things

as close to perfect as possible. But we know it's also important to give people that value sooner. So it's a balance for us. We have spent many months to try to get that experience the best it can be, both from the features that we've already launched, as well as new things that we are already prototyping to have as fast follows down the line. And I have to imagine the experimentation process

or the test and learn loop is quite fast with like all the users, right? I like your principles on human in the loop and all that. You mentioned it's a balancing act and I could imagine there's so many things you could be doing with AI and Gen AI, right? And so even if those principles are always followed, you could be incredibly ambitious, right?

in so much with Gen AI or mildly ambitious. And so how do you navigate that path? How far do you go? How fast do you go? It's a great question. And we debate this all the time as a team when we talk about prioritization and what to build and when to launch. For us, again, we're really trying to focus on what are those top user problems that we think generative AI can help fix.

And we're prioritizing those first. There are some table stakes features that we could absolutely and probably will build within Slack AI. So things around helping you compose messages. Slack primarily is a text-based tool, although we do have video capabilities through, we have a meetings product called Huddles.

but it's still primarily text-based. The top pain points are around how do I create focus and reduce some of the noise? How can I better find information and leverage the knowledge archive that I have in Slack? So we're really trying to put more of our energy towards solving those things that we've heard repeatedly from customers

And hopefully we'll have some opportunistic things as well, especially as we go throughout the year where we can also address things that are really cool and delightful that we certainly want to build into the experience, but maybe don't hit that very first set of features that we've launched. That's a great answer. Yeah, the generative part of that. I mean, I think that everyone probably their first blush is going to say, hey, can it type some text for me?

But I think that really isn't where Slack is competitively different in some ways. And what I mean by that is that you've talked about the history of being a communication channel. And when you hear communication channel, you think about typing messages, right? That's the production of messages. But there's so much organizational communication that's happening in these tools. And you've got that rich history of that. That strikes me as what's critical and differentiating here versus Slack.

Nothing wrong with generating tags, but the other part seems unique. Well, there might be some things wrong with that. Okay, you can push back. I accept that. Go ahead. Keep going. Well, I mean, I think that's the point that she's making, right? Because there is clearly...

A huge pain point on what's going on in the organization and what should I know and all that, which makes a ton of sense. But then there is something personal about Slack. And now you're going to have to wonder, is it Shervin that's typing it? Is it Shervin's agents that's typing it? And maybe at some point we will be in that world where, you know, we get messages and text and all that. And you wonder who wrote this. Is it the person or is it the person's agent? Right.

And I don't know. I mean, I think like that could be what's wrong with this is that at some point, then we just sort of

applicate the whole thing and let AI just do everything, which is not, I think, where we want to go. I think I was reacting to how nice the first use cases were, though, because actually, as a, I mean, I'll tell you, it will surprise nobody here that I'm an inbox zero kind of person. I don't like any little red alerts up there. I clear them as soon as I can. And that's actually, I've struggled with Slack because of that, because

At any point, there's so much coming through that I can't satisfy my inbox zero needs there without 100% attention. And what you're offering is a very different way of solving that pain point for me. So that seems very appealing. Absolutely. I mean, me personally, I can't imagine my life without some of the Slack AI features. And I have the privilege to be able to also have the things that we're still prototyping and haven't launched yet.

But the way that I consume information has changed. We think about this sometimes in different tiers of messages and channels. So today, everything has an equal priority. In general, a message is a message, whether you are in that channel with the CEO of your company or it's a social channel with your colleagues. We do offer channel sections, which is a great feature if folks haven't checked it out. It's a really nice way to organize your channels.

But can we do more there? Can we think through what is sort of a tier one channel or conversation? And those are things that you want to read every message for. And you probably won't get a summary unless maybe you're on a parental leave or gone for an extended period of time and there's just so much there. But there's a huge group of sort of these tier two conversations that you want to know what's happening and you want to stay in the loop, but you might not need to read every single message. And maybe you don't want it to show up

in Slack every single moment of the day, or it's just a really fast moving. I kind of just want the TLDR of what happened when I was out. So we're really trying to think about things in these different buckets of use cases and prioritization and help make that easier for our users to consume. - Hey Shervin, I just want you to know you're always tier one for me. I read everything you send, so I would never summarize you.

Well, I don't have to summarize you because your messages are so like terse and it feels like it's been written by a physician, not by a college professor.

All right. So actually, I was thinking about as you're talking about this talent mobility and changing workforce, and that seems like a big role here as well, that, you know, as we're having people leave jobs or retire, after having people shift jobs back and forth, how do you reconcile the need to sort of capture all that information with does information go with a person? How do you reflect that?

Yeah, one of the early premises of Slack was how it's different than email. So with email, if that person leaves the organization, that knowledge is really lost. With Slack, one of our founding perspectives was really that Slack should be this searchable log of all communication and knowledge. That's what Slack stands for. So we have this persistent place that it's really the organization's knowledge. You post something in a public channel,

You happen to leave the company, well, still the knowledge about what you had worked on in the project and the people and things like that can still persist and you can still get value out of that. Or if you're a new hire joining the company, you can still learn about what had happened, the learnings, the experts that you should connect with, etc. We actually think that building AI on top of that is a really great opportunity.

connection point to make those experiences more valuable because we have all that knowledge, all of that insight that we can pull on and just make it more valuable. It all already exists today, but maybe it's not fully tapped to its potential or it's not easy to make sense of or get you that quick answer that you're looking for. And we aim to make that experience better.

This all sounds amazing. What's hard about it? There's a lot that's hard about it. We know that generative AI is not perfect. So we have been spending a lot of time trying to get the best prompts that we can create on behalf of our users so we get the right

detail of information. We're not sharing an answer when we're not certain that we're providing sources back so people can reference the source content easily. So there's a lot of work to be done to make that experience better. We also are really laser focused about the user experience. So we have played around with how does the summary display itself? What's the correct entry point?

How do we make this feature and functionality obvious so when it's valuable to a user, they know it's there, but not so annoying that we're surfacing it all the time? For example, we talked about those tier one channels. Maybe summarization is not appropriate there. So we're continuing to do a lot of work to make sure that that user experience is great.

both from the actual how do you find Slack AI and when does it surface to you, but as well as the experience when you get that response back and make sure that the information is relevant, accurate, and well-sourced. You're leading the charge on AI and Gen AI in Slack. Tell us about the team. What's the team like? We actually formed this team through a group of people that was prototyping on the side of

Really just trying to build experiences in our spare time, a lot of nights and weekends, just really passionate about how this can help solve these core user problems using new technology. So it started there. And then as the year progressed, as we got more momentum behind these prototypes and got feedback internally and externally, we've actually put a more sustainable team structure in place and started hiring against the goals that we had set more officially.

But, you know, everyone is very excited about the future of AI. We've been able to attract and retain just an incredible team. Everything from obviously engineering, machine learning team, design, that user experience is so important.

Data science is critical to how we think about generative AI, as well as all of our go-to-market teams just really trying to hear customer feedback. How do we bring this to market? I don't know if you can answer this question, and if not, that's fine. But I'm curious what's under the hood and whether that's a pre-trained model that's sort of available, or is it you're training your own models given the uniqueness of the domain?

So what I will say is that the space is changing very fast and we're open to different options. And again, super...

super laser focused on what's best for our users. So we are evaluating pre-trained foundational models. We have things that are machine learning models that we've had in Slack for a long time. We are actually part of Salesforce for folks who might not know. And Salesforce has an amazing AI research team as well. And there's obviously open source models too. So we're evaluating everything. And I do expect this space to change over time. But

But we have taken a really deliberate approach to how we're thinking about data and trust. So we have built an architecture where...

Your Slack data does not leave Slack. Everything is hosted within our virtual private cloud, our VPC, so that when you're using AI in Slack, you can be confident that your data, your really sensitive company data is not leaving Slack's infrastructure and that we still abide by all of the compliance and enterprise-grade security capabilities that we have in Slack today. What's exciting that's coming next? What are the AI developments that you're pretty fired up about?

There's so much to be excited about, both at Slack and things that are happening outside of Slack with foundational models and other products. I can't wait to see sort of what the next year brings. We are continuing to iterate on different experiences in Slack.

We are constantly getting feedback from our customers, trying to stay rooted in user problems to help define our roadmap. But we're also listening to what's happening outside as well, because the space keeps changing, keeps evolving. And it's hard to predict one year to the next what's going to be possible, but we're also really trying to stay grounded. I'll be the first to admit that I don't have that crystal ball of how things will evolve in the next three to five years, but I'm really excited to be here for the ride.

Tell us a little bit about your background. How did you get interested in this? How did you get to be able to do these things? I've been at Slack for about five and a half years now, and it has been an amazing and fun ride. I led our product-led growth team for a long time, so our self-service group.

And just really, again, was passionate about this space. I've heard the same user problems or user needs be referenced time and time again, and was just sort of a self-learner to say, okay, we've got this new technology. Can we take a fresh approach? And I also did have the privilege of coming back from parental leave. So I could really kind of clear my head from the day-to-day and think a little bit bigger about the opportunity space and

Before Slack, I spent six plus years at Google, specifically on YouTube, and was fortunate to be on the launch team for YouTube TV. So I was a product manager over on YouTube TV, which was an incredible experience. When I joined, we were on the whiteboard trying to figure out, should this thing have three tabs or five tabs? And how do we think about what channels to put into the service?

So it was really exciting being part of that journey to launch YouTube TV and see its scale for a time. Before that, I got my MBA at Stanford. And then also I have my beginnings as a management consultant. So I started my career at Bain in Los Angeles.

It's been a ride. I think a lot of people who end up in product management have had this indirect path to get there. I don't think I knew what product management was when I was in college or in my 20s. So I'm really grateful to have found Slack, to have found product management, and more recently to help lead through this sort of AI revolution for the team and for the company. That's an amazing background. The self-learning part of that's really interesting, too, because I think

That's a shift that's happened over the course of even since we started this podcast, Shervin. At the beginning, we were talking about a machine learning model and very few people, I think, woke up on Saturday morning and played around with some machine learning model. I think practically everybody has gone and screwed around with a lot of the generative tools that are out there. And I think it's a fundamentally different way that people are learning about these technologies. And I think what you just said echoes a lot of that.

And I think there's really two sets of skill sets to think about. There certainly is the technology side, what's possible, what's unlocked. And it's important to understand. And hopefully, if that's not someone's expertise area, they can connect with others who maybe can work between strengths and weaknesses. But I think there's equally the opportunity to think about how can it be applied. So...

What would that user experience look like? What specifically are we trying to solve? Because I think it's exciting to just try a bunch of things or experiment with generative AI and what's happening with LLMs, which is great. And again, I'm not going to discourage people from playing around. But when you're really thinking about, okay, well, what am I going to build? What am I going to ship to customers?

The technology is part of it. That's kind of like the backbone. But I think what will really set different products and services apart is what value do they ultimately provide. Yeah, you need the left brain and the right brain and very cross-functional team. So now we have a segment called five questions. Did it tell you about that? No. Wonderful. That's how we like it. So I'm going to ask you five questions. Just give us the first thing that comes to your mind in rapid fire style. What do you see as the biggest opportunity for AI right now?

I think the biggest opportunity is finding true product market fit and solving real user problems. What is the biggest misconception about AI? Let's come back to that one. Okay. What was the first career you wanted? I wanted to be a doctor. Me too. When is there too much AI?

That's a great question. I think we are very far from too much AI. Perhaps that's when you can't tell if AI or a human made something and we've kind of gotten so far down the path and away from that human component that maybe that's a little bit too much AI. Aren't we there yet? I mean, already? I think we are in very, very early days. What is the one thing you wish AI could do right now that it can't?

I'm excited for AI to get more into the robotics space and actually take care of physical tasks for me. I am a new-ish mom and just there's so much stuff that burdens my life that I would prefer to have that time to either spend with my family or at work. So I'm excited for some of the physical, maybe robotics opportunities. I love that. A little bit less cerebral, a little bit more physical, please. Right? Yeah. All right. What is the biggest misconception about AI?

It's a tough question. I think the fact that any of us really know what the future looks like, but I think we're all trying to learn and be humble along the way. There's probably a misconception that we can create that 2030 product strategy and probably know exactly what the future brings. By the way, this is a hard question. Sam, what is your answer to this? The misconception thing? Yeah. I think my answer would be more sort of

People not understanding how limited these tools are. You know, it's sort of like, hey, I can do all these things and LLMs can do all these things. And it's just a Bayesian update of the probabilistic next word. And there's a lot less thinking behind it than there is just a pattern. I think that would be my misconception. Yeah.

Anyway. I think it's a hard question because I think there are very divergent perspectives out there today. And so there are so many opinions that it's hard to say that there is this perspective out there because I would argue that there isn't a clear –

consolidated, agreed upon sense of what the future looks like. So everyone's probably going to be a little bit wrong. That's an ill-posed question to begin with, right? It seems like it, right? Like I would say that it's going to replace humans. Like that's for me the biggest misconception. Although I know this is off topic, but like what you said about the probabilistic next letter or next word, it's not obvious that that's not how our brain works, by the way, right?

Yeah, we just could be doing a better job of it. I mean, it's not obvious. That's just not how our brain works either. It could be, yeah. A fun conversation is what kinds of things that humans do could AI not do, presumably do in the future? I think a lot of things around experiences, and Sharon, you mentioned, I think more of that human element hopefully makes us unique. And then I think it's those boundaries

big visionary things that happen that maybe aren't able to predict, like going to the moon or, you know, will AI kind of think through those experiences or just these big breakthroughs that happen that I want to say only humans can really dream up. Stroke of genius, right? Yeah, these strokes of genius, but maybe we'll be wrong. Yeah, we're not the probabilistic machines that Shervin thinks we are.

I've really enjoyed learning more about Slack today. I think increasingly people are using communication channels like Slack at work. And these tools like Slack really fit that distributed asynchronous work world that we're in. But calling it a communication tool is just not right and far too narrow. And I think you've really shown a lot of the opportunities that we have here. Thank you for taking the time to talk with us today. Thanks. Thank you so much. It was a pleasure. Thank you. Thanks for listening.

On our next episode, Shervin and I get some AI-fueled fashion advice from Jeff Cooper, Senior Director of Data Science at Stitch Fix. Please join us. Thanks for listening to Me, Myself, and AI. We believe, like you, that the conversation about AI implementation doesn't start and stop with this podcast. That's why we've created a group on LinkedIn specifically for listeners like you. It's called AI for Leaders. And if you join us, you can chat with show creators and hosts, ask your own questions, share your insights,

and gain access to valuable resources about AI implementation from MIT SMR and BCG, you can access it by visiting mitsmr.com forward slash AI for Leaders. We'll put that link in the show notes and we hope to see you there.