888: Teams of Agents: The Next Frontier in AI Collaboration, with Mike Pell
31:32 Share

This is episode number 888 with Mike Pell, director of the Microsoft Garage. Welcome back to the Super Data Science Podcast. I am your host, John Krohn. Today's episode features the highlights of a session I hosted a couple weeks ago in Brooklyn at the inaugural AI and Creativity Summit, which was run by Artist and the Machine.

It was an excellent full day event on a gloriously sunny day. My guest for an onstage conversation in front of a live audience was the extraordinary Mike Pell. Mike is the inventor of the PDF and Adobe Acrobat. He's director of the Microsoft Garage, a global innovation program. He holds over 20 US patents and is the author of three books. Today's episode is entertaining, optimistic, and forward-looking and will be of interest to any listener to this podcast.

In it, Mike details why AI agents are an exoskeleton that gives you capabilities you never had time to master. He talks about the coming shift from passive to proactive AI that will interject like a trusted coworker. And he talks about why he believes we're getting to the good part in the AI revolution and what that means for the future of work.

Note that near the end of the episode, Mike and I take audience questions, but the audience members didn't have a microphone, so we'll skip the dead air of them asking questions and jump right to Mike or me repeating back the question for you guys.

All right, now let's jump right into our conversation. Here we go. I've been at Microsoft for 23 years. The Garage, the last 10 of those. Don't tell anybody I've been there that long. It's just embarrassing. And so the Garage is described as a program that drives Microsoft's innovation culture. How do you cultivate that mindset among employees on a day-to-day basis? For example, are there particular practices that

that our audience could hear about that allows them to have a more innovative workplace themselves? - It's funny you say that, John, because I think most people want to do innovative things. Your job or your role may not

call for that, but we find that people really to the person want to do something more than maybe what they're doing. And so naturally we're all drawn to the ability to jump in and do something new and exciting. At Microsoft we figured out about 10 years ago when Satya Nadella became the new CEO that we could create a mechanism and sort of a way for people to do this as part of their day job without it seeing special.

Right, so we, that was our thing. We could never do whatever, 20% time or you get a day a week to do whatever you want because we're all too busy. What we figured out was if we tried to change the culture to be more curious and more experimental, that would bake it into everything we did. It would be in the water. And so it wouldn't be like a strange alien thing to do. It would just be what we all did. And that's what I think all of you could sort of practice yourselves.

But practically, how do you do that? It seems like an easy thing to say. Yeah, we'll have a curiosity culture. But how do you put signs up that say, be curious? No, actually, you don't put signs up. You tell people not to ask permission.

Which sounds terrible, right? Attorneys don't like that. If you ask somebody, can I innovate? Can I do something new? Of course the answer is going to be no. But you literally have to tell people, just do it. Just do it. Prove that it either works or it doesn't work. Bring me some data back that shows me that proves your point so that we can continue doing this. But it's not something where you can just say, hey everybody, let's all be creative because that's never going to work, right? But

But to the person, if you make a connection, you say, I know that you're very talented with this. I know you have a great idea over here. People will go off and do it. All right. So let's talk about some now some specific innovations that are making a lot of buzz. Enterprise agentic AI. So this year, 2025, seems to be the year of agentic AI. You had an interesting perspective to me that you relayed me last week.

around how AI agents are like an exoskeleton available to us at all times so that we can do things we could never have done before. Do you want to elaborate on that more? Sure. I think in general, AI is a superpower for people, meaning it's going to allow you to do things that maybe you would never have time to do, you didn't know how to do, or you really...

even given the time, could never really master. So to me, this is sort of an exoskeleton, like an Iron Man suit that you put on that gives you these capabilities without you having to dig very deep, which is the magic of it. Now the agent part is when you take some of those capabilities and package it up into something that feels like a trusted coworker or an advisor or someone who's very specialized.

So, you know, for us, artificial intelligence, all of you, I'm sure, have tried this out and did very interesting things, is the underlying technology, but the agent is the packaging that sort of feels like a person in a way. A quick audience question. How much has, it seems like maybe the mezzanine noise just dropped down, but was that making it, was all the noise up there making it difficult to hear us or it was okay? Okay, nice. I can see there's speakers all around you.

It seems to have quieted down a bit anyway, so I don't have to reprimand them. I think we're boring them. They all left. The mezzanine people? No, now they're so engaged listening to this. They opened the bar upstairs. All right, yeah, so great perspective there on AI agents being an exoskeleton. Let's now talk about human-AI collaboration more specifically today.

in the business place. So AI is increasingly becoming a co-worker in our daily jobs rather than just a tool. Microsoft has AI efforts that are branded like that. They're like co-pilot assistants that emphasize collaboration between humans and AI. So for example, the new Microsoft 365 co-pilot can join your meetings, it can summarize discussions, highlight action items as an active assistant, which sounds a lot like an agent.

freeing people from tedious tasks so they can focus on high value work. So yeah, I mean that wasn't really a question, but I do have a question that falls off of that. - Wait, I was gonna have a response. - Okay, go, go, go. - Okay, so the interesting part of what you just said to me is that yes, AI has a lot of different things,

The part that people don't realize about these models and these agents and these things, they're incredibly capable of so many things you don't realize. And it's not until you start asking it certain questions that you find out about this. It literally can act like an advisor or a consultant or an expert in a certain field. It's just people don't know that they can go that deep.

So the acronym GPT, the P is pre-trained, meaning it's got an incredible amount of knowledge. I know that you dig into this, but a lot of people just don't realize that there is so much to the model that you haven't even heard about. We haven't even scratched the surface of yet. And it feels like that expert that you talk to.

I made up my own question. Yeah, I'm so glad you had a response. It's perfect. But I did have kind of a follow-up question from that intro around human-AI collaboration, which is that, so with many people talking about AI co-pilots now, Satya Nadella, Microsoft CEO, even said that every employee will have an AI co-pilot that knows how they work. So how do you envision AI as a collaborator

on human workforces, like in practical terms. What does it look like when humans and AI work side by side on lots of projects and problems?

The shift from where we are now, which is a bit of a passive model, right? The AI just sort of sits there and it waits for you to ask it a question or to tell it to do something. The shift that we're going to encounter very quickly is that that will flip and they'll start to become more proactive. So as you're working, as you're talking, as you're doing different things, it'll actually not interrupt but interject and say, hey, Mike,

Did you really mean this part or were you thinking about what we did last week? So the whole, the polarity is going to change for sitting in the background just waiting to actually being an active participant, being a co-worker with you. And that's what the big change is going to feel like. You're going to feel like you actually got an incredibly smart co-worker who you can trust. And the trust part is the most important part.

Nice, yeah, I think as error rates go down, you can trust more and more. So when I'm using deep research from OpenAI, for example, it is unbelievable. So when you have that level of accuracy, that level of trust, the more tasks you can use it in, it certainly increases.

And as always, the trust is earned, right? It's got to do the right thing enough times that you literally start to trust it. Which is, by the way, the downfall of humanity is that we start to trust technology and we stop paying attention, right? And when we stop paying attention, that's when problems come out. So I would just ask all of you, as you're using all these new technologies, do not fall asleep at the wheel. Don't trust it so much that you don't verify. Classic KGB situation.

Trust will verify. So, yeah, right now, in terms of specific examples that we have AI helping out with, the kinds of things that we trust them with are helping with code, so like GitHub Copilot.

writing and summarizing text. - You know, I'll just tell you really quickly, GitHub Copilot almost didn't happen because the very first versions, they weren't perfect, right? And you probably know programmers, engineers are very particular. And so when there's a few syntax errors, someone looked at it and went, "Oh, this is never gonna work.

It's like stupid syntax error. That was the very beginning. If they had stopped there, we would never have what we have now, which is you basically can tell the agent to go code something up for you, and it's gonna do a pretty damn good job. But it's funny that we're so particular and we're so quick to judge early mistakes that we sort of lost sight of the bigger picture. - Nice, that's an interesting story. I did not know that.

But yeah, so where I was going with that is that, so things like GitHub Copilot for writing code, we have LLMs for writing and summarizing text, even generating images, video. From your view at the Microsoft Garage, what are the most exciting or impactful instances of human AI collaboration to come?

So this happens quite a bit for us in the garage. My job is very fun. I get to work with the world's most successful companies and disrupt their thinking. And the way that I do that is by showing them how Microsoft Copilot AI can actually accelerate parts of what they have spent a long time doing.

And it's just unexpected because it's so simple. It's like things that they do all the time. You have to do some market research. You need to figure out what your competitors are doing. You need to figure out a good market strategy. We have people that do great jobs at all of this.

But the time that it takes for a model to do that, you could go from something taking three days to it taking 30 seconds, which is absolutely shocking to them because they just didn't know. And so that's the biggest disruption. That's the biggest change we see is that people's business models, their processes, the systems, the way that they do the rhythm of business has been changed if they would just sort of know that this exists and then they start to use it and experiment with it.

So that gives kind of a sense of that it's underutilized. But is there something that you think maybe we can't do today that we will be able to do soon? Yeah, it's like what's around the corner. The thing that is most obvious is all of you have heard about agentic AI. So to me, when you hear agent or agentic, think task, the word task.

Being able to chain tasks together is sort of the current state of the art of where we are with AI implementations and working. So the thing that we can't do today is sometimes give these agents complex tasks that are completely ambiguous and have them complete them without a lot of input from us.

that's what's gonna be around the corner. Like right now, yes, through reinforcement learning, by telling it what it didn't do right, it can correct itself and sort of maybe find a way forward, maybe not. But in the future, teams of agents working together will help figure out these problems. And it's the teams of agents that is the thing that we're not anywhere close to right now. Like yes, individual agents can talk to other agents, but we're not seeing this swarming behavior of agent armies working together.

Yeah, it is developing. I mean, there's some open source frameworks for it that are taking off like crew AI. But I think this problem of if you have a, even just a 1% error rate,

on decisions that are being made with a single agent. If you had 10 agents working together, then you have to multiply those probabilities of some kind of issue arising through that multi-agent system. Or it goes the other way. We just don't know yet. It could reduce the error rate.

If the agents are looking for that. I guess you could set it up deliberately to have an agent that's reviewing. Yeah. That's the orchestration, right? And the oversight and all the other things that we're trying to build into these systems. Nice. Yes. Really good point there. That's why you pay me so much. So for professionals who aren't AI experts but are starting to have AI tools in their workflow...

What's your advice to collaborate effectively with AI? How can people take the best advantage of AI assistants or AI agents that are available today, particularly maybe if you're not a coder yourself? The first thing that I do is tell people, you do not have to take a class. People ask me all the time, what class should I take to learn about AI? And I tell them, don't take any class. Just talk to it like it's a person.

Pick your favorite one, whether it's Copilot or ChatGPT or Cloud, doesn't matter. Talk to it like a person. Ask it something you would ask one of your friends or a family or a co-worker. Start a conversation. That's how you're going to learn to use it effectively because then you'll start to figure out what types of questions are most effective to get work done, like work, whatever that work happens to be. The next advice I give them is don't try to use it for a big project. Use it for something small.

Get little wins, build on being able to do something quickly, and eventually you'll get enough expertise or enough, I would say, trust and confidence to then start to talk to your teammates about what they do.

And that's when the multiplier effect comes in. When you actually can work with someone else, not just you and the AI, but you, your assistant, and other assistants together, that's when you start to realize the power of using this in a team setting. Nice, very cool.

All right. So shifting gears here a little bit, we've been talking in my last few questions about human AI collaboration. I would now love to pick your brain, not just about the future of AI and how we can be working with it, but how work will be transformed by AI, human work. So

You highlighted for me in a conversation that we had last week the importance of soft skills like creativity relative to automatable hard skills. And you had the specific example of how Italian students, I don't know if this is all Italian students, but I guess Italian students frequently have oral exams. And you said that many American students would fail that exam.

But AI could be part of the solution. Do you want to elaborate on this? Yeah, absolutely. What we're seeing, I'll just generalize in corporate America, we're seeing students come out of school without some of the most basic of skills that you would expect. The ability to communicate clearly, to debate, to critically think about things, to be able to operate in the moment.

These are really smart kids, right? They are so much smarter than the previous generation. But these skills that used to be called the humanities are sort of missing from what they're bringing to their jobs. So what we're seeing is that we need the ability for students, for people early in their careers to master these soft skills, like being able to talk and communicate and have conversations, because the hard skill part of their role is going to be automated by the machine.

The engineering, the medicine, the biology, all the hard skill stuff will be available through the system.

What's missing is their business savvy and their ability to actually be a really valuable person. So the future of work that I'm seeing is not relying on using AI to get better at the hard skill, because that's going to be sort of a given. Use AI to become better at being able to be a great teammate, to be a great coach, to be a great collaborator.

That's sort of the, you know, that's the future of everybody's job. It's not just a few people. Every single one of us are going to have to get great at that because, trust me, parts of what you do will be automated in some way. You know, but that's not the important part. The important part is that you're still a person. You're super smart, right? You are more clever than the machine is ever going to be. That's what we want out of you and that's what we need out of the future work.

Nice. Building on that a little bit and how AI is reshaping the tasks we do, taking on more and more tasks at work, a recent Microsoft Work Trend Index report... That came out yesterday. Oh, really? Yeah.

I got this stat a week ago, but I found this stat that 75% of global knowledge workers already use generative AI at work in some form. So leaders...

business leaders are grappling with how to transform this individual experimentation into organizational change. Satya Nadella in his Ignite 2024 keynote highlighted that co-pilot is the user interface for AI. It's an organizing layer for how work gets done and every employee will have a co-pilot. So this suggests a future where AI is deeply woven into our workflows,

So now I have a question or two for you that delve into how AI might change jobs, workflows, and the skills we need. So according to Microsoft's research, 2024 is the year AI at work gets real, quote unquote.

From your vantage point, how is AI fundamentally changing the nature of work? For example, do you foresee AI being as ubiquitous in our daily work lives as the Internet or PCs became? And what does that transition look like? So I'll start with the last point first. It already is.

People don't realize that machine intelligence, you know, whatever you want, reinforcement learning, all these underlying things within AI have already been in your phone for years. They're how Netflix knows what you want to watch next. It's how Amazon knows what you want to buy next. It's already in the water, right? It's already out there. People have been using it for years and years. We just didn't call it AI.

Now we talk about it. A year from now, nobody's gonna talk about AI the way we're talking about it now. It's just gonna be the way things are. So on the last point, it's already done. It's already over.

Going back to how does this impact individual work, in the same way, AI is built into all of the tools that you already use, whether it's the Microsoft Office suite, the Adobe Creative Suite, your financial software you're using, your project management, all of it has an underlying layer of some kind of machine intelligence or something's going on that is helping you to do your job. The program you're using may not be obvious about telling you that, but that's what's happening in the background. So in some ways,

the infrastructure and systems we use already are already doing this. So we don't have to worry about that. But you consciously wanting to use it or wanting to go faster, that's where you just have to be willing to want to push and experiment. So Copilot, the reason Satya said Copilot is the UI for AI is what he's saying is it's the interface. It's how you access artificial intelligence. You basically just talk to it.

I was showing you on my phone. I just pull my phone out and I just talk to Copilot. Hey, Copilot, what do you think about blah, blah, blah, blah? And it'll just talk to me like a person. So the interface is the way that you communicate with AI, which has now gotten so easy that anybody can do it because it's just a conversation. Nice, nice, great answer. Crystal clear. And I'm glad that you just debunked my question right off the bat. That's my job. It's a silly question.

Now in retrospect, I wish I could take it back. So I have one final question for you. No, we're not done yet, are we? No, because I'm going to open up to the audience. So I'm saying this is my last question so that their subconscious brains can start thinking of questions that they might have and let them bubble up into the conscious realm. So my final question builds on your envisioneer title.

Paint us a picture of the workplace, say, five or ten years from now. What does a normal workday look like? Yeah, I mean, I guess, like, I could keep going, but that's probably, you know, that's the key question. I would say that it's not dramatically different than a workday today, just from a framing perspective. Like, we all have to make money somehow. The way we make money, the way we do our jobs will be different in that these systems...

we'll be doing a lot of things that are tedious and boring and time-consuming. So that frees us up, as we talked earlier, to use this amazing Iron Man exoskeleton to be greater at what we do. So I think what you're going to find in the workplace is that we're doing things we can't even imagine today in terms of impact and being cool and being able to do these parts of science and medicine that were just unimaginable.

That will be the future of work, like this crazy stuff that we're not even to yet. But the way that we go about it will be what we talked about earlier. The system will actively be your teammate. It will be your mentor. It will be your boss. It will be your coworker, your person that works for you. Your torrid workplace romance.

I'm just saying, you know, we will not even think about the fact that our coworkers and bosses and subordinates are our agents. It's just going to be the way it is. But the nature of what we're doing is going to be magical. And I'm an optimist, so I have a very rosy view of the future. But I do think we're getting to the good part across the board, whether it's medicine, engineering, social sciences, you know, all of it. Nice. I love it. I'm an optimist, too.

I think it's going to be a great future. But let's see what the audience thinks. Is the poop going to hit the fan? What are your opinions or questions you have for Mike here, the world leader in seeing where we're going with AI?

We just talked about this last night. Do you want to just quickly repeat the question back? Sure. I think the question was, there's so much going on in the world of AI, how do you keep track of what's what? How do you know what's best at this or that? Is that a good paraphrase? So we literally just talked about this last night, and I have the same problem. We all have the same problem. Never in the history of humanity has there been so much innovation happening so quickly on a daily basis.

So we were joking, half joking, that we should create an agent to keep track of what's happening. That would go basically, you know, source, crowd source, what is the leading sentiment today or right now for what's best for doing, like, hey, what AI model is best for coding right now? What AI model can make me a full-length feature film? What AI model can get me into Harvard?

Because it's going to change hour to hour, day to day. And so nobody can keep track of all this. But I think by crowdsourcing the sentiment out there in people's posts and discussions, maybe we can get close. There's also, for particular tasks, there may exist already a benchmark that you could take advantage of.

or for kind of general capabilities, there's LMSYS, which that's been renamed now to something better, but it's LMSYS, L-M-S-Y-S. If you Google that, you'll still get to the same... You mean Bing? Bing? So it's a leaderboard arena, or it's an arena leaderboard where people go and just...

is response A or response B better? So humans do that evaluation just for fun. And response A and response B were created by two different leading open source or proprietary models. And that gives kind of like an overall course point. What do you mean Bing? Bing search. You said Google that. Oh, yeah, sorry. I've got to get paid this week. Yeah.

I forgot what the leading search engine was for a second. So...

People use that verb all the time. People are always binging things. I say that constantly. As an interesting, just quick little anecdote. So when ChatGPT burst onto the scene two years ago, Microsoft took our search engine, Bing, and integrated the OpenAI model behind it. It was an experiment to see could it enhance, could this generative AI enhance search models? What happened was people...

people started using that sort of AI-powered search instead of search. And to this day, that's what I do. I never go to Google or Bing. I just go to Copilot and I just ask it what I would ask a search engine because I'm going to get an answer like a person would answer me instead of pages of links. So is it something maybe kind of like perplexity?

I don't know. I use what I know. Yeah, so the question is, with AI being the biggest thing since the electricity industrial revolution, some jobs will become less viable or less lucrative. And so, yeah, what can we do? What should we do? What will happen? Envision. Envision.

So it's true. Probably about six or seven years ago, Microsoft published a book called The Future Computed that talked about this whole thing. This was six years ago. If you look back through history, this is something that's happened over and over again. Like, yes, this may be more acute because of the speed of transformation, but this has happened before.

Certain jobs are lost or reconstituted in different ways. People have to learn new skills. We go in different directions. Different types of movements arise. That is exactly what's happening now. And so nobody can stick their head in the sand and think that this is not going to happen. It's happening. It's already happened.

So what we have to do now is figure out what is our path forward for us individually? What do we care about as people? What do we care about as families, as communities, as businesses? And let's focus on that. Let's not get wrapped up in the AI is taking my job. It's not useful. What's useful is what am I going to do about it?

What do I get to focus on now? How can I learn something different? How can I do something completely different? I always say, I was telling John the other day, I'm never going to be a great concept artist for cars. I love concept car sketching. As an artist, I think it's super cool. I'm never going to be that good. I'm never going to have enough time to learn how to do that. I can do that now without even trying. I just work with my AI assistant. There are so many aspects of hard work

I would say skills that can earn you money that you can use these models for to start earning money or building your future or a career in very different ways that were never possible before. So as an optimist, I see it as an enabler, not as a disabler. I am afraid we are out of time. But Mike may be hanging around after to be able to take more questions from other gentlemen and other colored blazers.

So fantastic. Thank you so much for joining this special live filmed episode of Super Data Science, the world's most popular data science podcast. It's been a treat to have you here. I just have to kind of imagine how the audience might be reacting to an episode, so it's nice to see heads bobbing and questions come out. Really appreciate it. I hope you enjoy the rest of the day here.

Thanks to Mike Pell and the audience for that fun, informative episode at the Artist and the Machine AI and Creativity Summit. I definitely recommend checking out Artist and the Machine events if you can. I think there will be one in Los Angeles coming up soon.

Later in 2025. Yeah. In today's episode, Mike covered how AI acts like an exoskeleton that enables people to perform tasks they wouldn't have time to master otherwise. While AI agents package these capabilities into entities that feel like trusted advisors or specialized coworkers.

He talked about how the future of AI assistance involves a shift from passive tools waiting for commands to proactive collaborators that interject with helpful suggestions during your workflow. And he explained how AI is already embedded in many everyday tools and will soon be so ubiquitous that people won't specifically talk about AI anymore. It will simply be how things work.

All right, I hope you enjoyed today's episode. To be sure not to miss any of our exciting upcoming episodes, subscribe to this podcast if you haven't already. But most importantly, I hope you'll just keep on listening. Until next time, keep on rocking it out there. And I'm looking forward to enjoying another round of the Super Data Science Podcast with you very soon.