Deep Dive
Shownotes Transcript
As a company, we're really obsessed with customer outcomes. So if I can do something without using agents, I will, right? Ultimately, all that matters is our customers getting the results that they're hiring our digital workers for. Hi, everyone, and thanks for listening to the A16Z AI podcast. I'm Joe Schmidt, a partner here at Andreessen Horowitz. And today I'm talking to Prabhav Jain, the CTO of 11X. For those who don't know, 11X is building autonomous digital workers that automate go-to-market workflows, helping organizations increase efficiency and cut costs.
But what does it really mean for a product to be agentic? AI agents have become one of the most hyped concepts in tech today, often used interchangeably with automation, co-pilots, or even simple workflow tools. But true agentic systems go beyond static role following. They need to plan, reason, and adapt dynamically to new information. They don't just retrieve answers, they make decisions, improve over time, and operate in environments where outcomes aren't always binary.
In today's conversation, Prabhav and I dig into what it takes to build AI agents that work in the real world. We'll talk about where agents are already making an impact, where they still struggle, and how 11x has navigated some of the hardest technical and product decisions in this space. We'll also cover how the AI infrastructure landscape is evolving, how to balance orchestration versus true autonomy, and what the future of agentic experiences might look like. It's an exciting discussion that you'll hear after these disclosures.
As a reminder, please note that the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. For more details, please see a16z.com slash disclosures.
For Bob, thanks for being here, man. Thank you for inviting me. Yeah. All right. So maybe we'll start this with maybe a high level question. A lot of people talk about agentic experiences today. You know, it's really good marketing, but maybe I'd like to just take a step back and start by asking, what does an agentic product mean to you? That is a great question. I think the term agent and...
and sort of agentic experience is thrown around a ton. If you're using LLMs at any level, you're now an agent company, which means every company out there is an agent company. But for us, agents really have to plan, reason, reflect, think, get better over time. To me, that is the definition of a true agent. The agent problems that really sort of interest me are the ones where the answer isn't clear. There isn't a right answer. Even a human wouldn't have the right answer. What makes a
a piece of writing gut. That's really hard to quantify. That's what really excites me about agents. The user interface and the user experience really matters. Over the last 25 years, we have been trained to understand web and mobile interfaces. And now you have these agents coming up which aren't deterministic.
they change what they output every time you sort of call them. And that's a very new thing that we have to kind of train and explain to prospective customers that like, hey, this thing doesn't quite always work the same way. So it's a really interesting concept that's come up with agents now. Yeah, it's interesting, like not always working the same way. I guess the thing that I'm always tied to is like, what is effectiveness? And how do you think about these agentic products that you're building or maybe you've seen, are they more effective than humans? Or is it really just like, hey, this is orchestration and we're just making rules and trying to build things?
different experiences. You know, what I've seen is that especially in like code generation, I sort of count that as like true agentic companies because not only can they like generate code, plan what they're going to write, generate the code, then run it, see errors that come up and then keep going, right? And I think that part of the ecosystem is quite mature now. I think for like B2B, like verticalized agentic applications, it's quite a bit more complex, right? Again, you can run code and see if it like runs.
But for these applications, they're much more complex. Maybe say a little bit more about that. What makes these B2B applications more complicated and harder to solve for? There's a lot of people involved. So for us, for example, we saw two go-to-market teams. What makes good content? What makes the lead that you're reaching out to a true ICP lead for you? Again, just kind of subjective.
And so I think that's what makes it trickier, whereas for a piece of code, it kind of runs and does what you tell it to do. Makes a lot of sense. And I think this might be a good time to segue into talking a little bit more about the 11X's products, because that might give us a backdrop to talk about more of these experiences. So maybe give us a little bit of a sense of what 11X has built today. So at 11X, we're really building digital workers for revenue teams. And we like to think that our digital workers deliver human results. And so whether it's sales, rev ops, marketing, we kind of want to focus on these as our main customers.
We have two products in the market today, which is pretty rare for a company of our, I guess, earliness to have two products. But our first product is called Alice. Alice is our AI SDR. You know, she can engage prospects across multiple channels. She can drive qualified meetings. She can drive pipeline. And Alice does this today by working with data that's in the first-party data that's in your CRM. Alice can leverage our extensive data warehouse of third-party data
Alice can target visitors coming to your website. And so the goal here is to make it dead simple for go-to-market teams to run these campaigns on autopilot. And under the hood, there's a lot of complexity. We're kind of abstracting away from that customer. You know, how do you find the right lead? How do you do extensive, deep research?
on both the lead and the company they're at. How do you scale the actual outreach across a bunch of channels? How do you make it super personalized? How do you make sure it converts? And to our customers, they don't see any of that complexity, right? Because that's kind of our job. When we're building these companies, when we're building this agentech company. Mike is our second product.
it's doing a lot of the same things that Alice is doing, but through the channel of voice. So it's our voice agent. We've focused and designed it for inbound and consented outbound use cases. Mike can operate across a bunch of languages 24-7. It integrates deeply with our customers' workflows, with their CRM, with the countering software. And it's really designed to superpower and supercharge their teams. And if we just...
step back to that first question for a second and think about a spectrum of fully agentic to fully orchestrated and maybe rules-based is maybe the opposite of agentic. Like, where do these products live today? And how do you think about where are those products
Should be, yes. You know, I think as a company, we're really obsessed with customer outcomes. So if I can do something without using agents, I will. Ultimately, all that matters is are our customers getting the results that they're hiring our digital workers for? And so what we tend to do is we tend to break down our agents into a set of tasks. Some of these tasks are agentic, some of these tasks are not agentic.
The key metric we always look at for this task, what is the best technology that's going to get the best outcomes? And so we're very purist about that. And so if you look at our agents, some part is just a complex orchestration, which is a complex problem itself. The other parts are actually agentic. So for example, when we write content, that's agentic.
When we do deep research, that's a Gen Tech. But when we are sending out an email on behalf of a customer, just normal stuff. Do what works. Do what works, exactly. If you think about the market, things have changed so fast over the last 12 months. Maybe walk us through the evolution of your two products and the 11x platform over that time period and the decisions that you've made to come from there to now. Yeah, totally. In 2022, GPD kind of took the world by storm.
It's like, wow, this thing can actually produce some pretty cool stuff. Our main use case at the time was like, hey, can we help our customer generate these emails, generate this content? And we were using basic prompting just like sort of every company out there. In the last, I would say, eight months, the world has completely changed. Now you have like agents
agent frameworks, monitoring, evaluation, multimodal models, reasoning models. Basically, the entire game has changed. I wouldn't say that took us by surprise, but just how quickly we had to move to keep up with the innovation was something that was a bit surprising. Ultimately, like I mentioned, we're very customer-oriented. So we're like, okay, these three new models came out this week. We have to ship all these other features. How do we balance both? And that was a tough problem because maybe one of the models that came out was going to increase ROI by 20%.
And then it's important for us to sort of test that. So we had to get into this muscle of very quickly testing things while also keeping our goals sort of focused on North Star for our customers, which is outcomes. For our second product, it's a voice-based product. That I would say in the last six months we've seen the most development in. It's a completely new set of challenges. How do you handle background noise? How do you handle turn-taking? How do you handle latency? How do you make the speech sound natural? There's all these things that we do as humans that's actually quite hard for agents to understand. I mean, even something as simple as talking through your AirPods.
The speech that you actually get back on the other side is very different from if you just pick up your phone and talk to it. And so there's all these things that we hadn't quite anticipated until we started building our voice agents. And that was in the very early days. People were like, okay, I want to build a voice agent. I'm going to start off with something that takes speech and converts it to text. Then I'm going to take that text and throw it into a GPT-4 or a Lama model or a cloud model. And then at the end of it, I'm going to take the text I get out and put it through some speech synthesis. Yeah.
And that was kind of like what everyone was trying to clobber together, trying to juice, you know, 100 milliseconds of latency here, 100 milliseconds of latency here, try to make the experience supernatural. Now, I would say over the last probably like two months, we've seen a lot of advancements in voice-to-voice models or models that do both the speech or text and inference within the same model, within the same box to make it much, much more natural. And so I think we're kind of pushing the forefront of this right now with our agents. And one thing we've noticed is that for the agents that we sell, that we're selling for our customers,
You have to get three things right. One, it has to be intelligent. If the end user says something, this agent has to be able to think on its feet.
Two, the speech has to sound natural. And three, it has to be quite accurate. And one thing we've seen is that for most companies building voice agents, they probably have to get like two or one of these three things right. We have to get all three right. So it's more of a deeper problem. But the good thing is no one's really competing in this space. No, totally. And if I think about the decisions and the platform shift that you guys have actually gone through to take advantage of a lot of this innovation space,
It's been a total whirlwind, right? I mean, I think you guys basically fully re-architected like almost everything. Maybe talk through a little bit about that decision to do the rebuild on initially the Alice product. I mean, even on the Mike product.
Why did you decide to do that? And then talk a little bit about how you structured the team and actually made this happen while also scaling really fast. Totally. I mean, this was hard. I won't lie, this was really hard. But ultimately, we realized that the new advancements that had come out would provide so much more value to customers. And if you always start from what can we do to deliver more outcomes for our customers, that is sort of our North Star. And like I said, we re-architected our entire product to be super agentic.
And we kind of had this thought exercise where we were like, okay, let's take our existing platform, let's project out a couple months. Would that deliver more results or complete rebuilding and project that out three months? And it was quite clear to us that would be a second approach. But at the same time, you have tons of customers on the first product who are actively using it and you're trying to rebuild
get parity, re-architect, and migrate, it's quite hard. And so what we ended up doing was we had the majority of our team working on the new product. And some small percentage of our team kind of like making sure the old product was working. Life support, having customer issues, trying not to build new features, right? Because we'd rather build into a new product. I would say that's quite painful a couple months. But huge kudos to 11x team. They sprinted.
yeah right and we got this out very quickly i think faster most people thought was like humanly possible and so that was super exciting for us and i think that those trade-offs are like really interesting to think about okay there's this product that you spent all this time on that has all these you know wonderful little features for whatever customer set and then there's this new thing that you're trying to build which will be better and so how did you think about like how to balance those trade-offs and even when did
actually rip the band-aid and rebuild? Like, how did you think about those decisions? You know, it was really interesting. Very early on, when we were rebuilding the net new product, we had created this prototype, right?
And then be sure to internally be sure to some prospective customers. And we were like, had so much confidence that this is going to be right. You know, for the first couple of months, were there going to be a lot of bugs? Of course. Were there going to be a lot of issues? Of course. But we knew that once we got past that hump, that this new product would generate a lot more value for our customers. It's really interesting. One of the things that we've started doing as a business, which no one in our space does, is during the sales process, we actually give our new product away for free for them to use. We're like, hey, we believe so much in our product. Here you go. Right.
Right. And no one else is doing that because most of their products out there require a lot of human support to actually get folks set up. But with this new product, we're like, boom, here's your login, go for it. And so that also gives a lot of confidence that we're sort of moving in the right direction. I love that. And we talked a little bit about that. We've touched on this rather of all the changes. You mentioned how many different model providers. There's the DeepSeek news, you know, a month ago.
How do you think about testing and iterating as a part of this? And, you know, different providers get better or worse depending on new releases. And there's new models that are coming out, it feels like, every month. How do you think about architecting the experience and then basically testing and like what you're doing on a day-to-day basis to understand how to build the right product? You know, I think the way we think about our agents is our agents perform a set of tasks.
Some models are optimized for some tasks, some models are not. Sometimes it's not even a gen take for one of those tasks. And I think we don't have any secret cows internally. So we're very open to trying out whatever is there that will be the most effective at that specific task. And that's what our customers are counting on us for. They want us to build the best agents that achieve these outcomes.
And so, you know, whether it's like writing content or tool calling or instruction following, each of these tasks are quite different per model. And we tried everything out there, right? And because we tried everything, we sort of designed
our tech stack to make it so it's really easy for us to swap in and out models. But when you're swapping models all the time, the things that you have to really get right is if you swap a model, is it actually better at scale? You can't just run a few examples, oh, it looks good, let's roll it out for everyone, right? Which in some tasks, it's easy because there's an objective answer. In other tasks, if I show you an email and say, hey, what do you think? Is this good? You might say it's great, I might say it's really bad. It's so dependent on the use case, and that is, I think, a challenge that many people in the industry face. If there is no subjective answer,
how do you know if it's good? And so that's where we have to have a bit of human in the loop, talk to our customers and make sure for their use case, it's actually good. And on that use case point, that's kind of interesting because you have so many different customers. I don't know how many customers we have at this point. And they're using it in so many different ways. Like on those subjective questions,
questions. Is there a process that you maybe you've built to try to think about? Is this good? Is this, you know, is this better or worse rather? Yeah. You know, one thing we always try to convince our customers is, you know, what they might consider good might not actually perform the best. Yeah.
And so we always try to convince our customers, hey, let us run this on autopilot for a bit, just so you get some data. Because oftentimes, we would see some customers trying to take our system and make it write the exact same thing that the SCRs are. At that point, you should just use SCRs, like why use our product, right? And oftentimes, it's things that are unintuitive that work really well. And so for us, we have so much data on what performs across different segments, whether the email is open, clicked, where someone's replies, whether that results in a book meeting. Maybe we send an email, the person visits your website,
Then to sign up for a demo. We have data on all of these things. And so now we can sort of leverage that rich data back and be like, hey, this is what works. And it's kind of funny, in the early days of Salesforce, people bought Salesforce to learn how Salesforce used Salesforce. And so part of what we have to sort of educate customers, this is how 11x, use 11x to grow and accelerate our growth. This is how you should be doing it as well. So it's interesting to think about Alice and Mike as these combination of so many different tools and the model providers you just touched on.
How many different products are even a part of Alice and Mike? And how do you stay on the cutting edge of all of those different infrastructure providers and the changing market and constantly testing and iterating? I'd love to hear you talk about that. Totally. If you look at the modern salesperson today, they use 20, 30 different tools as part of their sales stack. And part of our product is consolidating all those tools, all that complexity under the hood and abstracting that away from the customers. You're just focused on your own outcomes. Yeah.
And to do that, you know, not only do we have to integrate a lot of vendors, but we also have to build a pretty seamless data warehouse system, whether it's third-party data or first-party data, just to give our customers a ton of insights into what's actually happening. And that's one part of our stack. Another part is just there's a ton of vendors we work with for things like crawling infrastructure or hosting on-prem models to have low latency. And, you know, the way we think about this is we really care about velocity, right?
So if there's a vendor who's thought deeply about a problem and we can leverage their work and they can scale with us, we're very happy to use them. And internally we've built this muscle where we can leverage a vendor to do an integration in days or sometimes hours. And that's really good for us because now we can leverage this collective intelligence that sort of exists out there for companies that have gone really deep into a specific problem, just use that for our use cases and suddenly our evals are just shooting up and
And so that's been really cool to see. But at the same time, that means you're managing like a ton of integration under the hood. Our customers don't care if one of our providers goes down. They just want, they just care about their outcomes. And so we have a lot of redundancy and stuff into place to make sure and a lot of monitoring to make sure that, you know, they have a seamless experience, even though we're coordinating so many orchestrations under the hood.
And it's interesting because some of these vendors, if you're thinking about the SDR use case, some of these tools that the SDR use have been around forever. There's an established base of infrastructure providers for those tools. But then there's the stuff like web scraping that maybe is changing really fast or all the different parts of the voice stack that are changing really fast. So how do you think about this is a different type of testing and iterating and exploring? How do you think about those tools and how do you stay on the cutting edge? Because it's just incredible to me how fast
the pace at which our team is exploring these. Totally. You know, it's really funny. I think if you went to the last YC demo day, you probably saw 11X as a customer for such a large percentage of those companies. I think, and part of the reason is because we're sort of in touch with all these companies. We're like, hey, this is a problem we have. If you build it for us, we'll use it, we'll pay you.
Yeah. Right. And so I think it creates like this ecosystem where people are like, hey, if I want to get something tested, I want to go to 11x. And so we kind of have this distributed army of folks, you know, across building these cool things that we can leverage and be like, hey, this is good or this is bad for these reasons. And if they improve it, great, we'll use it. If they don't, then we won't. It's exhausting keeping up with all of the investments. But it's also really cool because
all these teams are experimenting with different ways, like things that people hadn't tried before. And so we get to learn from them and see ultimately what delivers outcomes for our customers. And that's been exciting. Another interesting aspect of this is, you know, we've grown so fast. Now we're building...
for scale, but we're also doing all this testing and iterating. So how do you think about that problem of, hey, we now have a ton of customers that expect a really high level of service. We are taking over parts of the sales motion or the entire sales motion in some cases. So how do you think about the scale problem and building for scale versus exploring all these different tools?
You know, that's a really tough question. I think the short answer is you have to just do both, right? You can't just focus on one, right? Customers are expecting a bunch of new features, they're expecting the product to scale with them, right? And so the way we sort of thought about it for our R&D and product teams is, hey, we will sort of think in themes on a weekly basis, right? Like on one week, we might be sort of talking through how we could accelerate some feature developments. Another week, we're like, okay, this perfect system is crumbling. We need to make sure we can go and scale it. Yeah.
And so every week we kind of try to make investments in both of these buckets. The amount of investment we make might change relatively week over week, but we kind of have to make progress on both because at the pace we're adding customers, every part of the system is going to be, oh crap, we didn't think we'd get this much scale this quickly. So we have to go on and spend time scaling that up. Maybe as you've scaled, you keep adding customers,
They're always asking for you to go in certain directions based on their business. And so how do you, on the one hand, listen to that demand, but also think about not letting that create product drift or scope drift on the product? You know, this is one of the things where we kind of have to stay true to our mission, right? We're trying to create these digital workers that deliver human results. Some feature requests move us away from these digital workers. They move us more towards traditional SaaS. And, you know, at some point we have to say, no, like we can't do that. But if
if our hypothesis is right, what we are building is enough of the painkiller that customers will still stay. And that gives us good confidence if they're like, oh, can you build this custom integration for us? And we're like, no. But here's all this other good stuff that you can use in our platform and they still stick around, that gives us good confidence. In general, it's really valuable to listen to
and pain points customers have and what their current solutions are. But if you were to just take their solutions at face value, our product would be like this behemoth of 100 different custom integrations. But we truly deeply listen to their problems and use that to inform our roadmap. Because ultimately, they have a problem. They're making buying decisions based on that problem. So we have to solve that problem.
Just maybe not in the same way that they think it should be solved. On that note, you know, you guys went multi-product so early. I think that would probably be against the advice of, you know, many senior product leaders and people that have gone before. Why did you make that decision and any learnings from that experience?
Like you mentioned, it was a very deliberate choice that we kind of made very early on. We knew that different sales strategies and motions work for different companies. For some, it's an outbound motion. For some, it's inbound. For some, it's phone calling. For some, it's email or SMS or another channel. We knew that different things work for different companies. And so we were like, how do we set up our company to be anti-fragile?
we need to be able to support all these different motions. It's a very hard problem to do that quickly. Within a year, we had multiple products in the market, both hitting pretty good scale. It was very difficult, but the way I think we handled that is we structured, we would have really small pods working on the next 0 to 1 bet. And the entire goal of that pod is get to PMF. You are a startup, imagine YCW is coming up in a month.
get to PMF, right? And we've built a platform. And so we want folks to leverage a platform. But in some cases, you know, again, the company isn't very old. We said, just do what it takes to get to PMF. Validate that what you are building is something customers actually want. Once you get there, then we can sort of put it into the rest of the machinery, design it for scale, all that good stuff. But in the beginning, anything you do that's not delivering customer value is a waste. Because you don't even know the product's going to work. And so we're very practical as a team where we're like, hey, we need to figure out how this works and then we'll invest in making it amazing. Yeah.
But first, just get the answers to those questions. There's a long way to go still. You know, there's so much still to build. But if you just pause for a moment and look back at all the decisions you made and all the product you've built,
What would you have changed if you knew what you know now and could go back and tell yourself back then? Yeah, it's a really interesting question. One of the things you asked me earlier about agents and I was like, design really matters. I think we had a designer tool. I think that's one thing I think we would have changed if I go back is have them very involved from day one of the process because it determines exactly how the user is experiencing your product.
a lot of times that actually changes what we build. The actual experience the user has sort of under the hood. And so, you know, one thing, and now we have, you know, great designers on our team, but if you started with that, it probably accelerated our development quite a bit. And maybe just to double click on that, like what are the design decisions maybe that you've now been able to make that you think you would have liked to have made earlier? I think it's just kind of interesting. Like it's so hard to sometimes like grok, like what is this agent doing? It's kind of invisible. It's completing work. Like,
Are there ways that you feel good about how people see that work being done? Yes, I think one good way is taking the user along for a journey. So basically it's, hey, what is our agent actually doing? Now if you go to our product, you can see our agent is like,
crawling these websites. It's like writing this research report and we showed that to you. So it's like the agent isn't just like doing something in the background and boom, here's some content. It's like here's step by step everything it's doing. And the design for that was really interesting, right? Because if you run that agent again, it might do slightly different things, right? But really taking the user along for that journey to be like, hey, this is what is happening at this step. This is what the agent is doing. It wrote this content because it found this information elsewhere. I think that really helps build trust with customers because otherwise they're like,
Why did you write that? If I think of a few other things that I think I would have changed, one is I think moving from London to SF earlier. This is like the mecca of AI agents. There will be a lot of Londoners that are very unhappy hearing this, but I agree with you. Keep going. There's just so much development happening here and we learn so much from our peers. Everyone is facing similar problems. And the rate at which we can accelerate by kind of
being in this mecca, having this collective intelligence across so many people trying so many different things, it's really sort of accelerated our development. So I think moving early would have accelerated our growth even more. I think the third thing is, I kind of alluded to this earlier, which is
a lot of the reason people buy 11X is to learn how we sort of accelerate our growth using 11X. And so I think being more opinionated about how to use our product, exactly what people should be doing instead of like, here's a blank slate, go. Yeah. Right? I think that would have helped us set customer expectations and gotten them more success even earlier. Yeah.
This is our opinionated view of how you should run a cold outbound or your warm outbound or your inbound phone calls or any of these things I think would have really level set with the customer like this is what you should be doing and this is what success looks like. A lot of times customers are like, what does success look like? Am I comparing it just to my human SEO? But sometimes it's not an apples to apples comparison. What if I'm running like a brand campaign on LevitX, right? It's really setting that from day one to be like, hey, this is what success is. This is how you should use a product. And that's the direction that we're moving towards now. Yeah.
this product is not straight up magic. It's like it is built for a certain type of use case. No, I think that's so right. Things are changing so quickly in the market and
AI tooling is constantly, you know, the whole landscape you just said, like it changed in the last six months for voice and it's going to change that much more in the next six months. How do you think about the people that are best equipped to solve these problems? And like, how does that inform the type of team that you're trying to build at 11x? So maybe walk us through how you're thinking about hiring and talent. You know, I think at the IC level, one of the things we really look at is one, the ability to move fast. Yeah.
As you said, the landscape is changing so quickly. We have to be able to react to that very quickly. The second piece is really, and this is regardless if it was an agent company or not, ownership is really important. Ultimately, we have this internal model, we will die for our customers' pipeline. We will do whatever it takes to make them successful. Because we'll only be successful if our customers are successful. And so really having that deep level of ownership, I will do whatever it takes to make our customers successful, is something we really look for. And the third thing is, it's a chaotic environment out there.
you've got to be able to handle chaos. A new model might come out, we might change a specific feature because of some new thing that has been unlocked. And so we have to be open. You could have been working on something for two weeks and actually, we're going to throw that away. Yeah, we're going to kill that now. Correct. And so I think not being married to the work you've done, but really staying focused on what will deliver more outcomes, I think is something that we really look for in folks. And I think at the leadership level, it's all the same things, but also the ability to both have vision and strategy
but also be able to go like heads down to execution. Because once you have the vision and strategy, now you just got to build it and execute it. Right? And that is super important. So for all of our leaders, I think we'll go scale up and down on a moment's notice. And that, I think,
And that's one of the reasons why we have so many former founders at 11X. Because they have had to do that so many times in the past. This has been an absolute blast. Thank you so much for coming in and sharing all your learnings. We love working with 11X. I love working with you. And, you know, I look forward to many more fun things like this in the future. Amazing. This is an amazing chat. And I'm really happy we're able to share some of the stuff that we've been doing. Thanks, Prabhakar. Awesome. Thank you.
Thanks for listening. We hope you learned something about what it takes to build AI agents that don't just automate tasks, but reason, adapt, and drive real-world outcomes. If you liked this episode or the podcast overall, please rate, review, and share with your networks.