Should we stop talking about models?
32:45 Share

The question is, in three or four years, is it going to maintain a competitive barrier to entry? You know, I use this framework that's like, is the answer a puzzle or a mystery? Because I think we all try and be really smart and solve everything. And a puzzle is something you can think yourself through brute force, intellectual horsepower will get you. And a mystery is something that's only discovered by going on the journey to get to it. And I think, is the model going to be performant over time, best and like?

I think we're joking ourselves if we think that that's a puzzle. It's a mystery. In that world, I think it's a net new entrant who wins. So if it's not a chatbot and you have to have some sort of crazy product experience delivered, you know, that doesn't exist today, that we haven't seen, I think it's net new.

I think if it's a chat-like experience where the differentiation happens at the model layer, but the UI remains somewhat consistent with where we are or simple, and the end user cares about the frontier of capabilities, I think it's going to be OpenAI or similar.

And then I think every other scenario, it's going to go to the incumbent. And I think we're going to see Meta, Google, and Apple all just have a big piece of that market on their platforms. All right. Hi, everybody. Welcome back to Hallway Chat. I'm Fraser. Hi there. I'm Nabil. Welcome to Hallway Chat. I know you got a thing you want to jump into, but I am going to start with

Whether we should be doing this as a podcast or series of tweets, I don't know if you saw, but the venture firm USV posted this last week. Basically, I think they're recording their Monday partner meetings as their version of their intro podcast. We love that firm and they tend to talk about a lot of different things. And they just posted their version of a little hallway chat, what we talked about this week. And it's kind of a great little read. I don't know if you read it. Did you read it at all?

I didn't read it, no. No, I'll send it to you. There's things in there like, hey, this week we talked about how it's really important to digitally sign all types of content to maintain authenticity in the world of AI. We talked about the theory of like, won't be evil on the front end and can't be evil on the back end as a kind of like hybrid architecture to how to think about art. It was like, it was kind of cool. And it made me think like, maybe we shouldn't

Have everybody standing 40 minutes reading this when they can just read a tweet with headlines. I don't know. Make it all a little more efficient for everyone. It's the bifurcation of media consumption, right? You either want a two-hour rolling conversation or you want to have the terse summary, as terse of a summary as you can get.

I think that's probably right, actually. Yeah. Maybe the answer is both. Maybe the answer is we should have the two-hour version. We should have the three-second version. We could have the eight-second version and you just pick your medium and we'll have AI cut it 55 different ways.

Maybe. I can't wait to see it. I have tremendous respect for them. They're all smart. So kudos. Although if we do all the media stuff in all the different mediums, that means that eventually we'll have to show people the video of us actually talking instead of just audio. And I'm not sure if I'm comfortable with that just yet.

Let's jump in. You know, last time we ended because we had to go. And it was a moment in time when you were talking internally a lot about how we just should stop talking about the models when we discuss companies. And I thought it was an interesting conversation that we should pick up here. Like, where was that coming from? What did you mean? And why were you saying it? Yeah, I mean, the provocative thing I said internally was just like, how about if we just never talk about models ever again? Yeah. Yeah.

And there's a little bit of a prompting. And yeah, it came from this feeling that we personally, internally, and frankly, I think I've picked it up talking to founders as they're pitching as well. We may be focused on the wrong things. I mean, the core thing about a model is that it's an enabling technology. This is what we talk about. This world of generative AI is an enabling technology. And it reminds me a little bit of...

Back in the day, there used to be this adage that you would rather have an engineer from Georgia Tech or Waterloo than you would from MIT because the MIT CS grad who just graduated has only been taught a bunch of theoretical CS that's kind of like neat and wonderful and academic use and build technology for technology seg, but could care less about how that's going to be used in the real world. Whereas a more practical CS program, they're going to be builders and hackers, and that's what we want. We want builders and hackers.

It's kind of like two points. The first one is that, which is we need to focus on the end user. You know, the models are not even equally good at everything. And so the question is not which database you used or which model you used or anything like that. It's, is the end user going to be happy with this product and going to come back to you? And so then you earn the right to keep working with them. The second kind of root question is, is the model even important to the company at all?

And that is, is the model good for your particular use case? If you don't have a really clear use case, you have no way of evaluating whether the model is any good for that use case. And we can fandy about like, oh, it's a model for music or text or sound or images. But like,

We've seen like, that's not the answer. The answer is it, is it really good at rendering faces, you know, for deep fakes? Or is it really good at inflection of tones in speech? Because that's what the use case is. Like the answers are in the nuances of the way you use it and evaluate the product. Not because it's good at some like big 10,000 foot headline.

And then lastly, even if you think the model is kind of awesome and like soda, like state of the art right now, you still have to ask the question, is it going to be the best thing over time? Right. Right. Like, is it? I don't care. It's this early stage startup broken in 15 ways. The question is, in three or four years, is it going to maintain a competitive barrier to entry? You know, is the answer a puzzle or a mystery?

because I think we all try and be really smart and solve everything. And a puzzle is something you can think yourself through brute force, intellectual horsepower will get you. And a mystery is something that's only discovered by going on the journey to get to it. And I think, is the model going to be performant over time, best and like? I think that's, I think we're joking ourselves if we think that that's a puzzle.

It's a mystery. It's an unknowable thing because we've seen many, many times, you and me have several portfolio companies we work with that started out with models and are taking, now taking situations where like the open source is catching up

And they're totally fine with like, okay, we're going to flush the model down the toilet. Let's go use the open source model. There's no reason I should have the time and expense and energy to build a model internally when this other model is actually going to be cheaper, better, and I can work with outside partners to make it good. Because ultimately, my customers are still happy and I'm still working with them. And frankly, by working with them, I have now learned some earned insight that will help me build a better product next year that somebody else who's not in the market with this customer is not going to have. Those are the things that...

that matter. Everybody's super smart. Everybody's trying to answer a bunch of questions. I just think it's super important that we don't ask the wrong question. Because if you ask the wrong question to a founder or to each other, you will always still get an answer. And so it's just much better to root back to like, is this even a question worth answering? And what is the risk we're underwriting? And the risk we're underwriting is that can this enabling technology, whatever you happen to use, serve a customer need in the near term?

And then from there, there's too much fog of war to know what's going to happen after that. I agree with all of that, but certainly there's exceptions. Like foundation model companies are model companies. Like should you...

We're investors in Anthropic. I feel like we feel pretty good about that investment. Yeah, yeah. Well, what's different then, right? Why was there a moment when talking about models would have been the right thing in one market? And why do you think that talking about models in many of the other markets that we're seeing people talk about models doesn't make a lot of sense?

Because that's where this is coming from, right? It's like we then spent now, it's a couple of weeks ago, but we met a bunch of people over a compact number of weeks who were coming to us to talk about models. Yeah. And we were, as in your parlance earlier, like we were asking the wrong questions because ultimately you're like, we shouldn't even be talking about these models. The question is, what's going to happen for these customers? Yeah.

Yeah. And is the product market fit? Is there a customer that's happy? Can they serve those customers over time? Do they have product and think and taste in order to follow them on that journey? Are they earning insights by that contact with the customer? Those are the most important things. You have to make sure the tech is competent to be able to build stuff. They have to be able to ship code and velocity matters and all that stuff. Right.

So how are foundational models different? Well, it's pretty simple. They're foundational models. Like, that sounds simple. Like, no, no, we, that sounds pedantic. It's not. We've had AI for decades. We've had models for decades. Some small, some large.

Nothing about that has changed. The way we would have evaluated an AI company, I mean, you know, we invested in Cruise, which was a lot of AI and a lot of models. We invested in Grammarly, which is a lot of AI and a lot of models back in the day. You're not asking any of those foundational model questions five or 10 years ago as we're investing in those companies. You're asking about how it's going to change the world and whether you think it's really going to change the world and then

and then making the call. It's not cart before the horse. I think there is a thing called a foundational model. And I think it's not surprising that every other founder who's building any kind of AI model would now rename their thing a foundational model. Right. That's the issue here, right? We didn't see people pitching pinpoint models or specific models. They were foundation model for X, foundation model for Y. Right. Yeah. But a foundation model for making cupcakes is not a foundation model.

I saw a post this last week, I think it dropped into Slack, that some of these cool one-shot prompts have wildly different levels of efficacy, even across all the foundational model. So we stare at these loose evals, which give us some random number. But even individual prompts are, you know, you'd give them a C- in ChatGPT and an A- in Cloud, and then the reverse for some other prompt.

And so, no, and you see, you know, if a model has been trained and tuned a lot more for coding data, it's going to be better at coding. We know that these large foundational model companies are kind of looking at different areas where they're kind of like piecemealing in data and structuring the data properly in order for it to get more efficacious in those areas. So it doesn't mean that one foundational model won't really be a massive, massive, massive company, you know,

We're obviously very bullish about Anthropic. But no, man, I think even foundational models, it's worth asking the question, is it good at the thing I want it to do? It still comes down to a customer trying to do something. Makes sense. We're hearing from a lot of founders. Remember that founder that told us that she used, I can't remember which one, but Anthropic for one task and then

GPT-4 for a different time. We see that across lots of startups, man. Yeah, all the time. For folks who are willing to test the model, they absolutely are seeing that like, I like the tone and tenor of Claude for X and then I'm using, you know, Gemini Pro if they give me access for Y. Yeah, the thing that I thought was interesting is that they had just seen very clearly different

qualitative aspects that manifest themselves in the product experience for different parts of their product. And that is not what you would expect if these are all foundation models that generalize broadly across everything. There really is only large language models. Multimodal, eventually multimodal, large language models. It's an N of

one particular situation in foundational models, which has almost never existed in computing before. And it looked like we're going to have a three or four horse race between maybe meta, I don't know how you feel about that, ChatGPT, Quad, Google. We'll see if anybody else possibly emerges. And everything else is an AI company. It's an enabling AI company. And maybe they're using the foundational models

What we increasingly see is they're using a mix. They're using a foundational model for some stuff. They're using some smaller proprietary models for other things, using open source models for other things. Oh, that's great. And that's not what you're underwriting. And I think it's just easier to evaluate that way. We have done a couple other large model investments. Mm-hmm.

You know, one in bio, perfluent, right? One in adept, action transformer models. But we are also, in those particular cases, very much in love with the use case. So we did talk about models a lot back then, but I would argue you could have done the reverse and we would have gotten to maybe a faster, better decision, frankly. You could have worked from use case down and capabilities and still gotten it.

Yeah, you know what? I'm now reflecting. So Ali Madani is the founder of ProFluent, who's scaling the transformer into biology, just to give some context. And I'm now replaying conversations with him over the past handful of months. And he positions it, he doesn't call it a foundation model for biology. He positions it as they have a chassis that they're then going to pull into specific verticals. And we saw the first one was gene editing.

which is like amazing, right? But that is a use case first framing of the problem with an enabling technology that might have some, you know, ability to span across verticals. And so if I give him like credit, he's been talking about it in that way since the start. I mean, to be, to be,

there and give us wide credit for our people. Like, that's a lot of what made us excited about Ali, like, and what he was doing, right? He's not just going to be a science project where he's just going to, like, throw data at the problem. He's like, no, like, these are the use cases I want in the world. These are the places I think we can build proteins for so you can help the world. Like, and I'm going to go do that. And I use Transformer architecture. Yeah, yeah, yeah, yeah. Just as an aside, let me geek out for, like, 20 seconds. Their release came out since we've most recently spoken. The idea that

that we used to get excited about using deep learning for natural language processing techniques five years ago. And it could figure out like the double negation in a sentence. Like I remember sitting in my company and being like, oh my goodness, I figured out the not not problem. And now not only is the transformer writing like full essays, but you know, all these company has used the transformer

the transformer to design a brand new protein that doesn't exist in nature that's doing gene editing in a mammalian cell. Like, it's crazy. I think we lose track of how fast the world has changed over the past five years. I agree. I agree. And just to be clear, this doesn't mean I'm suddenly some anti-AI guy. Like, I think you know that, right? Like, I think

I think GAIA is going to work its way through the entire GDP of the United States over the next 20 years over the world. And I think we will eventually hit AGI. We don't have to have a debate about when. And it's just a question of how do you evaluate now versus later? And being off by two years in startup land is a dead company. And like off on timing is just wrong.

It's not off on timing. And so if we're trying to evaluate what you're going into right now, you just still have to just ask the question about whether a user is going to be satisfied. And you can start from there. Yeah, listen, I've never once thought that you were a Luddite. Right? Like...

I think that your pragmatism is appreciated. And what you're saying is most of these other models, the industry was asking the wrong questions of, and many people still are. And when you ask the wrong question, like, listen, we happily spent a lot of time debating, you know, what's the defensibility of this model, this, that, and the other thing. But if you think of it as the enabling technology for the end user,

I think you ask very different types of questions. And I mean, it's also just kind of encouraging, not just us, obviously, but like, we're just being transparent about what we're talking about internally. But I find that same loop happening with founders.

Yeah. And probably not productive. Yeah, yeah, yeah, yeah. Some number of minutes ago, you asked me what I thought about Meta and their position delivering big models. And I'm going to use this opportunity to transition into something that I want to talk to you about today. But before I get there, I'm going to give myself the odd pat on the back. I think in terms of like the API world where the builders are gaining access to

you know, the most capable models. I think a year ago, my observation in post was that there would be two, maybe three small number of players who do this. I said, open AI and Anthropic. And I said, maybe DeepMind. And it feels like open source would then be gapped by one to two years on capabilities.

And I feel just as good about that even more so now in terms of like raw capability. And I think there's a lot of evidence suggestive of that, which is great. Like I think if the open source community is 12 to 18 months behind and that becomes like a...

not a commodity, but like a much more accessible layer for product builders. We're going to see a lot of beautiful things happen and then they'll pay up in every sense when they want to use the most capable models from a small handful of players. The conversation that I want to have with you today is put aside the API and like raw model capability discussion. Put aside the market of AI for work productivity because I also think that that's a very important market, but it's different from the one that I want to talk to you about

There has been a lot of news over the past couple of weeks from Meta and others into the AI for consumer space. And this is a very important market. And I've been trying to think through a comment that our friend Diane from Anthropic made at the dinner recently that like, what happens in that market depends on where we are in the S-curve. And I thought it was just a wonderful framing and

I don't know. I have thoughts that I want to put by you. So I'm not Mitch Wood on this. I think I'm either on the left or the right of the curve. I think if the model is the product for consumers, like if consumers are interacting and the model is... What is your mom going to use in four years when it comes to AI using a chat GPT-like product? That's right. If the model is the product and there's...

differentiated value to the end user of the frontier capabilities, I think that goes to a group like OpenAI. He says...

being the person who launched Jats GPT or part of No, no, no, no, no, no. He says being the head of product No, no, no, no, no. No, no, no, no. My strong belief on that one is like I remember going to the whiteboard and drawing like a matrices for Brad. Like, I think the real potential there is around, uh,

AI worker productivity for white-collar workers. And I think that a lot of their use cases that they're publicizing around Moderna and stuff like this really reinforce that. I think there's certainly the AI in an API that builders use. There's an AI for work set of products that are going to be a completely different market. And then there's the, what is the consumer using when they think of using a broad AI product? What

for the past year and a bit would have been chat GPT for many people. I think we're at the early stages of the S-curve for that. And I think there's a wonderful amount of ambiguity as to how things are going to play out over the next handful of years. And I've been trying to think through what dynamics lead to what outcome. I'm going to insert for a second here. Do you still think it's a chat interface in four years, the winner? So you're assuming in four years, there's a winner of this market.

And they're the Google of the I'm talking to an LLM that your mother's going to use. Right. Do you think that's still a chat box? Well, listen...

Let me dodge that question directly by setting up the conversation, right? There's a future where there needs to be real product exploration. It's a mystery that you're going to cut through and figure out the right product experience that delivers profound value to the end user. In that world, I think it's a net new entrant who wins.

So if it's not a chatbot and you have to have some sort of crazy product experience delivered, you know, that doesn't exist today that we haven't foreseen or that we haven't seen, I think it's net new. I think if it's a chat-like experience where, and chat can blossom into many different things. Like if you have tool use and all sorts of other capabilities that are coming into a basic chat-like experience, if the differentiation happens at the model layer,

But the UI remains somewhat consistent with where we are or simple. And the end user cares about the frontier of capabilities. I think it's going to be open AI or similar. And then I think every other scenario, it's going to go to the incumbent. And I think we're going to see Meta, Google and Apple all just have a big piece of that market on their platforms.

Well, I'll contend with your earlier point. You listed, I think, a handful of people there that could maybe win the kind of like Google, OpenAI, Meta, maybe Anthropic, and maybe nobody else is your point. It's going to be the incumbents and these handful of people. I just think Meta's... Isn't Meta Google Wave?

Like, like I love that Zuck is doing what he's doing. I love, you can just tell when he's talking in a podcast that he feels this, like he has founder energy in such a wonderful way that you know, the guy's not going to give up. He's in the middle of it. We're going to get some good models from Lama over the next couple of years. Like it's awesome. I liked all that. But when it comes to consumer surfacing, like he's just,

She jammed it into Instagram and every other bit of everything. It's like Google Wave. I don't think they have the canvas to have this conversation properly with a customer, to have a customer come to them with the expectation of interacting with a Facebook platform in the right way. Maybe there's a case that messaging something like WhatsApp is maybe, maybe there, but it just feels very tacked on.

It is very tacked on. Of course it is. But it's so aggressive. It's like wonderful Zuck energy. But here's what we can assume to be true, right? Is they shared, OpenAI, Sam shared that they have 100 million weekly active users at some point. Sure. Let's just say that's a massive number. That's a massive number.

You know that the large majority of those individuals, the large majority of those users are on the free tier, which is 3.5. Yeah. Right? Yeah. That means that the large majority of users are now getting the exact same experience that you can get within the top level feature of WhatsApp, Instagram, and every place else that Zuck has just jammed it in. And I saw that Chrome...

Google now in Chrome in the search bar, if you start your search with at, you actually get taken to Gemini. And so we're just going to see them get more and more aggressive to jam that into their distribution channels. I'm sure we're going to see the same thing at WWDC. And so then the question to you is, it's certainly not Wave like we have seen for a year and a half.

That a large majority of people who are turning to these products, whether it's Claude or a chat GPT, are doing it for a basic chat experience that is now in the hands of both Google and Meta. And so for you to think that they're going to fail, you have to think that there's a net new product experience that needs to be delivered for this to win.

I think there's a few things worth covering. One is I'm always going to be coming off looking for the ways that startups beat large companies because it's literally dedicated my whole life to it. And so finding the scenes where they screw up and they have problems and therefore this is the path, like it's literally what I wake up to do.

So that's the first thing, because it also happens over and over and over again. The pattern of large company wakes up to the new innovation thing and then jams in a fast solution, not fully thought through, trying to use their power and energy and still loses. Partly because, as you put it very aptly a couple weeks ago, they shipped their org chart, not just what customers want.

that's just a, that happens all the time. Like for, for every time that a war is won, the, you know, Microsoft does compete with Netscape and does release Internet Explorer because they did get out to market properly and they use their incumbent bundling advantage of an OS to get there. There's a hundred other examples where it just didn't work. And, and so the,

The Google Wave comment was, you know, Google deciding, oh my gosh, like Facebook's really big. We got to have social too. And then just like shoving social in every little orifice that they possibly could in Google in front of your face, not in the right context where you want to think about these problems and interact with these things. And the context matters for a consumer. And that's my problem with the meta thing. Like they just shoved it everywhere brute force. Most of the places are not

really thought through and are not really contextually relevant. And then in the Google situation, it's built into search. Great. It comes up in my Gemini at the top. I think probably for a bunch of people that may be their first experience with real AI is the top of Google just because of how big Google is. Makes sense. But as we already talked about like four episodes ago, there's kind of three different types of search.

Right? Yeah, yeah. And so the question is, I think for the third type of search, the research search, the deep back and forth search, the chat, the like, I'm trying to dig, dig, dig on something. I do think that's a new user interface. I don't think there's any chance Google or Meta wins that. They'll never be thoughtful enough to build the right surface areas for it. There's probably some amazing person inside of each of those orgs who knows that and loves to ship it and they'll want to let them. And so they have to leave to start a new company where they can join Cloud and they can do it there.

But there will be a new set of companies that will meet those new affordances. That's my bet. Yeah, I don't personally believe in what I'm just going to say. But the argument against your argument is that to date, what we've seen is that consumers en masse don't value the frontier of capabilities.

And they don't care about differentiated product. They want a good enough model. They want an answer that works fast. That's right. They want a good enough model that's now basically broadly available to anybody in a chat-like interface. And that's a case, that would be the bulk case for what Zuck is doing. I think that's kind of true as long as you understand why you're going to it. And when I click the little message bar on Instagram, I don't know why I'm there.

I don't know what I'm like. The same way that if you shoved a music player right in there, I would be like, what? Why didn't you just launch this as an app? What are you talking about? On the Instagram Explored tab, when they even give you the perplexity-inspired tongue-in-cheek, they just ripped off the perplexity UI. It is jarring. You're like, wait just a second. I want reels in the photos that I like. I don't want...

write me a recipe for this. Can I just say one last little rant here and then let's move on.

you know, the fact that everybody's using 3.5 because they're all in the free plans is actually quite interesting point. It's an unfortunate point because I can't tell you how many times I try and talk to people, even fellow founders and VCs and people in the industry. And I'm like, wait, you're on the free plan. Like, what are you doing? You don't even like, you're not even, and then they're complaining about it, hallucinating and not giving good outputs. And like, oh, my retention is off. And I'm like, well, you're not even using the good technology. I don't know what you're talking about. I think,

We do suffer a little bit of an industry trying to communicate to consumers. It's fine for enterprise because they can just test all the models and figure out what is efficacious. But we have done a very bad job. I'll blame you. You should have done this at OpenAI before you left. Like the product marketing around why you would upgrade and move from this plan to this plan, why you should pay for Opus at Claude.

like why you should be on Gemini Pro. We've not found a good set of language to make people understand that it's a categorically different experience. It is so much better when you are using state-of-the-art models for so many things. Not everything, right? For like 40% of the things, it's fine to use 3.5, which is why I caught everybody by surprise that Chachapiti spiked so quickly. But like,

We don't really do a great job as an industry explaining to people what that real experience is going to be like. And that's kind of unfortunate. I don't think they care. I think that we like happily so because it's our you and me and others in this industry, like we have to care about trends and where things are going to go. But most people, this is something that they're going to interact with a couple of times here and there. And it makes their life a little bit easier. And then they want to go to the softball game after work.

I get it. I get it. But that's why product marketing is so important because they are distracted and they just want solution. And if they get frustrated that solution isn't there, it's our fault for not explaining to them that the solution is right around the curve. It's just not on the free plan. Yeah, but they don't care enough about having that problem solved more elegantly to spend the 20 bucks a month.

The crux of this discussion of where things are going to go is, are we always going to see that users don't value the frontier of capabilities in a consumer-facing application and therefore won't be willing to pay? And in that case, I don't think it's going to be the large labs unless they change their business model. I think I heard you say earlier, which is what I actually agree with, is I think there's going to be a net new product experience that gets delivered

The likelihood that what we shipped because Noah iterated on a UI with Tina was the right UI, that seems so preposterous to me that somebody is going to show us a beautifully new creative way to shape the product experience. And I think at that point, we will see that users then care about it.

We're still pre the end game. I like that, Fraser. I like that. Yeah. And in that case, like I actually think that that's a net new entrant who wins that. Or there's a small team that one of these folks who actually does get it out the door and can come from up there, but it's still not new. I'm looking forward to that. You'll send me a text if you see it. Okay. Wonderful. Let's be done. Yeah, let's do it.

Thanks, everybody. We'll see you in a couple weeks. We'll see you. I don't know. We'll see you when we see you. Yeah. We'll see you with a couple of tweet summaries. And just let us know what format you'd actually like this in. That would be helpful. We can do it. We can transform. Later.