Windsurf CEO: Betting On AI Agents, Pivoting In 48 Hours, And The Future of Coding
52:35 Share

One of the things that I think is true for any startup is you have to keep proving yourself. Every single insight that we have is a depreciating insight. You look at a company like Nvidia, if Nvidia doesn't innovate in the next two years, AMD will be on their case. That's why I'm completely okay with a lot of our insights being wrong. If we don't continually have insights that we are executing on, we are just slowly dying. This notion of just a developer is probably going to broaden out to what's called a builder. And I think everyone is going to be a builder. I think software is going to be this very, very democratized thing.

Welcome back to another episode of The Light Cone. Today, we've got a real treat. We have the co-founder and CEO of Windsurf.

one of the people who literally brought vibe coding into existence. Varun, thanks for joining us. Thanks for having me, guys. Where is Windsurf now? Like all of us intuitively know, I mean, well, we use it, but you know, how big is it now? Where is it? Yeah, so the product has had well over a million developers kind of use the product as a

has hundreds of thousands of daily active users right now. It's being used for all sorts of things from modifying large code bases to building apps extremely quickly, zero to one. And we're super excited to see where the technology is going. - Let's get to the brass tacks. How'd you get started? - The company actually started four years ago.

We didn't start as Windsurf. We actually started as a company called Exafunction. We were a GPU virtualization company at the time. Previously, me and my co-founder had worked on autonomous vehicles and AR, VR, and we believed deep learning was going to transform many, many industries from financial services to defense to healthcare. You were right. Yeah. Yeah.

We might have timed it wrong though. Ultimately, we built a system to make it easier to run these deep learning workloads. And similar to what VMware does for computers and CPUs, we did that for GPUs. The middle of 2022 rolled around though, and what

What happened at the time was we were managing upwards of 10,000 GPUs for a handful of companies and we had made it to a couple million in revenue. But the transformer became very popular with these models like Tech's DaVinci from OpenAI. And we felt that that was going to fundamentally disrupt the

the business we had or that small business that we had at the time, because we felt that everyone was going to run these transformer type models. And in a world in which everyone was going to run one type of model architecture transformers, we thought if we were a GPU infrastructure provider, we would get commoditized, right? If everyone's going to do the same thing, what is our alpha going to be? So at the time we basically said, "Hey, could we take our technology and wholesale pivot the company to do something else?" And that's- That's scary. Yeah. That was a bet the company moment. We did it within a weekend.

So, you know, weekend me and my co-founder had a conversation along the lines of I don't think this is going to work. We don't think we know how to scale this company. And at the time, we were early adopters of GitHub Copilot. We told the rest of the company on Monday and everyone started working on Codium, which is the extension product the Monday immediately after. I'm just curious to like dig into this pivot story because it's like pretty rare, actually.

to hear like the details of a pivot, especially a late stage pivot. Like at the time that you guys decided to pivot to Codium, how far along was the company? How many people? One of the things about the company was, I guess, like we tried to embrace a lot of the YC sort of analogies here of, you know, ramen profitability, all these other sort of key insights. We were only a team of eight people at the time, even though we were making a couple million in revenue. So we were kind of like free cash flow positive. It was the peak of zero interest rate at that time. So the company was a year and a half old.

We had raised somehow magically $28 million of cash at the time. And I think the big point here in our minds was it doesn't matter if we're doing kind of well now, if we didn't know how to scale it,

we kind of need to change things like really fast. I guess the thing that's remarkable is when you started the company, you had this thesis where you were betting that a lot of companies were going to build their own custom deep learning pipelines to train birds, right? That was like the thing that was working.

But in 2022, you saw the hockey stick shift. That suddenly there would be one model that would rule them all. So you were foreshadowing a lot of the future. And there's a lot of it that came from that conviction. So I'm curious, what were those signs? Because you had to be really embedded into it too. We're making already seven figures. You could have raised a Series A. You're like, we're going to throw that.

all that away and burn it. Yeah, so actually, even crazily enough, we had raised our Series A at that time. Okay, you raised it. But whether or not we should have been able to is a different question. No, I think you're totally right. I think one

One of the things that was happening at the time was we were working with, largely speaking, these autonomous vehicle companies because they had the largest sort of deep learning workloads at the time. And we were sort of seeing, hey, that workload is growing and it's large. But we were betting fundamentally that the other workloads, which were these other natural language workloads, kind of like these workloads in these other industries like financial services, health care, would take off.

But I think once we saw these models, these generative models handle so many of the use cases, right? Maybe an example is in the past, you would train a BERT model to actually go out and do sentiment classification. But very quickly, when we tried even a bad version of GPT-3, like the very old version, we were like, this is going to kill sentiment.

send them in classification. There's no reason why anyone is going to train a very custom model anymore for this task. I think we saw the writing on the wall that our hypothesis was just wrong. And that's one of those things. You go in with some thesis on where you believe the space is going to go, but if your hypothesis is wrong and information on the ground changes, you have to change really fast. So then what did you decide to do? So it's like you decided, okay, we're going to pivot.

And when we work with founders, that's kind of stage one. So you're not half foot in, half foot out. So you had that conviction. We need to try something out. How do you figure out what was going to be the next step? I think we needed to pick something that actually everyone at the company was going to be excited about. I think if we had picked something that was what we thought could be valuable, but people were not excited about, ultimately we would fail.

We'd fail immediately. We came with an opinionated stance of we were early adopters of a product called GitHub Copilot. We thought that that was the tip of the iceberg on where the technology could go. Obviously, everyone at the company was a developer. Dev tool companies, generally speaking, usually don't do that well. And in the past, they have not. But hey, like when you have no other options, it's a very easy decision, right? Like you're going to be a zero...

with a high probability anyways, you might as well pick something that you think could be valuable and everyone's going to be motivated to work on. MARK MANDEL: Everyone's forgot this now, it feels like, because GitHub Copilot's in the background. But at that particular moment, it felt inevitable that Copilot was going to win, right? Like, it just had everything, like the GitHub connection, Microsoft distribution.

Open AI. Open AI, yeah. It seemed like no one could compete. So how did you have the bravery to be like, oh, yeah, we can totally crush Copilot? Yeah, so this is where there's an irrational optimism piece. I've said this before to the company, but I think startups require two distinct beliefs, and they actually...

kind of like run counter to each other. You need this irrational optimism, because if you don't have the optimism, you just won't do anything. You're just a pessimist and a skeptic. And those people don't really accomplish anything in life. And you need uncompromising realism, which is that when the facts change, you actually change your mind, right? And that's a very hard thing to do both. Because the thing that makes you succeed through irrational optimism, like, you know, is exactly opposite to the things that allow you to be a very realistic company.

So irrational optimism. We basically said, hey, we know how to run and train models ourselves. We actually trained the first autocomplete models ourselves and ran it on our product, gave it out for free. I don't think we had the exact roadmap on where this was going, but we just felt there was a lot more to do here. And if we couldn't do it, then I guess we'd die. But we might as well bet that we could do it.

were your early versions better than copilot, GitHub copilot at the time? So our earliest version that we shipped out was materially worse than GitHub copilot. The only difference was it was free. Uh, we built a VS code extension after pivot within, I think two months we had shipped the product and given it out to hacker news, uh, like posted something on hacker news. Um, and, uh, we, we built that out. It was missing a lot of key features. Like the model that we were running was like an open source model. That was not nearly good as the model that GitHub copilot was running. Uh,

very quickly then our training infrastructure got better. So we actually went out and trained our own models based on the task. And then suddenly it actually got capabilities that even GitHub Copa didn't within two months, basic capabilities. Now we'd find this like hilarious that it was even state of the art, but our model could actually fill in the middle of code. So when you're writing code, you're not only just adding code at the end of your cursor, but you're filling it in between two parts of a line, right? And

that code is very incomplete and looks nothing like the training data of these original models. So we trained our models to make it actually very, very capable for that use case. And that actually allowed us to pull ahead in terms of the quality, latency. We were able to control a lot of details within a couple months. So the beginning of 2023, I'd say the autocomplete capabilities were much better than what Copilot had.

Was that a totally new capability for you guys? Because we've just been building GPU infrastructure. It sounds like you basically hacked together the first version by taking an off-the-shelf open source model, sticking it into a VS Code extension, and just wiring the two to talk to each other. But then right after that, you had to train your own coding model from scratch.

And you guys have been like following the transformer stuff, but like you hadn't built it. Like in order to do that, I seem to have to like download all of GitHub and like train a whole model from scratch. How did you all figure out how to do that in like only two months? Yeah, it's a great question. So first of all, when we ran the model ourselves, the reason why we were able to run it and provide it for free is we actually had our own inference runtime at the time.

And that obviously came from the fact that we were originally a GPU virtualization company to start with. So that enabled us to actually ship the vZero with an open source product quite quickly. Immediately after that, you're totally right. We had never trained a model like this in the past, but I think we hired people that were smart, capable, and excited to win. So we needed to figure it out. There's no other option, right? Otherwise you die.

makes the decision-making really, really simple. So yeah, we had to figure out how do you get a lot of data? How do you do this at scale? How do you clean this data? How do you make it so that it's actually capable of handling this case where code is very incomplete? And we shipped a model like very, very quickly after that. Wow. And you did all of that with like eight-ish people in two months. Yeah, that's right. And then right after that, because you were running your own models,

You started getting interesting customers, right? Yeah. So basically what happened, the product was free at the time. So we ended up getting a lot of developers using the product across all the IDEs. So VS Code, JetBrains, Eclipse, Vim. Companies started reaching out because they wanted to not only, I guess, run the product in a secure way, they also wanted to personalize it to all the private data inside the company. So very quickly afterwards, in the next coming months, companies like Dell, JP Morgan Chase started to become customers of our

our product. Now these companies have tens of thousands of developers on the product internally, but we started actually making a lot of focus of the company, making sure that the product works on these very large code bases. Some of these companies have code bases that are well over a hundred million lines of code. And making sure that the suggestions are fast is one thing, but making sure it's actually personalized to the code base and the environment that they have was almost a requirement at the time. You did that pivot, you built it in,

Two months, then shipped it, and within a couple of months you got these big logos. Yeah. So, I mean, obviously these companies take some time to close, but pilots were starting within a couple months or a quarter after that. Obviously we had no salespeople at the company, so the founding team was just trying to run as many pilots as possible to see what would ultimately work. At what point did you expand beyond just the VS Code extension into supporting all these other IDEs? That was actually very, very soon afterwards. How did you think about that?

There's one argument that you could make, which is there's lots of VS Code developers. You had a tiny team. You could have made the argument that I was just focused on building a great experience for VS Code. You'd only captured a tiny percentage of the market of all possible VS Code developers. And that's not what you did. You expanded horizontally very quickly and built extensions for all those IDEs. Why? I think maybe the fundamental reason that

that we thought was quite critical is if we were going to work with companies, companies have developers that write in many languages. Like for instance, like, you know, a company like JPMorgan Chase might have over half of their developers writing in Java. And for those developers, they are going to use JetBrains and IntelliJ.

IntelliJ is over 70 to 80% of all Java developers in the world currently use IntelliJ right now. So we would just need to turn away a lot of companies. Like a lot of companies would not be able to use us as the de facto solution. Like we'd be one of many solutions inside the company. So because of that, we made the decision. But luckily, because we made the decision early enough, it changed the architecture of how we built the product out, which is to say we are not like building a separate version of the product for every single IDE.

We have a lot of shared infrastructure that actually lives on a per editor basis. So it's actually a very, very small amount of code that actually needs to get written to make sure we can support as many IDs as possible. So this is one of those things that an early decision that we made ended up making it much easier to make this transition. How about the transition from Codium to Windsor? At the time, now we're probably going to middle of 2023. We start working with some very large enterprises. Within the next year, the business has gone well over eight figures in revenue.

just from these enterprises using the product. And we have this like free individual product. But I think one of the things about this industry that we all kind of know is the space moves really, really fast. And we basically are always making bets on things that are not working, right? Actually, most of the bets we make in the company don't work. And I'm excited when I'm like happy when we're, when we're, let's say only 50% of the things we're doing are actually working. Because I think when, if a hundred percent of the things we're doing are working, it's,

I think like it's a very bad sign for us because it's probably like one of maybe three things. The first thing it is, is like, hey, we're not trying hard enough, right? That's probably what it means. The second thing is we somehow have a lot of hubris, right? And the hubris is like, we believe everything we do is right, even despite the facts that are...

that are sort of on the ground. And then the sort of third key pieces here is we're not actually testing our hypotheses in a way that like tells us where the future is going. We're not actually at the frontier of what the capabilities and technology ultimately is. So we believed actually in the very beginning of last year that agents were gonna be extremely huge. And we had prototypes of this in the beginning of last year and they just didn't work.

But there were different pieces we were building that we felt were going to be important to making agents work, which is understanding large code bases, understanding the intent of developers, making edits on the code base really, really quickly. And we had all these pieces. The thing we didn't have is a model that was capable of calling these tools efficiently enough.

And then obviously in the middle of last year, that completely changed with the advent of like Sonnet 3.5, right? And with that, we basically said, okay, we now have these agentic capabilities, but the ceiling of what is we can show to our developers on VS Code was limited.

We were not able to provide a good enough experience. And we thought what was going to happen is developers would spend way more time not writing software, but reviewing software that the AI was going to put out. We're, I think, a technology company at heart. And I think we are a product company. But I think the product serves the technology, which is to say we want to make the product as good as possible to make it so that people can experience the technology.

And we felt that with VS Code, we were not able to do that. So the middle of last year, we decided, hey, we need to actually go out and actually...

have our own ID out there. So that's what triggered actually creating Windsurf. The way that you did that was you forked VS Code. We forked VS Code. Was that a whole new set of capabilities that you guys had to learn, like basically how to like develop on this VS Code code base that I'm sure is super complicated? Yeah, we needed to figure that out. That was one, once again, another thing where we ended up shipping Windsurf out within less than three months of starting the project.

That's when we shipped it out across all operating systems. Wow. And what happened? Did it take off immediately or was it unnoticed for a long time? It took off pretty quickly, I would say. I think the speed at which it took off among early adopters was quite high. There were obviously some very rough edges. And this is one of those things where because of the rough edges, obviously people started coming and leaving the platform fairly quickly. But what we saw was as we improved the capabilities of the agent, as we improved the capabilities of the

passive experience, even the passive tab experience has made massive leaps in the last couple months. We started realizing that not only were people talking about the product more and more, people were also staying on the product more and more at a higher rate. How many people worked on shipping Windsurf? And it was done in a

In a period of one or two months? A couple months. So yeah, like less than three months. This was another... I wouldn't say it's a bet the company moment because it's not a fundamentally different sort of paradigm compared to like moving from a GPU virtualization product to an AI code product. But yeah, it was anyone that could work on it needed to kind of like drop what they were working on in the past and work on it immediately. And at that time, how big were you guys? The engineering team was probably...

still less than 25 people. Wow. This is crazy. Interestingly, our company actually, from an employee standpoint, actually didn't have that few people. We actually had a fairly large go-to-market team because in the AI space, one thing that's a little bit weird about our company compared to most other companies is we have a fairly large go-to-market team. We were selling our product to the largest Fortune 500 companies and

It's very hard to do that purely by letting them swipe a credit card. You need a lot of support. You need to make sure that the technology is getting adopted properly, which is very different than just give the people the product and see it grow effectively. So from an engineering standpoint, we've always run fairly lean.

But because of the market interest, we've always had a lot of people on go-to-market, actually. Who are sort of the ideal people to go into that function? Is it really good engineers who want to be forward deployed? Yeah. So we have two components of it. We have account executives. So these are folks that...

In general, we try to find people that are very curious and excited about the capabilities. In fact, people that would use Windsurf in their free time because they're providing the product to leaders who also love software and technology, right? So if they're just completely just unaware of the technology, they're not going to be helpful. And then we also have these deployed engineer like sort of roles, similar to what you said, that get...

get their hands really dirty with the technology and make sure that our customers get the most value from the technology. I mean, the wild thing is because everyone uses Windsurf, it sounded like you're having even these AEs who are non-technical become like just vibe coding champions. Yeah. No, one of our biggest users of Windsurf at the company is a non-technical person who leads like partnerships at the company. He's actually replaced by...

buying a bunch of sales tools inside the company uh and this is one of those things where i think windsurf is giving power back to the domain experts right in the past what would happen in an organization is he would need to talk to a product manager who would talk to an engineer who and the engineer would have a large backlog because they're this clearly doesn't immediately make the product better right so this has to be a lower priority but now he is actually kind of uh

empowered to actually go and build these apps do you have any programming background at all no interesting none because that's definitely one of the controversies on twitter at the moment it's just like is vibe can you actually vibe code unless you know some amount of coding already yeah one of the things we do have is if we do need to deploy one of these apps we have a person that actually focuses on making sure that these apps are secure and deployed but the amount of leverage that that person has is ridiculous instead of him going out and building all of these apps uh

The v0 could actually get built by people that are domain experts but non-technical inside the company. With the Codium launch, you went head-on against Microsoft and GitHub and these huge incumbents. With the IDE launch, you went head-on against Cursor. It's the hot startup of the moment, I'd think.

Anyway, again, how did you all think about that internally? This might be a weird thing about our company, but our company just doesn't have-- morale is not really affected by what other companies do. It's not possible if-- our company has gone through a lot of basically very turbulent times. The fact that we needed to pivot at 10 employees and just completely kill our idea is a normal thing for the company.

And then second of all, kind of like the companies that are relevant in our space has always been a fluctuating set of companies. Like, you know, I really respect all the companies in our space, but yeah, Copilot, if you were to go to the beginning of 2023, everyone would have thought GitHub Copilot was the product that everyone would be

would use and there was no point building. And in the middle, kind of Devin came out and like everyone was like, hey, like Devin is going to solve everything. Right. And I'm sure they're doing good work now. But and then after that, obviously, Cursor is doing a really great job. So I think what really matters to us most is are we actually do we have a good long term strategy? And are we executing in a way where we're getting towards that long term strategy while being flexible with the details? Right. And as long as we're doing that,

I think we have a fighter's chance, right? And that's like the way we've always operated. Do you educate yourself at all on the competitor's products? Yes. Yeah. Yeah. I think we don't want to put our heads in the sand and kind of tell ourselves our product is awesome. And just kind of, because it's very easy to do that, especially given the fact that before we worked on Windsurf, the company was also growing very, very quickly from like a revenue standpoint. What sort of opinions did you have or what taste for opinions did you have for the full IDE platform?

That was maybe different to Cursor. What I'm actually just asking is Cursor is a very well-liked product, obviously. And so at a product level, why are we like, oh yeah, like actually we want to build it this way? Yeah. No, I think it's a great question. So maybe the first point is at the time, actually, when we started working on Windsurf, all the products were basically chat and this autocomplete capability. I think that's basically what GitHub Copilot was, what Cursor was at the time. I think we took a very opinionated stance that we thought agents...

were where the technology was actually going. We were the first agentic editor that was out there. And...

I think the biggest sort of takeaway was we didn't believe in this kind of paradigm where everyone would be at mentioning everything. Right. This almost reminded us of like the anti-pattern of what Google and the search engines were before like Google's improved their product a lot, which was kind of like these landing pages that had like every distinct kind of like bucket of things you could search for. But Google came out with this very clean search box. Even Google at the time, you would need you would get better answers if you wrote and

or like SiteLink. And now it's gotten way better, right? And I guess we had a belief that the software would get more and more easy to build, right? And we would build from that starting point. When we saw all the other players in the space making their product so configurable in a way that we thought was, I think, good for users now for what the technology was, but something that wouldn't be unnecessary down the line. So we invested in capabilities like how do you deeply understand the code base to

understand the intent of the developer. How do you actually go out and make changes in a way that's very quick to the code base? So we took the approach of, hey, instead of having this read-only system where you tag everything, what happens if you could make changes very quickly? And that's why at the time, we were kind of the first to do that. Now, if you were to ask, is that a very obvious decision now? I think it's very obvious now. It looks very obvious. And this is where one of the things that I think is true for any startup is you have to keep proving yourself.

Right. Like every single insight that we have is a depreciating insight is a very, very depreciating insight. Like technology. The reason why companies win at any given point is not like they had a tech. They had a tech insight.

like one year ago, right? Actually, if a company wins, other than the fact that they have, you know, a monopoly, you know, it's actually like a compounding tech advantage that keeps sort of existing over and over again. And I think the example of this that I find most exciting is you look at a company like NVIDIA. If NVIDIA doesn't innovate in the next two years, AMD will be on

on their case, right? And Nvidia will not be able to make 60, 70% gross margins at that point, right? Even though it's like one of the largest companies in existence right now. By basically having good insights to start with, you're able to learn from the market and maybe compound that advantage with time. And that's the only thing that could be persistent.

It sounds like a moat is, you know, we think of it as a noun, but it's actually a verb. Yeah, something that could change with time, right? I also think for us, and I tell the company this, if we're not continuing to have insights, and that's why I'm completely okay with a lot of our insights being wrong. If we don't continually have insights that we are executing on, we are just slowly dying. That's like what's actually happening. I think the interesting thing is that

It's easier now looking back and connecting the dots on your journey, how a lot of these technology bets you took actually did end up compounding what Windsurf ended up becoming, right? Like it was happenstance that you being really good at GPU deployment, VMware optimization ended up being the thing to be really good at being blazingly fast autocomplete because it's faster than other products, right? So that kind of compounded there.

There's the aspect of you building all these plugins for enterprises and being so good at reading large code bases. And you did something that was contrarian. There was a lot of products when we worked with OIC companies. A lot of...

CodeGen tools use vector databases because we work with a lot of companies and that was the standard approach how a lot of folks were building. But you guys did something very different, right? Yeah. So one of the things that sort of I think got really popular is this term RAG got very popular. Yeah, you heard anti-RAG. Yeah, I don't know if we're anti-RAG. RAG obviously makes sense. You do want to retrieve some stuff and based on the retrieval, you want to generate some stuff. So I guess the idea is correct.

that everything is retrieval, augmented generation. But I think what people got maybe a little too opinionated about was the way RAG is implemented. It has to be a vector database that you go out and search. I think a vector database is a tool in the toolkit. If you were to think about what users ultimately want, they want great answers.

and they want great agents. That's what they actually want. And how do you end up doing that? You need to make sure that what's in the context is as relevant as possible. So what we ended up doing is having a series of systems that enable us to pack the context with the most relevant snippets of code. And the way we ultimately did that was it was a combination of keyword search, RAG, abstract syntax-free parsing. And then on top of that, using, as you mentioned, all the GPU infrastructure we have to take large chunks of the code base and in real time, re-rank it, right? As the query is coming in.

And we found that that is the best way for us to find the best context for the user. And the motivation for this is because people have weird questions. They might have a question for a large code base of upgrade all versions of this API to this API. And if embedding search only finds five of them, it's not of the 10. It's not a very useful question.

useful feature at that point. So we needed to make sure the precision recall was as high as possible, which meant that we used a series of technologies to actually get to the best solution. There's a bit of a thing going on with a lot of AI startups getting built, taking too much of an intellectual shortcut with what works for the problem domain space. But you took it from first principle, right? So you built like a way more

complex system that did AST, Parsons, SyncStats3, all this stuff, which is like cool. Yeah, I think maybe one of the things that is potentially interesting to discuss there is we started off, a lot of the companies started off working on autonomous vehicles. And the reason why that's kind of important is these are systems where you can't just YOLO these systems, which is to say you build the software and then you kind of like let it run. You need really good evaluation.

I think at the company, we don't strive for complexity. We strive for what works. And then the question then is, why is the system so much more complex now? It's because we built really good evaluation systems. MARK MANDEL: Oh, interesting. How do the evals work? MARK BLYTH: Yeah, so the evals for code are actually really cool. Basically, the idea is code-- you can leverage a property of code, which it can be run.

And we not only have real-time user data, we can put that aside for now, but we can take a lot of open source projects and find, I guess, commits in these open source projects with tests attached to them. So you can imagine a lot of cool things we can do based on that. You can take the intent of a commit

delete all the code that is not the unit test, right? And then you can see, hey, are you able to retrieve the parts where the change needs to get made? Do you have a good high-level intent to make those changes? And then after making the changes, does the test pass? You can do that task. You can mask the task. And by masking the task, it's more like the Google task. And what I mean by the Google task is it's trying to predict your intent, which is to say, let's say you only put in a third of the change.

but you don't get the intent. Can you then fill out the rest to make the test pass? So there's so many ways you can slice this and each of them, you can break it down into so much granularity. You can be like, what is my retrieval accuracy? What is my intent accuracy? What is my passing? What is my test passing accuracy? You can do that. And then now you have a hill to climb. And I think that's actually important before you add a lot of complexity for any of these AI apps. I think you need to make a rigorous hill that you can actually climb.

right? Otherwise you're just shooting in the dark, right? Why would we add the ASG parsing if it's unnecessary? Actually, it's awesome if it was unnecessary, right? I don't want to add a lot of complex stuff to our code. In fact, I want the simplest code that ends up having the most impact. So the evals were actually really, really critical for us to make a lot of these investments at the company. How much of the development that you do is like

basically driven by improving the scores on the evals versus like basically vibes based. Like you guys are all using Windsurf yourself. You're getting feedback from users all the time. And then you have just like a sense that this thing is going to work better. And then the evals are just sort of like a check that you didn't screw up something else. It's a little bit of both, but obviously for some kinds of systems, I think evals are more important than vibes.

but like more easy than vibes, just because for the system that basically takes a large chunk of the code, chunks it up and passes it to hundreds of GPUs in parallel, giving you a result in one second. It's very hard to have an intuition of like, is this way better? Because that's a very complex sort of retrieval question. But in the other hand, there are much easier things that from a vibe perspective are valuable. What if we looked at the open files in a code base? This is actually a harder thing to eval.

Because when you're eval-ing, you don't know what the user is doing in real time. This is one of those cases where having a product in market helps us a lot. And we're able to take a lot of user data on how people use the product to actually actively make the product much better. So that maybe starts with vibes, and then after that, you can build eval.

So it's a little bit of both, basically. PRIYANKA VERGADIA: I think there's been a lot of chatter on the internet about VybE's code is only for toy apps. Windsurf is actually being used for real production large code bases. Can you tell us about how the power users use it for more

more hardcore engineering. So this is an interesting thing where a lot of us at the company, I'm not saying that this is common, didn't get a tremendous value from ChatGPT in the way that probably a lot of the rest of the world did. And that's not because ChatGPT is not a useful product. I think ChatGPT is an incredibly useful product.

It's actually because a lot of them had already used things like Stack Overflow at the time. And Stack Overflow is a worse version than ChatGPT for the kinds of questions you want. But that was just a thing that they already knew how to use, right? And so they were able to get away with not using or relying on chat nearly as much. But basically what happened is very recently with agents...

The agent is making larger and larger scale changes with time. And I think what developers now at our company do is they have felt the hills and valleys of this product, which is to say, if you don't provide enough intent, it actually goes out and changes way more of the code than you actually need. Right. And this is like a real problem with the tool right now. But they understand the hills and valleys. And now the very first time they have a task, they are putting it into Windsurf. They're not

their first thing is not to actually go out and type in the actual editor. It's to actually put the intent and actually make those changes. And they're doing kind of very interesting things now, like deploying our software to our servers actually now gets done with the workflows that are entirely built inside Windsurf. So just a lot of boilerplate and repetitive tasks have been completely sort of eliminated inside our company. But the reason why this is

possible is kind of because we're able to operate over a code base that has many millions of lines of code really, really effectively. So if you were to give some tips to the audience, how should a user that uses WinServe properly provide this intent so that the changes are more surgical? Because what you're saying with

the agents creating all these broad changes. I've seen that happen. But how do you get those precise changes? What do you do? How do you feed the system? How do you become a... Shout at it with all caps, right? Yeah. No, so I think this is one of those things where I think...

you kind of need to have a little bit of faith in the system and let it kind of mess up a little bit, which is kind of scary. Because I think a lot of people, for the most part, they will write off these tools really quickly. Obviously, no one at our company would write off the tool because they're building the tools themselves. I think people's expectations are very high. And maybe that's like the main piece of feedback I'd give, which is that, you know, our product actually for these larger and larger changes, it might make 90% of the changes correctly. But if 10% is wrong, people will just...

right off the entire tool. And I think at that point, actually, probably the right thing to do is either revert the change, we have an ability to actually revert the change, or just keep going and see where it ultimately can go. And maybe the most important aspect is

commit your code like as frequently as possible. I think that maybe that's like the big, big sort of tip there, which is that, you know, you don't want to get in a situation where you've made 20 changes and on top of that made some changes yourselves and you can't like revert it. And then you get like very frustrated at the end of it. And one thing I wondered like in that vein is whether we need to change the way Git works with this AI coding approach.

Have you thought at all about whether doing Git commit all the time is the right move or whether there needs to be a deeper infrastructure change? Yeah, I think we have. So one of the things that we always think of is in the future, you're going to have many, many agents running in parallel on your code base. That has some trade-offs, right? If you have two agents that kind of modify the same piece of code at the same time, it's hard to actually know what's going on. That's another thing is that it's hard to have multiple branches checked out at the same time. Yeah.

with different agents working on them independently. That's right. Oh, the merge conflicts. Oh, God. There's a lot of that. But hey, that's how real software development works too. When you have a lot of engineers that operate on a code base, they're all kind of mucking around with the code base at the same time. So that's not a very unique thing. I think Git is a great tool. I think it's maybe a question of how can you skin Git to work in a way that works for this product surface? An example is Git has these things called work trees.

which is like you can have many work trees and versions of the repository all in the same directory. And perhaps you can have many of these agents working on different work trees. Or instead of exposing the branch concept to you, you actually can maintain a branch yourself that you can apply to the main branch of the user kind of repeatedly. One of the things that we think about at the company in terms of why we think our agent is really good is we try to have a unified timeline on everything that happened.

The unified timeline is not just what the developer did, but actually what the developer did in addition to what the agent did. So actually, our product, if you end up doing things in the editor, if you end up doing things in the terminal, all of those things are captured. And the intent is actually tracked in such a way where when you use the AI, the AI knows in that situation. So in some ways, we'd like this thing where the agent is not operating on a completely different timeline, but it's something that's getting merged in at a fairly high cadence.

I think this is like an open problem. I don't, I don't think we have like the exact right answer for this. What are other things that you envision changing about once Hrf in the future? How, how is it going to evolve? There's probably a lot of people that think the vibe coding is kind of a, kind of a fad, but I think that's going to get more and more capable with time. I think, you know, whenever I hear someone saying, Hey, this is not going to work for this complex use case. It's,

It's like, it feels like a Luddite saying something. It's like, if you look at the way these, these, uh, these AIs have gone better sort of year over year, it's, it's actually, it's astonishing. Like I'll give you an example of, of something that I held like kind of near and dear to my heart, which is, you know, there's this math Olympiad called the Amy. And I used to do that, uh, do that in, in high school. Um, and you know, I was very excited about, about how well I would do. I've,

my high score was somewhere close to 14. That's a very high score. But the crazy thing is that was one of those things that I thought, oh, wow, like the AI systems, they're not going to get anywhere near as good. And beginning of the year last year, it was probably like well under five maybe. And now the average that OpenAI has put out is like it's getting 14.5 to 15, right, for 04 mini. So it's almost like you have to keep –

projecting this out, right? It's going to get crazy. And basically every part of the software development lifecycle, whether it be writing code, reviewing code, testing code, debugging code, designing code, AI is going to be adding 10 times the amount of leverage very shortly.

It's going to happen much more quickly than people imagine. Going back to your current engineering team, I'm just curious, if they have all this time freed up from not having to deal with version upgrades and boring boilerplate stuff, what do they spend the extra time on? One of the things about our company and probably every startup that is building in this space is the ceiling of where the technology can go is so high.

It's so high. So it's actually that, you know, if developers can spend less time doing boilerplate, they can spend more time testing hypotheses for things that they're not sure work, right? In some ways, engineering becomes a lot more of like a research kind of culture, right? Where you're testing hypotheses really, really quickly. And that has some high cycle time attached to it, right? You need to go out and implement things. You need to build the evaluation. You need to test it out with our users. But that's like the things that actually make the product way better.

Does that mean you're going to hire a different type of engineer going forward? Are you looking for different things? Yeah, I think for engineers that we hire, we want to look for people with really high agency that are willing to be wrong and bold. But weirdly, I don't know if that's changed for a startup.

Startups should never be hiring people that the reason why they're joining a company is to very quickly write boilerplate code. Because in some sense, and this is not the goal, but a startup can succeed even if they have extremely kind of ugly code.

MARK MANDEL: Right. That's not usually the reason why a startup fails. MARK BLYTH: Sounds like my startup. MARK MANDEL: Yeah, yeah, exactly. That's not usually the reason why a startup fails. The reason why a startup fails is they didn't build a product that was differentially good for their users. That's why they ultimately failed. MARK BLYTH: This is all true, but also, in reality, you always need some sort of workhorses to just kind of get certain things done. I feel like in the old days, this was like building Android apps. It's like you'd hire someone to do it because there are very few people who would just be willing to do it. MARK BLYTH: Yeah. MARK BLYTH: Maybe in your vision,

for engineering, you don't need those people. That's not actually a useful skill to have in an organization anymore because the AI is just like your infinite workhorse. Is that fair? Yeah. Maybe the aspects of software that are really niche, that are undesirable for a lot of people to do, except a handful of people, those things get democratized a lot more, unless that has a lot of depth attached to it, at least for the time being. Yeah. If something is like, hey, you

you know, we need to change the system to use a new version. And there is someone that deeply always got in the weeds with version changes. I don't think you have people that are just focused on that right inside companies. How about how you interview people? Yeah, I think I think we have a fairly rigorous and high technical bar. And that has a that's a combination of we give interviews that actually allow people to use the AI to kind of solve a problem.

Because we want to validate if people kind of hate these tools or not. There are still some developers that do. And obviously, if you do, we're probably the wrong company to kind of work at. But also, at the same time, we do have sort of interviews in person where kind of on site where we don't give them the AI and we want to see them think, right? It would be a bad thing if ultimately when someone needs to write a nested for loop, they need to go to chat GPT, right? And I'm not like that's fundamentally because because it just it just

feels like that is a good proxy for problem solving skills. And I think problem solving skills are just at a high level, still should go at a premium. That is the valuable skill that humans have.

Yeah, a challenge that a lot of companies we've talked to have had, that we've even had ourselves, is that Windsurf has gotten so good that if you give people Windsurf, it's difficult to even come up with an interview question that Windsurf can't just one-shot. Anyone can do it because you literally just copy and paste the question into Windsurf and hit enter. And so you're not really evaluating anything at that point. Yes. I actually think that's true. And you're totally right. There's

very few problems now that something like an 04 Mini is not able to solve now. Right. I mean, if you look at competitive programming, it's just it's just in a league of its own already at this point. The crazy thing is interviews by nature are going to be kind of isolated problems.

right there by nature because if the problem actually required so much understanding to do you wouldn't be able to explain the problem so that's like perfect for the lms where you give them an isolated problem where you can test and run code extremely quickly so yeah you're totally right like i think if you tell if you only have algorithmic interviews and you let people use the ai

I don't know. You're not really testing anything at that point. Does that mean that you've gone away from just algorithmic questions and you ask different, like much harder questions that are actually well-suited to being able to use an AI? Yeah, we have questions obviously that are both system designing plus algorithms related, but these are questions that are fairly open-ended, right? There may not be a correct answer. There are trade-offs that you can ultimately make. And I think what we want to do is just see how people think

Right. Given different tradeoffs and different constraints. Right. And and we're trying to validate for like intellectual curiosity. Right. And if someone ultimately says, I don't know why, why, why, that's totally fine. As long as like they've gone to a depth that we feel kind of shows kind of interest, interest and like good problem solving skills, if that makes sense. You can tell when someone is curious. Right. And wants to learn things.

It's like very obvious. The next thought after this, which might be counterintuitive, you're at the forefront of building all these AI coding tools. It hasn't affected at all your hiring plans. On the contrary, you actually need way more engineers to execute. Tell us more about that. So I think that just boils down to, I think the problem has a very high ceiling, right? There's so many more things that we really want to do. The mission of the company is to reduce the time it takes to build technology and apps by 99%.

It's going to take a lot of work to go out and do that. Now, granted, each person in our company is way more productive than they were a year ago. But I think for us to go out and accomplish that, I think it's a Herculean task. We need to start targeting more of the development experience, right? Right now, we've helped a lot with the code writing process

and maybe the navigation of code process. But we have not touched much on the design process, on the deploying process. The debugging process is fairly rudimentary right now. There's just so many different pieces if I was to look at it. Like, you know, if you say you have 100 units of time, you know, we have an axe. We've cut off maybe like 40 or 50 in that time. But there's just a lot more snippets that we need to cut out basically at this point. It does feel like when I'm using Windsurf, like I am...

Often the extremely slow bridge between different pieces of technology, copy and pasting data, back and forth from my... That's probably actually still a large chunk of your time. All the pieces have gotten so fast that now it's like the glue between them. But like, I'm the glue, but I'm like much slower. Yeah.

Can I go off the reservation and ask a weird question? Go for it. Okay, one of the things that, I mean, I think Pete on our team, he just released a great essay about, you know, prompting and you should let users have access to system prompts. The other thing that he came up with that we've been using all at YC internally is a new agent infrastructure that has direct access, read access to our system of record, our Postgres database.

And in the process of using this, we're starting to realize if CodeGen gets a lot better, which based on this conversation, I think we can count on that getting 10x, 100x better from here. What if instead of building package software, there's just-in-time software that the agent basically just builds for you as you need it? Does that change the nature of

and SaaS and, you know, what happens to all of us in Windsurf? I don't know. I think this notion of just a developer is probably going to broaden out to what's called a builder. And I think everyone is going to be a builder and they can decide like, you know, how deep they want to go and build things. Maybe our current version of developers can go deep enough that they can build more complex things, right? In the shorter term. But yeah, I think software is going to be this very, very democratized thing, right? I imagine a future in which, you know,

what actually happens when you ask an AI assistant, Hey, build me something that like tracks the amount of calories I have. Why would you have a very custom app that goes out and does this? It's probably something that takes like all the inputs from your AR glasses and everything, and has a custom piece of software that kind of comes out and like an app that is there. And like, it has tasks that go and tell you, uh, you know, are you on track with, with all the calories you're, you're sort of consuming here. Um,

And I think that's a very, very custom piece of software that you have for yourself that you can keep tweaking. I can imagine a future like that where effectively everyone is building, but people don't know what they're building as software. They just, they're kind of just building just capabilities and technology that they have for themselves. Do many people use Windsurf who don't know how to write code at all? Yeah.

It's actually a large, a large number of our users. Yeah. Interesting. Yeah. How did they end up getting into Windsurf? Like, do they work at some company where like some programmers show them how to use it? Like I, I, I tend to think of Windsurf as targeting more like the professional developer market. That's like using this as a new superpower versus the like non-technical user market. That's doing like what Gary was talking about. We were shocked by this too, because we were like, Hey, our product is an ID, but there's actually a non-trivial chunk of our developers that have never opened

the editor up and they just, you know, our agent is called Cascade, right? And just to live in Cascade, we have browser preview. So they just open up the browser preview. They can click on things and sort of make changes. The benefit is because we kind of understand the code, when they come back to the repository and the code has actually gotten like quite gnarly, we're actually able to pick up from where the developer left off or the kind of builder left off and keep going from where they were. I will say we have not optimized tremendously

tremendously for that use case, but it's actually kind of crazy how much is actually happening there. Do you think in the long term that this ends up being one product that targets both of these audiences? Or do you think actually there's different products for

different audiences. There's like a windsurf, which is like focused on like serious developers who want to see the code and be in the details. And then there's maybe other products for folks who are totally non-technical who don't even want to see the code. I don't know what the long-term is going to look like. Something tells me it's going to become more unified. But one of the things that I will just say is like as a startup for us, even though we do have, you know, like a good number of people, there's a limit to what we can focus on internally as well.

So for us, we're not going to be focused on how do we build the best possible experience for the developer, as well as build the experience where we have so many things for the non-developer. But I have to imagine that this idea of building technology, as you get better and better at understanding code, you're going to be able to deliver a great experience for non-developers.

as well, but I don't know what the path dependence is. I assume like a bunch of companies in the space will go from non-developers to then supporting an ability to edit the code, right? And I think we're starting to see this already where, you know, the lines are sort of getting blurred right now. You probably care about it for your evals at least.

Yeah. No, you need to. You need to care about it for your evals. Maybe that's like the hard part for me to imagine for the pure non-developer product. What is the hill you're climbing if you're not like kind of understanding the code? How do you know your product is getting better and better? It's like an open question. Are you completely relying on the base models getting better, which is fine, but then you should imagine that your product is an extremely light layer on top of the base model, which is a scary place to be, right? That means you're going to get competed on

across all different axes. MARK BLYTH: How do you think about that in general, I guess? Something we've talked about a lot on this podcast is just the GBT wrapper meme has completely gone away, I feel. Though every big release from one of the labs sort of brings it back a little bit, and everyone's a little bit scared that OpenAI is just going to eat everything. How do you think about that?

I think the way I think about this is like, yeah, the company, as I mentioned before, it's a moving goalpost, which is to say today, if we're generating sort of 80, 90% of all committed software, I think when the new model comes out, we're going to need to up our game. We can't just be at the same stage. Maybe we need to be generating 95% of all committed code. And I think our opportunity is the gap between where the foundation model is and what 100% is, right?

And as long as we can continue to deliver an experience where there is a gap between the two, which I think there is, as long as there's any human in the loop at all in the experience, there's a gap. We'll be able to go out and build things. But that is a constantly transforming sort of goalpost for us, right? So you can imagine when a new model comes out, maybe the baseline on what the foundation model by itself provides has doubled. The alpha we provide on top of what the base model provides needs to double as well. It feels very, like for me, the reason why this is not the most concerning is

Let's suppose that you were to take the foundation model and it's providing 90%. It's reducing the time it takes by 90%. That actually means if we can deliver one or 2% more percentage points, that's a 20% gain on top of

what the new baseline is, right? Like I guess 90, if 90 becomes 92 or 93, which is still very, very valuable, right, at that point, because effectively the 90 becomes a new baseline for everyone. So I think basically the way we sort of operate is how can we provide as much additional value as possible? And as long as we have our eye on that, I think we're going to do fine. What advice would you have for our startups that are working in the like AI coding space? We have a ton of them. What are the opportunities that like you think are going to be open to new startups? Yeah.

I've seen a lot of things that I think could be like particularly interesting. I don't think any of these technologies we've really adopted, but there's so many different pieces of how people build software. And I'm not going to say niche, but there are so many different types of workloads out there. I've not really seen a lot of startups in the space that are just like, we do this one thing really, really well. Like I'll give you an example. Like we do these kind of like Java migrations really, really well. Crazily enough, if you look at this category, the amount that people spend on this is

is probably maybe billions, if not tens of billions of dollars doing these migrations every year. It's a massive category. That's an example. Migrations from what to what? So an example, this is like actually kind of crazy. JVM 7 to 8 or something like that. Yeah, JVM 7. Rails versions. Even more than that, actually. It's like a lot of companies write COBOL, have COBOL. And crazily enough, most of the IRS software is written in COBOL. Apparently in the early 2000s, they tried to migrate from COBOL to Java. I think it was a,

five plus billion dollar project. Surprise, surprise, it didn't happen. You think they could one shot it now? I don't know if they can one shot it. I'm just kidding. But imagine if you could do those tasks very well. It's such an economically valuable sort of task. I think we obviously don't have the ability to focus on these kinds of things inside the company. That's a very exciting space if you could do a really good job there. The second key piece is there are so many things that developers do that are also not making the product better, but important, like automatic repurposing

resolution of like, you know, kind of alerts and, and, and, and bugs in the in software. That's also a huge, huge amount of amount of spend out there. And I'd be curious to see like, what a best in class product in that category actually looks like. I'm sure someone that if they got truly in the weeds on that, they could build an awesome product. But But I think I've not heard of one that is like tremendously taken off. I think those are actually both really great insights. And one thing I like about them is that there's like, it's not just an opportunity for like

two startups. Each one of those is like a bucket that could have like 100 large companies in. We actually do have a company from S21 called Bloop that does these COBOL to Java migrations with agents. That's awesome. Yeah. It's a gnarly problem. It's a very gnarly problem, but if you were to talk to any company that has existed for over 30 years, this is probably something that is costing them hundreds of millions a year.

So reflecting on this journey, I mean, we're all really thankful for you creating Windsurf. It's supercharging all of society right now. What would you say to the person who, you know, basically the you from five years ago before you started this whole thing? The biggest thing I would say is change your mind much, much faster than you believe is reasonable, right? It's very easy to kind of fall in love with your ideas over and over again. And you do need to, otherwise you won't really do anything. But

but pivot as quickly as possible and treat pivots as a badge of honor. Most people don't have the courage to change their mind on things and they would rather kind of fail doing the thing that they told everyone they were doing than change their mind, take a bold step and succeed. Varun, thank you so much for joining us today. We'll catch you guys next time.