Scott Wu - Building Cognition - [Invest Like the Best, EP.401]
01:13:04 Share

As an investor, I'm always on the lookout for tools that can truly transform the way that we work as a business. AlphaSense has completely transformed the research process with cutting-edge AI technology and a vast collection of top-tier, reliable business content. Since I started using it, it's been a game-changer for my market research. I now rely on AlphaSense daily to uncover insights and make smarter decisions.

With the recent acquisition of Tegas, AlphaSense continues to be a best-in-class research platform delivering even more powerful tools to help users make informed decisions faster. What truly sets AlphaSense apart is its cutting-edge AI. Imagine completing your research five to ten times faster with search that delivers the most relevant results, helping you make high-conviction decisions with confidence.

AlphaSense provides access to over 300 million premium documents, including company filings, earnings reports, press releases, and more from public and private companies. You can even upload and manage your own proprietary documents for seamless integration. With over 10,000 premium content sources and top broker research from firms like Goldman Sachs and Morgan Stanley, AlphaSense gives you the tools to make high conviction decisions with confidence.

Here's the best part. Invest like the best listeners can get a free trial now. Just head to alpha-sense.com slash invest and experience firsthand how AlphaSense and Tegas help you make smarter decisions faster. Trust me, once you try it, you'll see why it is an essential tool for market research.

Every investment professional knows this challenge. You love the core work of investing, but operational complexities eat up valuable time and energy. That's where Ridgeline comes in, an all-in-one operating system designed specifically for investment managers. Ridgeline has created a comprehensive cloud platform that handles everything in real time, from trading and portfolio management to compliance and client reporting.

gone are the days of juggling multiple legacy systems and spending endless quarter ends compiling reports. It's worth reaching out to Ridgeline to see what the experience can be like with a single platform. Visit RidgelineApps.com to schedule a demo and learn more about joining the thousands of investment professionals who rely on the platform. We'll

We'll hear directly from someone who's made the switch. You'll hear a short clip from my conversation with Katie Ellenberg, who heads investment operations and portfolio administration at Geneva Capital Management. Her team implemented Ridgeline in just six months. And after this episode, she'll share her full experience and the key benefits they've seen. We are using our previous provider for over 30 years. We had the entire suite of products.

From the portfolio accounting to trade order management, reporting, the reconciliation features, I didn't think that we would ever be able to switch to anything else. Andy, our head trader, suggested that I meet with Ridgeline. And they started off right away, not by introducing their company, but who they were hiring. And that caught my attention. They were pretty much putting in place a dream team of technical experts. Then they started talking about this single source of data.

And I was like, what in the world? I couldn't even conceptualize that because I'm so used to all of these different systems and these different modules that sit on top of each other. And so I wanted to hear more about that. When I was looking at other companies, they could only solve for part of what we had and part of what we needed.

Ridgeline is the entire package and they're experts. We're no longer just a number. When we call service, they know who we are. They completely have our backs. I knew that they were not going to let us fail in this transition. Hello and welcome everyone. I'm Patrick O'Shaughnessy and this is Invest Like the Best. This show is an open-ended exploration of markets, ideas, stories, and strategies that will help you better invest both your time and your money.

Invest Like the Best is part of the Colossus family of podcasts, and you can access all our podcasts, including edited transcripts, show notes, and other resources to keep learning at joincolossus.com. Patrick O'Shaughnessy is the CEO of Positive Sum. All opinions expressed by Patrick and podcast guests are solely their own opinions and do not reflect the opinion of Positive Sum.

This podcast is for informational purposes only and should not be relied upon as a basis for investment decisions. Clients of Positive Sum may maintain positions in the securities discussed in this podcast. To learn more, visit psum.vc. My guest today is Scott Wu. Scott is the co-founder and CEO of Cognition, which is an applied AI lab that has created the first AI software engineer, which they call Devin.

In just a year since founding Cognition, Devin functions at the level of a junior software engineer, capable of handling complete engineering workflows from bug fixing to submitting pull requests. Scott is a former competitive programming champion, and he describes the engineering field as simply the art of telling a computer what you want it to do. Scott predicts AI will surpass the world's best competitive programmer within one to two years and sees this technology not as replacing programmers, but as democratizing software creation.

We discuss the bottleneck in software development, the future of AI in various industries, and the challenges of leveling up Devon. The implication of Scott's work and Devon itself is enormous.

Please enjoy my great conversation with Scott Wolf. Scott, since not everyone listening will know Cognition intimately, can you just start by describing what the business is and does? It's quite young. It's made a lot of progress very, very quickly, which is going to be a topic of our conversation in general. But maybe just start by describing for those listening what Cognition does.

Yeah, absolutely. And thank you so much for having me. We got started actually just around one year ago, and we are focused on building AI to automate software engineering. The product we're building is called Devon, and Devon is the first fully autonomous software engineer. And what that means is that Devon is basically able to do all of the parts of the software flow.

And so you work with Devin the same way that you would work with a coworker. And Devin is in your Slack, Devin is in your GitHub, uses all of your tools and works in your local developer environment. And so you can just tag Devin. You can say, hey, Devin, I've got this bug that's going on in the front end. Can you figure out what's going on? Devin will do not just the parts of writing the code, but actually all of the parts around it as well, which includes getting the local environment running, reproducing the bug itself, taking a look at

logs, figuring out what's going on, testing the bug, adding in some unit tests, and then making a pull request for you as the engineer to go review. What's the right way to think about, if I think about Devin as an actual engineer, its level of sophistication? Maybe you could put it in like salary terms or something or years of experience terms or some other thing like help manage my expectations for how much I could have this thing do without any human involved. Yeah.

It's very much a junior engineer today. That's how we think of it. That's how we describe it. I would say right around when we got started, high school CS student or something like that is probably fair. Maybe six months ago, an intern. And I think today it's very much an entry-level junior engineer.

There's going to be a lot of tasks. I mean, for example, if you wanted to say like, all right, I need you to re-architect my entire code base, rewrite the whole thing, make it all 10x more efficient, and then I'll check in in like a week. That's probably not the right way to use Devon. Whereas actually a lot of it, it really does function, turning every engineer into an engineering manager, where you get to work with your own team of Devons and hand off tasks. And so a lot of these little things which you would want to delegate

and spread off, you would say, hey, Devin, take a look at this, take a first pass. Maybe you have to fix a few things or maybe Devin asks a few questions that you have to answer or things like that. But the whole idea is that you're working asynchronously with your team of Devins while you're doing your own stuff or doing things separately. Should I think of Devin as a singular or a plural? The way that it's all set up is

We bill by usage. So we have a plan. The plan is 500 bucks a month for engineering teams by default. And then obviously as you use more, and then we build more. And the whole idea with it obviously is that you can have your own army of Devins. And each Devin is its own thing. And let's say you have five different bugs or issues or something that you're trying to work on today. A lot of our users do this where you just spin up five different Devins

And each one is tackling a different one. And obviously, they ask you as they have questions or put up code for you to review or things like that. But now you're actually jumping between your different Devins and tackling these tasks in parallel.

It seems like GPTs are GPTs, general purpose technologies. And one of the cool things about general purpose technologies is it's very hard to predict how the world will use them to build other things. Now that you've had Devon live for a while in private and now generally available, what has surprised you most about the ways in which people use it to do things? A lot of the emergent use cases...

are things that we wouldn't have predicted ourselves. One of the first things that companies that we work with have been using Devon for actually is a lot of these migrations, replatforms,

modernization and version upgrades, things like that. For example, we work with banks, we work with healthcare companies, and many of these folks have code that is literally decades old. And they would love to be on the latest stack on the latest technologies, make things faster, make things more efficient and make things more secure, it would make things easier for their developers to work on as well. But it's just such a pain, you as you can imagine, if your code base is 30,000 files,

to actually go through and do this kind of modernization effort. And so it's the kind of thing that's very tedious and it requires a little bit of active thought and effort to get into, but it's obviously there's just so much volume of it. And that was one of the first things where we really, really saw folks using Devon. A lot of our customers did their own internal studies of the usage and they were finding that every hour of an engineering's time using Devon was equivalent to eight to 12 hours actually without Devon.

So what that looks like is you're just asking questions and reviewing the code rather than having to write the code yourself. And so some of these usage patterns where a lot of the kind of more routine tasks that needed to get done, Devin was just really able to do in one fell swoop. Can you explain the key differences between

those innovating in the IDE like a cursor or something that just make it much easier for a programmer to program versus something like Devon, where it feels more holistic, like I'm handing off a whole thing and coming back and getting a complete piece of work. We've been doing this call it this agentic approach, like a full automation ever since we got started. And the way that I describe it, I think the biggest step function change is about

asynchronous versus synchronous. So a lot of these code assistant tools to call that, which you mentioned a lot of great examples, really great tools, honestly. What happens obviously is you use language models to do an autocomplete or you have simple things where it writes certain files or functions for you or things like that. And if it writes the right things, you hit tab and confirm those or confirm the suggestion that gets given.

Saves you that time of typing. I think in general, what folks have seen with those kinds of technologies is they see something like a 10 to 20% improvement across the board. So all of the different software work that they're doing, it makes them a 10 to 20% faster engineer, basically. By the way, engineering is huge.

That's not a small number at all. But with Devin, what we kind of see is there is a certain level of task, obviously, that Devin is capable of taking on. You're not going to ask it to do the entire re-architecting or something like that. But for the tasks that Devin can do, it's more like a 10x rather than a 10% because it really is you just handing off

the task to Devin, and you go ahead and do your own stuff. You work on other things. Maybe you're running other Devins as well. And in the meantime, Devin is really just a co-worker that you've given the task and works on that and gives you a full review and a full piece of code. There's a lot of meaningful differences in the product experience that all stem from that.

So Devon has to have the ability to test its own code, to run things live, to go and look up documentation, to be able to pull up websites and stuff by itself. You have to be able to talk to it in your GitHub or your Slack or your Linear or whatever technologies you're using as a team to communicate with each other. It's quite a different product experience. And honestly, we've seen from folks that it's complimentary. We use the Assistant suite of tools as well. And then we also use Devon. A lot of folks use both.

Because of where you sit and your own personal experience, I'd love to use the opportunity of chatting with you today to really talk about the past, present, and future of software engineering. I would love to start with the past. Maybe, you know, in my mind, it's this growing abstraction layer where it gets simpler and simpler and you have to know less and less things down close to the bare metal itself of semiconductors to do it.

do what you want to do in a computer system. Maybe you can describe your perspective on the history so far to this point of computer programming and the major milestones that we've reached before we talk about today and then the future. Well, I love the way that you actually just described it, which is I think software engineering this whole time, programming or whatever we want to call this, has always been basically just the art of telling the computer what you want it to do. That

That has been the goal. It is really that simple. And it's kind of funny, obviously, because the form factor of that has changed so much over time. And obviously, like a long, long time ago, it was all about punch cards. But even before that, it was about these gas tubes or things like that. That was programming. And then there was assembly. And assembly, that was programming. There was BASIC and there was C and these other languages and frameworks and technologies. We had cloud, we had mobile, a lot of big, broad shifts in the technology. I think the

the undercurrent that's going through this all is over the last 50 to 100 years, we have been able to automate so much of the processes that we need in our economy. And automating that is essentially, it boils down to being able to explain to the computer exactly what you want the computer to do for you. Really, really fascinating to study a lot of the history of it, because I think often it actually was the case that many of these technologies

At the time they came out, they were very interesting technology. And then there's a question of how do you make it useful? What is the use case? And then sometimes it was kind of surprising. I mean, probably the most famous example that folks know about is the ENIAC back in the 40s, which was the first real automated computer. And the initial use case, obviously, that really got it there was missile trajectories, ballistics, and all the physics that you want to calculate with that.

If you want to be very, very accurate and doing 10 digits of precision or something, you don't want to be doing that by hand. With punch cards, for example, the first real big thing that really caused that to be valuable was the US Census. And that was the thing that needed to be automated.

The first Macintosh, for example, like a lot of the initial use case actually was for graphic designers, actually. And then number two was actually the spreadsheet was the killer app that made it useful. With each of these, there was this technology that came about and then it took some time to really make it useful. We're seeing a very similar paradigm here where language models came out and obviously they took the world by storm.

ChatGBT in November of 2022, almost exactly two years ago, it was the moment when everyone realized what was possible. And now we're seeing what are all the natural applications of that. And I think one of those obviously is software itself. Accelerating software itself is just this recursive thing where you get such massive continued gains.

I don't know what age you were. I think you were 17 or something, won the competitive programming challenge. So literally, you were the best programmer alive or at least in some major way. At that time, and maybe today, what distinguishes, you hear about this idea of the 10x or 100x engineer, what are the qualities of those people that make them better than others? In those competitions, what is the competitive frontier? What are you trying to best your opponents at in that field?

Well, thank you so much for the kind words. There's these competitions, which are kind of more like algorithmic problems. There's software. But in all these things, I honestly think the soul of programming the whole way through, even as we're saying it's changed in form factor so much, I think the soul of it is really just being able to understand, deeply understand these

abstractions and just work at this abstract level. I think that's true in software today where the soul of software engineering is the part of taking a customer's problem or your own product's problem or something like that and really just figuring out, okay, here is exactly the solution that I need to build.

Here's how I'm going to represent it. And here's how I'm going to architect it. Here's all the cases and the details that I need to be thinking about. Here's the flows maybe that I'm going to be setting up on my website. Here's the database schema that's going to make this all work and make this efficient or something like that. And the thing that's kind of funny about it, obviously, is that is the part that every engineer loves about software. And it is also, I think, as you're saying, it's the thing that really distinguishes somebody from being a 1x versus a 10x versus 100x.

At the same time, it's probably only about 10% of what the average engineer does, because the other 90% of their time obviously is debugging, implementing, setting up testing, dealing with their DevOps, figuring out what went down with their database, whether that just crashed or all of these kinds of various things, which look a lot more like implementation than this creativity. But I would say even despite that, the creativity and the problem solving is actually really what defines programming.

In those competitions, how did you get better? Maybe just walking through the competitions themselves, how they were structured, what you did to try to win, how you won. I'm really interested in that chapter of your life. The competition was called the IOI, the International Olympiad of Infra-Moratics. It was very much the Olympics of programming. So every country sends their top contestants.

There's gold medals, silver medals, bronze medals. You qualify at your local level and then you make it to the team selection at your country and you qualify. And yeah, I mean, that was my entire life growing up, actually. I'm from Baton Rouge, Louisiana. There are not that many people who are into competitive programming in Baton Rouge, Louisiana, but I just really, really loved it. My older brother actually did it first and I got super into it through him. And it was the number one focus of my life for the first 17 years of my life, honestly. At the core of it, it's really just problem solving and

And you have code as the medium by which you build the solutions and all the things that you're designing around and so on. But the core of it is really whether you can figure out these algorithmic problems and understand the right approaches. I wonder if you could sketch for us what your best guess is as to what will happen in this world in the next five to 10 years. Computer programming writ large, obviously you're directly trying to affect that trajectory and what the answer is.

But you also occupied this cool seat with maybe five or 10 others at the very bleeding edge frontier. So not only are you pushing that frontier, but it probably gives you the chance to see a little bit further. As you look out, what do you see? What do you think? Maybe you could draw confidence intervals or something for us, best case, worst case or something.

But if you think forward five years, where might we get in terms of what's possible with engineering? I mean, for one, you're asking about competitive programming. The best programmer in history is actually a guy named Gennady Korotkevich. He's got from Belarus. And I would imagine that we have AI that can beat Gennady in the next one or two years.

beating the best programmer in the world at this programming competition. I think that is going to happen. And I think it's going to be like an AlphaGo style moment where it's obviously like, holy shit, like this is truly there. The tech today, honestly, already is folks have been putting out things. Google put out something on the IMO, the Math Olympiad and OpenAI put something out on the IOI, but they're already at a level where they're competitive with some of the best competitors in the world.

Getting to number one is honestly not too far from that. I think what happens to software engineering as a whole is I really come back to this point of what we're saying that all of programming, it really is just about telling your computer what to do. A lot of this work that's happening with AI and AI agents as truly just the next generation of human computer interface. And so what I mean by that obviously is if you think about a programming language, let's say Python, for example, perhaps the most popular language out there.

Python is great. It's beautiful and everything. But if you really just look at it, it is such a stark compromise that you have to be making from both sides. From the human side, obviously, you have to learn how to code. 99% of people in the world don't know how to code. And it takes years to learn all the little details. And here's how you do like a lambda function. And here's how you do this little particular thing. And these are all things you have to learn just to be able to talk to your computer.

And from the computer side, Python itself is notoriously actually slow and inefficient and so on, even compared to other languages. But obviously, compared to how fast you could truly run things on bare metal assembly, if you knew this is exactly what I want to do, you can make all of the systems honestly 10 or 100x faster. And the way I think about it is Python is maybe the best language out there.

by a lot of people's definitions. And really, it's this awkward compromise that we have to make because this is the only way that humans and computers are really communicating today. One of the great things I think that AI will really solve is essentially this translation problem where a human, any human, doesn't have to be somebody who is trained in code or something, can describe precisely what it is that they want to build. And the computer will build the efficient and feature-complete version of that that actually just works and does all these things.

It's going to take some real time to get there, obviously. I think that's the future that we're headed towards. Meaning, just to use a simple example, I, using natural language, want a very, very specific application to stick on my phone or something that maybe within five years will be at a point that me with no particular CS experience can get exactly what I want in a way that works with high fidelity without having to rely on human programmers. Do you think that's feasible? I think that's right.

If I had to put down a number right now, it's obviously very hard to predict. I think five to 10 years is like the reasonable realm for that. One point I would make on that, even the way that we talk about applications, we kind of assume obviously that there's some application that has to generalize. One way to put it maybe is software engineering is so expensive today. It is so, so expensive that the only software that you can actually build that's

that's worth building is basically applications that reach millions of people. It's like, okay, it was worth it to code Facebook because of how many people use the exact same software every day. YouTube or something or DoorDash or whatever. And obviously there are somewhat more niche things, but there's honestly lots and lots of things out there where it would be better served with code. There's communities of...

Call it a couple thousand people. There's little groups or little niches or things like that. There's personalized products even for one person. And so whatever your day-to-day chores are, they really are better served with code. That code can do all of the things that you might need to do and just make all the execution of that much cleaner. The reason we don't have that, obviously, is because...

there's not really anyone in the world for whom that makes sense to write custom software just for their own personal use case every time they want to do something new. But I think we will get to that point where you are just able to execute these things. And maybe it's code on the back end, but from the perspective of us, it's really, yeah, just explaining to the computer exactly what we want and seeing that happen. There's going to be a lot of middle steps, but yeah. I'm curious what you think about one layer down or up, depending on how you think about it, which is the same question

for agents specifically, and whether or not all of these systems are going to cause applications themselves to be less common. And agents we interact with through voice or through text or through other means to do more and more of the things that we currently use applications to do. Since you're building an agentic system, I'd love you to just teach us about agents. Everyone's saying that now. You hear

You hear the word agent in every single call. I feel like that scene in Seinfeld talking about write-offs, you're like, you don't really know what that means, do you? Tell us what agents means to you and why it's such an important construct in this coming age of technology. Like we were saying, GPT came out two years ago. It's a huge surprise for everyone. And I think for a while, all

All of the products that came out of that were basically what I would call these text completion products that work off of that. And by the way, it makes sense, obviously, because the native form of a language model is put in this input text, tell me what the output text is. But that's kind of where you saw every application at first. GPT itself is obviously like that. You had a lot of Q&A products, marketing, copywriting. Hey, I need you to write this paragraph for me that explains X, Y, and Z. Can you just put together a nicely written thing? Boom, here's the output. Customer support, customer

customer is asking something. And then it's like, what would you say here? Here's all the information you need to know. Boom. Code, that was the same thing too, right? Where the first big code products, I mean, GitHub Copilot is probably the biggest of these, is a text autocomplete where it's like, here's the file so far, predict me the next line. And if it is correct, I'll just hit tab. And it'll obviously save me the time to type the line. It doesn't save me the time of thinking of the line because obviously it's still me that's deciding what's correct and what's not correct, but it does save the time of typing the line. And I think

What we're now seeing and what everybody is talking about now with agents, although of course it's very early, is this new paradigm where models can actually act and make decisions in the real world. There's a lot to it and it's how humans do it too. It's not like a human just produces this perfect paradigm

paragraph. Here's everything that I needed to give you and then you're all done. A lot of it is going and interacting with the real world, looking up websites or something, trying out things, seeing what works or what doesn't work. In the case of code, obviously running the code and looking at the output or looking at the logs and things like that. And I think maybe

Maybe the simplest way to put it is in all of these verticals, I think we're going to see this where basically, hey, having a legal Q&A bot is great. But the next step is to have a lawyer. Having something that can answer questions about math problems or something that you have is great. The next step is to have like a full teacher. And obviously, in software, it's the same thing where it's like having the text autocomplete for coding does make you faster yourself as an engineer. But what if you had a whole engineer that you're working with? That's kind of the core of this agentic work.

Maybe you can describe then how literally Devon is built. Behind this question, I'm also curious about taking your experience as advice for other builders who are using core foundation models as a key part of the thing that they're building and building in such a way that is complementary to those underlying foundation models and won't just get subsumed by them. Everyone asks about this.

I'm sure every VC asks every founder, like, how will O1 not just do this? So tell us about how Devon is built and how you think about building with and alongside these evolving foundation models. We are 100% laser focused on solving all the problems along the way of software engineering. And the truth is software engineering is just very messy. My point with that is just that there's this idea of like solving these math problems in a vacuum, for example. There's a lot of research that's going towards that.

But there's also this entire thread between of how do you take that IQ and turn that into something that is actually meaningfully useful for software engineers in their day-to-day work. So there's a ton of stuff that needs to be done on adapting the capabilities for a particular tooling and particular stack. And then there's also a ton of stuff that has to be done on the product side as well. I'll just give several examples of that. A simple one is, obviously, Devon has to be plugged into all your tools. Solving problems in a vacuum is great, but Devon has to work in your GitHub. It

It has to be working in your code base. It has to use your Confluence docs or whatever it is. That's where the information is. It has to read the bugs on Jira. And on top of that, obviously, Devon needs to learn with the code base over time. And so even a really, really, really sharp software engineer, they join your company on day one, they're probably not going to be all that productive, at least compared to

somebody who's been there for five years and knows the entire code base in and out. Devin needs to learn the same way of understanding, hey, here's what this file does. Here's what these functions are for. If I want to do the schema migration, here's step one, two, three to get that done and all that. And then there's obviously all of the actual step-by-step decision-making that goes into it, which is somewhat different. We were talking about fixing a bug and

I think a very natural flow with fixing a bug is, okay, I got a report for a bug. First thing I'm going to do is I'm going to try and reproduce the bug myself and see what's going on. I got to go run all the code locally and try to make the bug happen. If the bug does happen, I'm going to go and take a look at the logs and see what the error was or what happened exactly, what the variables were. And then from there, I'm going to go read the relevant files. Maybe you can make some edits, test things out and whatever. And then I'm going to go test it again and see if it works.

And if that all works, then I'm going to add a unit test to make sure it doesn't fail again and whatever. And all this step-by-step decision-making, it's a very different problem, obviously, from text completing what would somebody really smart on the internet say? How do you use the tools available to you and make a sequence of decisions that maximizes your chance of success? I think the other question you asked was how, as a builder, to think about this in relation to, obviously, this steady stream of improving models. It's a really interesting one because

A lot of other technologies that we've had have been a lot more step function. Mobile phones came out. And the first question was, what are you going to do when everyone has a smartphone in their pocket? In some ways, the core of Uber or Airbnb or DoorDash or all these things. And you can kind of think about, okay, this is the technology shift. And let's think about what happens over the next coming years now that we have this shift. AI is a lot more of a very incremental thing. Thank you.

Here's GPT-3 and 3.5 and 4. And each little technology gets a little bit better over time. And part of what that implies, I think, is, yeah, just really thinking about what are the models not going to do by themselves over time. And a lot of the stack and a lot of the things that we care about, obviously, are around building the

capabilities and the product experience specifically for software. I have no doubt that the foundation models will continue to increase in IQ and increase in IQ and so on over time. But a lot of the detail of all of the tools that you work with, how you make decisions with all of that, even how you communicate with the user, what is the user interface where you're talking with Devin and giving feedback and seeing what Devin's up to and checking in and making sure nothing's wrong and all these things.

There's a lot, a lot of detail, obviously, in that flow that really brings out the inherent IQ, to put it that way. We've always thought about it in that way of how do we make sure everything that we're doing is complementary with the model increases? And by the way, complementary with everything else. The hardware is getting faster all the time. The whole stack is getting more efficient everywhere. And we have our part of the stack and we have always thought about it as how do we do as well as possible in our part of the stack specifically for software?

What improvements are you hoping for from the foundation model companies that would make Devin more effective than it is today in the future?

I should mention, by the way, the Foundation Model Labs are all super, super excited about agents. And it makes sense. I think one thing which is kind of interesting, just practically, is let's say you ask GPT a question. Who is the fifth president or something? You ask one question, you get one answer. And that's a single model query. And obviously, there's a consumer pricing and everything. But if you're going by the API pricing that costs one millionth of a dollar or something like that, something ridiculously low.

Whereas if you're asking Devin, hey, I've got this new feature that I need you to build out. Can you build it and test it and make sure it looks good? Over the course of that, Devin's probably going to be making hundreds, if not thousands actually of decisions over the next half an hour or something while it's going and building that out. And it also just practically means it's actually just a lot more inference on GPUs

And so for that reason, what we'll see over the next while, honestly, even the next one year, I think agents are going to become a lot more widespread. And what we'll see actually is I think the dominant usage of these GPUs, of these language models and everything will be for agents, largely because every single agent query is just so much more magnitude of underlying language model calls than a single Q&A query. And I think what happens next with that, which I think these foundation model labs are all thinking about is, yeah, how do we build our

our models to be as optimized for agents. And so I think a lot of these things of working with multi-turn traces, being able to handle long context outputs, for example, like context is a big thing that people are spending a lot of time thinking about. And with an agent, again, it's like, if you have half an hour of decisions, obviously it helps a lot to know what's been going on in the last half hour. So some of these particular technologies that will really help with that. How do you use Devon in your own building of Devon?

We use a lot of Devon actually when we're building Devon. It really is this new workflow, a very asynchronous workflow. And so I'll just give an example. We do all of our communications in Slack. We have a few channels. We have a crashes channel where every production crash gets reported. We have a front-end channel where we talk about any like front-end feature requests or maybe like, hey, let's switch the flow around this way or hey, this design is a little bit off or things like that, front-end.

things that come up. We have a bugs channel where users are filing bug reports of, hey, this thing didn't work or something, or this thing broke or whatever. In all of these channels, one thing we do every time something gets reported is we tag Devon. You can just go through the channel. Hey, this thing's broken. At Devon. Hey, somebody requested this. At Devon. Hey, we just had a crash. At Devon. Devon

Devin's not perfect. Devin's a junior engineer and often does need a lot of guidance. And I would say probably about 50% of the time, Devin's output on one of these is just like a pull request code that's straight out of the box, ready to merge. And the other 50% time, obviously you can make the call of, okay, do I want to talk with Devin and coach Devin through this? Do I want to go and make some changes myself and just touch?

touch up whatever needs to be fixed, or do I want to just go start from scratch if it's especially far off or whatever? But having this be part of our workflow, you had to report the bug anyway. And so you might as well just tag on an app Devin at the end, and then you get a first pass for free. And so a lot of workflows like this, a lot of it is just thinking about how to put it into the asynchronous workflows.

A similar one, honestly, is if one of our engineers is building some big new project and they're going and implementing things or building the IDE, for example, this happens all the time as an engineer. You're going to go through the code and you're going to see all these little things that are like, man, why did we write it this way? This is so messy. Or like this thing is technically correct, but it is not the most stable way to do this or lots of little things. You get caught up in a lot of these little things as you're building your bigger project. It does distract and it takes away your focus, right?

And what we do now, obviously, is you just kick off Devin from the IDE for each one of these. And it's like, hey, can you go refactor this one real quick? Hey, can you change this to be type safe? Hey, order this in a different way so that there's not going to be a race condition or something. And you're doing all these little fixes and you're kicking them off async. And then you're doing your own project in the meantime. In both of these cases, a lot of it is really just figuring out these asynchronous flows.

It seems like part of the answer to in the near term, you said earlier that maybe 10% of the engineer's time is spent on what a friend of mine calls positive engineering, building things, the creative part, 90% might be on make sure things work more almost like a mechanic than a creative, productive engineer.

So does that mean that in the near term, what Devin and other systems like it are going to help us do is eat that 90% and keep the programmer focused on the 10%? And then what happens beyond that? If we beat the world's best programmer in two years, why do we need programmers? Will there be programmers? What will they do?

So I think it's exactly right. Eating the 90% of all the miscellaneous implementation and debugging and iteration and whatever. I think what happens then is honestly, you just get to do 10x more of the 10%. And

And I think the cool thing about it, we should say it outright, obviously, there's a lot of fear in AI about what's going to happen and what happens next. But I think one of the really beautiful things about software is there really is 10 times more software to go right if we just had 10 times the capacity. Over the last however many years, we've always been bottlenecked by supply. If you talk to any engineering team, I haven't met a single team who said, yeah, we built everything that we wanted to build. There's actually no software left to do. Every team out there has 20 projects that they want to do.

They have to pick three to take on this week because there's just so few software engineers and software just takes so long to build. I honestly think we're going to see a lot of this Jevons paradox where we're just going to be building more and more and more software as it gets easier. It's what we've seen already, but maybe to a somewhat lesser scale. Building software now versus building it in the 80s probably is about 10x easier or so. And this shift will happen a lot quicker, obviously. But over that 40 years, we've gotten a lot more programmers and we've built a lot more software. And I think we're going to see a shift that looks like that as well.

I think in the long term, I think the soul of software engineering, again, is each person getting to decide what are the problems that I want to solve and what is exactly the solution that I want to build. AI is not magic. Obviously, it's not going to go and solve all your problems for you, but it will give you the power to specify exactly what the solution is that you want to have and just be able to have that happen and build that out. Do you think in general that all this means that business people need to start thinking harder about demand and distribution? If we've been supply-constrained,

and that's about to go away. Does that mean that the people that are going to win in the future are those with better demand and distribution? I think in software, especially, we have always been capped by supply and not demand. Obviously, there are a lot of things that are more fixed in terms of their demand. I think there's a lot of interesting questions of what happens to the economy with this and a lot of interesting second order effects. The simplest way to put it, where does all the money in tech go? Where does the money in venture capital or

What does it go into? And it is primarily software engineers. So when you're getting 10x more bang for the buck for every software engineer that you're bringing on, it does change the landscape a lot of how much people can build with products and how much people can do. And then I think down the line, you see a lot of pretty interesting effects with businesses. I see a few different categories of businesses broadly.

If we're being blunt, I think the toughest category to be in is business that's heavily reliant on switching costs and market. A lot of businesses out there, software businesses, that the reason they make all the revenue that they make is because it is so painful to switch to the other products or the other technology or something like that. If Devin is just able to do all that implementation and migration and those costs go to zero, obviously it's going to be a lot more about having the better product. That's going to be a real shift, I think, that companies are going to deal with. At the same time, it's not a nihilist view. There are a lot of things that will be

just as strongly, if not stronger. Network effects, I think, will always be golden. I think data and personalization is going to be even more powerful. So a business that owns the infrastructure, for example, or a business that has all the personal relationships and the data and so on, being able to serve for each person, an individualized product that is truly optimized just for them, instead of this clunky thing that kind of works for everybody and touches all the millions of people, but is not the perfect product for anybody. I think a lot of those businesses will be able to do very well.

What kinds of questions are becoming most common when you spend your time talking to the senior most people at the big established technology companies? There's definitely a lot of folks asking, how do I think about my engineering headcount and how do I work with this? I think we're very much in the paradigm where people like to say this is

AI won't replace you, but someone using AI better than you might. And so a lot of it is how do you get smart and how do you make sure you're on top of the latest technologies and you're ready for the shift, which are honestly going to quicker than most people imagine and getting your whole team set for that and getting things built for that. I think there's also a special case actually with these technology businesses themselves, folks who are building setbacks

SaaS products or things like that, where we talk so much about agents. I actually think one of the kind of interesting product changes is that a lot of these companies are actually going to be serving agents in addition to humans. A lot of great businesses out there that are basically meant for this collaboration, coordination, a lot of interfacing. We talked about companies like Slack or GitHub or Datadog or Atlassian or things like that, where there's a lot of work and compiling that they do. And also a lot of it is about the interface with how humans interact.

I think the way I think about it is right now, those are really valuable tools for agents to be using as well. Sure, you can imagine this nihilist view where an agent is just building everything on its own from scratch. But the truth is, there's a reason that we look at logs the way that we look at logs. The reason that we set up our documents the way that we set up our documents, and it does just make it easier. It makes the problem easier of getting the context that you need. So what will happen actually is we're going to have a lot of agents who are using these products as well.

So businesses that are in that space, a lot of folks are actually thinking about how do I make my product ready for my new wave of agent customers and in addition to all the human customers. I was talking to Martin Casado about these climbing cost curves that technology creates. And the first one that's clearly happened already in AI is like the cost of marginal content creation has started its march towards zero in all different forms of media. It seems like...

With you and what you're building, there's that same curve story happening in the cost of software itself. How do you build a software business in that reality? If I fast forward 10 years or something sufficiently long, why should Devin cost anything? Why won't there just be like an open source Devin that does everything for me? And what do I need to pay for software if we're entering an era of marginal cost of creating software being very close to zero? Yeah.

I mean, I think right now, Devin is really the only one out there in the market that's doing that. But as you're saying, I'm sure there will be lots and lots of competitors pretty soon even. And folks that want to do similar things, Encode is great for the consumer. We were just talking about the 10X engineers and things like that. The truth is that...

Every level of decision-making quality and every level of skill that you can have in your software engineer actually really does have this exponential reward curve, and it makes a big difference. And one of the things that we're getting, obviously, is this ability to really, really personalize Devon to each of their customers. And so like Devon, I had a lot of our customers today, when we worked with Devon, for example, it's just like, Devon knows the entire code base in and out. Devon wrote half the files in the code base. And

And so there's all these details. And obviously, it's not just the lines of code, it's the decision making that went into it and thought process and so on that Devin has internalized and working with the entire engineering team. And it really is that trade off where it is about here's an engineer who's maybe smart and sharp and stuff, but is starting day one on your code base today versus here's an engineer who's been with your code base for years and years and understands every file and has

has built all of these things and knows exactly what your engineers mean when they ask certain things. And there's a real trade-off there. It comes down to these things where it's kind of like pure technology switching costs, where it's migrating all your files to some new thing or moving to some different platform. Yeah, I think that is going to get cheaper and cheaper over time. At the same time, obviously, if you have the level of personalization, if you have the network effects or whatever, it's not about the cost switch. It's really just about you're able to provide a better product.

Maybe say a bit about pricing. This is such an interesting question. It seems like someone at OpenAI put their finger in the air and decided 20 bucks was a reasonable price to charge and a whole business ecosystem has spun up and that seems to be the default pricing. You made a very different pricing choice. It's 500 bucks a month. OpenAI just released the $200 a month option. So we're marching upward, but talk me through how you thought about and think about pricing in this new world. Yeah, we're going to see more of that, obviously. I mean, I think for us,

It's funny, we had a lot of questions with our launch, for example, questions of like, how much do we anthropomorphize Devin? How do we explain the structure of it and so on? But I really do think it is a pretty different paradigm where with agents, it does come down to usage for better or for worse. There's reasons for both the inputs and the outputs for that. For the inputs, like we're saying, I mean,

It is just more expensive to run. That is the case. You're doing more queries and more work. There's more GPU compute that goes into each job or each task or something like that. But I think with the outputs as well, obviously, when you're starting to take on tasks that are real end-to-end tasks, it's just much cleaner and much clearer to measure the concrete impact that something like this has.

I think the proxy that we essentially want to have is value-based pricing, where it's basically, we want to make a lot of the projects or the bugs or the features or whatever that you're building 10x cheaper to build. And so we want the cost of Devin to run on those to be 10x less. It's obviously hard to have a perfect proxy for that, but the proxy that we have essentially is usage. We have a unit called ACUs, agent compute units, and it's basically...

all the decisions that Devin makes, the box that Devin uses, the code that Devin runs, or things like that. And it roughly corresponds to basically for every hour of Devin's work that you're doing. Again, it depends on different machines, but you're paying something like $8 to $12 or something like that. Obviously, Devin can do a lot in an hour. The setup is such that we want it to be 10x cheaper than having to do those yourself. But I think that is going to be broadly the new paradigm when we start to have AIs that are doing tasks instead of answering questions.

Any thoughts on this big fun, almost has become like a cocktail party style question around whether or not we've achieved AGI or not, and whether or not that's a question that you care about? I'm curious what you think about this one. I'm in camp number two on that. We already have AGI.

Someone said this in 2017, you ask if we have AGI, the answer is no. In 2024, you ask if we have AGI, the answer is, well, you got to define AGI, depends on this. I think there is a somewhat recursive definition of AGI, obviously, that people like to say, oh, it's doing 80% of human labor. It's like, well, if you could do that, then the humans would be doing the 20%. So it wouldn't be the 80% anymore. There's some recursiveness. But the way I would say it practically is

I think there's going to be a tale of things where we will push the capabilities, we'll push the capabilities. And for a while, there's going to be lots of little things that are like, well, actually, humans are still better at this one thing. And then we'll solve that thing. And then people will be like, oh, well, humans actually can still do this one thing. And we obviously have a lot of pride as humans and our own species of wanting to find the things that make us unique, and so on. But I think that like practically,

What matters for the world, obviously, is, is the AI good enough? And is it built into products and distributed well enough that it's actually affecting the world and affecting the economy and affecting people's day-to-day lives? In some sense, that question matters more than, is it doing 95% or 96% or 97% of the human's workflow?

It's more a question of, are humans actually out there doing 25 times more because of the AI technology? And I think that actually is much more of a practical question. And in some sense, it's like we're talking about the IQs and things like that. The math problems that these base AIs can solve already, honestly, are huge.

shockingly good among humans would be in the upper echelons of humans, if not quite number one in the world yet. And I think the biggest distinction is actually not whether we are number one in the world or only top 0.1% human or something like that. The distinction is a bit more of, is that level of intelligence actually used

by every human out there in the world and done in a way that's actually giving them a lot of value. That's going to be the big shift that actually gets us to the massive GDP growth, massive efficiency gains, quality of life, whatever you want to call it.

Investors mostly are obsessing over scaling laws because it just has such big implications for public equities, for companies building new products in the private markets. And so that's where they tend to focus. If you are training your attention on what's going on in the foundation model world, what do you spend your time wondering about, asking your friends about, learning about? Where is your eye trained in that world?

Yeah, my hot take on this is I think scaling laws are a little bit of a myth. I think we have made continued progress on these models and push the capabilities and things like that. And there's been a lot of new technologies, obviously, that we figured out. And there's way better post-training and there's some of this stuff, obviously, that's going on reinforcement learning and things like that.

Sure, there is some return to scale, but I think a lot of what we're seeing is basically it is also the introduction of new technologies that are making this more efficient or allowing it to grow with scale beyond that. And so people sometimes like to talk about scaling laws as if it's you

You just sit back and as long as you got the hardware for it, it's just going to get better and better and better. And I think really, obviously we have needed more and more hardware, but it's also been innovations that have actually gotten us there. I think practically with foundation models, we always obviously want to have a pulse on what's going on and what things are going on next. We work very closely with all the foundation model labs. We do evaluation of early models with them. We do custom training of models with them, things like that. It's

It's like we're saying it is an incremental thing. These things will get better. And a lot of the question as a decision maker is to basically just have a very clear sense on what are the capabilities that will get better and what are the either the capabilities or the product or the human experience of it that will always need to complement these for you to deliver something of value to your customers.

I'm really curious to learn a bit about the formative experiences for you as a business person. Obviously, you had, we've talked a lot about formative experiences as a technologist, building cognition and also IOI and these other competitions for the first part of your life.

I'm sure everyone's seen the video of you when you were 12 or something winning math competitions. Obviously, you've had some pretty cool formative experiences. What have those been on the business side? For example, I'm interested to hear about your experience with Lunch Club and what you were building, why you're building it, and what that experience or that chapter of your business career taught you.

Yeah. Lunch Club was my first company. I dropped out of school actually to start the company. It was a great experience. First of all, it was AI for professional networking. We made millions of meetings happen and it was super fun team and product to build. I would say one of the big things, honestly, that I learned is, it sounds simple, but it really does come down to the fundamentals. I feel like the common pieces of startup wisdom that everybody talks about, in some sense, they're so obvious that it's not obvious that

it's not even worth mentioning if people feel that way where it's like go as fast as you can. Everybody says that. Everybody knows that. Never compromise on recruiting. Stay as close to your customers as possible. Focus on building a product that people love and want. These are almost tautologically true. But the thing is,

The reality is there's always another gear that you can kick into with your fundamentals. And that's how I've really thought about it now, having gone to see it. Simple example is like, everybody says you can go fast. In our case, we incorporated the company in January. We were building the first versions of the products. We got it out to the first group of initial users in February. We did a launch in March. We raised a big round in April. We did a big partnership with Microsoft in May. While we were doing that, we got on a lot of enterprise customers.

We grew the team. No matter how fast you're going, and every startup in the Valley obviously thinks they're going quite fast, it's still worth it to really push for even faster. And no matter how close you are to your customers and how deeply you understand their problems, you're probably not going to regret spending even more time with them and understanding that deeper. And no matter how high your bar is for hiring, you're probably not going to regret trying to go for folks that are even higher still. And I think that is the biggest lesson I've learned with startups.

the competitive landscape, does it feel like this big, exciting greenfield or does it feel like

Battle lines are being drawn and insanely competitive for every available opening and sharp elbows. Characterize what the playing field feels like right now. You all work out of this amazing house that I've been to. It's such a cool environment, which feels so startup-y. And yet the next day you're with Microsoft. The contrast here of real true startups in startup houses and raw cultures that are emerging

They're interacting like you are with the biggest players in the world of technology inside of a few months. It's just a bizarre set of circumstances. How would you describe the playing field to an outsider? Yeah.

I think it's still very greenfield. I mean, the technology itself is so early. We're like one or two years in, in some sense, to all of this stuff really being valuable and getting there. One thing that I think about a lot is software is just so big. We obviously know that, but there's so many facets of software and so many details and so many different things that can be done and can be built. And so one way that I think about it is each of the use cases that we're talking about, there are enormous and really, really meaningful businesses that are built and just do that use case.

A lot of folks are talking about AI for data observability. For example, Datadog is a great business. It's making billions of revenue. It's something like a $50 or $60 billion market cap. It's doing very, very well and growing quickly. And as you can imagine, there's so much to do with AI for data observability in some form. Incident response is a huge thing that's only going to get bigger and bigger. And there's a lot of great businesses in the space, like PagerDuty is a great business. But AI is going to make these things, again, orders of magnitude bigger. It's true with testing. It's true with modernization projects. It's true with microcomputers.

Migration, a lot of these big guys like Microsoft, Amazon, Google, or anyone, the amount of effort they have to go through to migrate customers onto their platform even today is enormous. It's billions and billions of dollars every year. So all of these are just such enormous spaces, and there's still so many questions even about the product form factor. And as the technology gets better and better and better, the kinds of use cases that we'll be able to support as a space are only going to get bigger.

What do you think the world is overestimating and underestimating right now about your whole world, about the automatability of software engineering? The speed of it is probably a big one. One thing I always think about, if we're just looking all the way across AI, for example, I think agents in general are going to really work everywhere. Code is one of the first for a lot of reasons, but I would almost argue one of the first agents, if you want to call it, that actually was self-driving cars. That's

That's an actual AI that's making decisions in the real world and taking in feedback and iterating with that. People are familiar with the story of self-driving cars where I actually lived in Mountain View in 2014. I would see the Waymo's driving around all the time. I never got a driver's license. I had just gotten started working at the time. I'm not going to need one. They're basically there. I'm sure I'll be able to use it.

We did finally get there. I think there's still a lot of room to grow, obviously, but it did take a lot of time. I think the difference between self-driving and code and honestly, a lot of other applications I think we'll see over the next year is I think with self-driving, you really, really need to have it be like 99.99999% sufficiency. And there's not a world where it's like, or there is, but it's not as big of a difference where it's like human and machine combined can do a lot more together.

And I think that's people's typical example. It's like, okay, you know, there's going to be all the edge cases to figure out and all that, which is true, by the way. But the nice thing with software, for example, is there actually is just a really great way to do human plus machine and do a lot more. That's the case with Devin. Devin is not the 99.9999% solution. It looks a lot more like the 2014 Waymo or honestly, probably even a little earlier than that. But the thing is,

Having Devin take a first pass or having Devin send you the code and you take a look and give a review or whatever it is like that, where Devin is doing the 90% of you're doing one nine instead of six nines. It's still super, super useful. It actually does save you 90% of the time, whereas you're not going to get into a car that is only 90% consistent.

And so I think there's going to be a real form factor difference in how soon we're going to see it actually really transform practical software, not just because of the capabilities, but also just because the product space is better set up for that.

What was the lesson that Lunch Club taught you about the world of entrepreneurship? Was it something around market? Was it something around product? If you think back on that experience, what's the big takeaway and how are you applying that learning to what you're doing at Cognition? Probably the biggest one that stands out, honestly, is Cognition.

Sometimes it's easier to solve a bigger problem than it is to solve a smaller problem, which is why folks talk about this. If you're going after something that's truly enormous and it's the thing and it's the most exciting thing, you'll be able to bring together a group of people who's all really excited to make it happen and is really passionate and dedicated to pushing the frontier in a way that actually sometimes make it easier. Our team, for example,

Our team is about 20 people right now. Of those 20, I think 14 of them actually have been founders themselves before. And it's an interesting one where we've all done companies. And honestly, it's a really, really talented team. And a lot of these folks would get a blank check to raise money if that's what they wanted to do. And they wanted to do their own thing. But a lot of it is just how do we come together and build something that's really great? I think that's one of the big lessons for me.

What has been the most difficult or stressful moment so far in the one-year cognition journey? Oh, there's been a lot of it. Let's do a few then. I love these. One of my favorites...

All of these launches, you know how it is with startups, obviously. Things are so last minute. So we had the GA launch this week, and then we obviously had our initial product launch back in March. And for both of those, the few days leading up to the launch were some of the most stressful, but just the most memorable experiences. And our launch in March was huge. There's so much you have to do to get into it. And obviously, there's the product itself and testing with early users and making sure everything is truly...

working the way that it's supposed to, getting the info ready for it, setting everything up. And then there's obviously all of these other things with filming the video, think about the blog post, think about the content, getting things out there, working with customers to get testimonials and things like that. And our March launch, actually, we were doing all of that in addition to, we were doing a fundraise right before the launch as well, actually. And so we were doing that. We had candidates who were in the pipeline, pretty meaningful candidates that we were working on and figuring out and everything during that as well. And so that weekend,

It was a very fun time. That launch was actually, it was an article that went out by Ashley Vance, Bloomberg article about our company. And that was the article, obviously, that marked the launch. It was supposed to go out at 6 a.m. Tuesday.

We asked them to push just a little bit later. We got it to be 9 a.m. Tuesday instead of 6 a.m. We needed those extra few hours. We weren't done filming the video for the launch as of that night at 1 or 2 a.m. We were still recording lines. And then there was all the blog posts and putting that all together. It was a crazy weekend. Yeah. Did not get very much sleep that night, basically.

It's just a really fun moment to get together and really do everything together. It's one of the reasons, actually, I think that us all just living and working in a house together is great. It's because there's something about the shared experience, sharing the late night energy and everything. I feel like it wouldn't be quite the same at an office. And yeah, this launch as well, you know, actually, we were all in Utah.

which was kind of hilarious, actually. Last minute decision, it's like, okay, we're going to use that. And we got on a plane that day, we found a cabin that would fit us all, got into that. And we were all just working on the launch. And it was the same story always. Even having done this, we got no riser. And so we were still filming at 2am the night before and putting things together for the launch and getting all the content ready and getting the info ready. And obviously, in this case, it's a full GA thing too. So it

We got to be ready to handle 10x the load or if more than that. So all these kinds of things that we had to do to get ready. And those are some of the funnest moments, honestly. I'm just extremely interested in launching things. I think there's so much rolled up into it that it's like business encompassed or something. Can you tell me the best and worst part about each of those two launches in retrospect? Honestly, the experience with the team is the best part. Just getting to build it together and do it together with the team. It really is the soul of

startups. Yeah. The worst, I mean, there's scares and stuff that happen all the time. The night before, it's just like, hey, we did our load testing. This didn't work at all. All the info went down. There's no way we're going to be able to handle the load. We were expecting to handle this many concurrent Devon sessions per minute. This did not work at all. And the last minute things we have to do, the onboarding flow, for example, last minute changes that we had to make. My co-founder, Steven, actually put out a post talking about how we shipped the whole thing where you could do a Slack connect with us and work to

directly with our team. That got started about 1 a.m. the night before the launch and shits the morning of the launch. And Devin built a lot of that because obviously we needed the extra bandwidth from Devin. In the moment, every single thing that you're optimizing for is how do we make this not incredibly embarrassing for us all? And sometimes you just have to put it out there and do your best.

How do you think about something like 01 Pro Mode and people using that to do coding versus Devin? Yeah, I think these tools are great. And I think a lot of companies are thinking about code and are focused on code. And like we're saying, code is such a big thing. There are a lot of cases where, yeah, sure, it's like you want to build something, a single file of code out of the box, and you want to think through particular problems or whatever, and have that be a really great single file of code or something.

Where, yeah, you can just go to Olin Pro and consult it or an AI stack overflow, if you will, where it's like you can paste in an error and it'll give you its best guess from what it has. Here's what I think the error is.

With Devin, there's a ton of just these iterative flows that make a lot of natural sense. And so, for example, if it's not just writing a single file and that's all, but it's like, hey, I need you to work in my code base and build a new feature and plug it into all the existing stuff and test it out to make sure that it works, then obviously all of those iterative steps, those are the things that Devin is doing for you.

And similarly, if it was a debugging thing, like an AI stack overflow thing, for example, actually one of our first Devon moment, I want to say is we started building this as more like research type technology and just trying to understand multi-turn agents, iterative decision-making and stuff. And eventually it made sense to build into the Devon products. But one of the first things that really convinced us is we were trying to set up our database for our own work. We were setting up MongoDB and anyone who's done it knows what this flow looks like, basically, where it's just

Give you a set of commands, you try it and then it doesn't work. And then you read the error and you copy paste it and you try to figure out, you Google what's going on. You find something and you run that and then you run into a different error. You do that 10 times until the thing works.

Everybody said it was always a little bit different. And there's always these little details that will get you. In this case, we'd spend a while trying to set up MongoDB and it wasn't working. And so we just gave it to Devin. We're like, hey, can you just please get this working locally? And then Devin did it. And that was the thing. So what we were saying earlier, it's not just a stack overflow where it just gives you the one answer of, oh, you ran into this error. Oh, maybe you should try this.

Like Devon sees the error and says, okay, let me see what's going on. Let's check what ports are open. Let me see what the schema of this database is. Let me see if the socket connection is actually set up. It's going and running all those commands to go look at that. And then it uses that information to decide, okay, what's the next thing I'm going to try? And then it tries that. And if it run into a new error, then it's able to debug that. The integrated autonomous flow is I think the biggest difference. How hard is it to build that? I'm not technical enough to ask the best version of this question, but

But if I think about part of the secret sauce being the iterative agentic nature of the whole thing, what are the hardest parts about building that capability? The obvious one is the model capability itself.

The models are roughly like, here's what a smart person on the internet might say. But if you're now thinking, okay, I'm doing a step-by-step process to make decisions to maximize my chance, not now, but after a hundred steps that I'll have solved the issue, then it's obviously a pretty different thing that you're solving for. And so a lot of it goes into the capabilities. But I think beyond that too, there's so much with the infra and with the whole experience that gets quite involved, as you can imagine. And so, for example, Devon has its own machine that it works in.

And a lot of the features that you'd want to support, for example, like rolling back state of the machine or dealing with hot loading these quickly and being able to spin up a Devon instance with a new machine immediately, being able to go and navigate through all the previous decisions and go back and forth between these different things or the ability to work with different Devons in parallel or things like that. Yeah, there's a lot of infrastructure problems that go into that as well. And I think it just comes down to this thing where

The software engineering is so messy. And so it really is just so different from doing problems in a vacuum and building all of these high touch components with the real world where you're actually able to do these things in the real world. You're actually able to use all your integrations. You're actually able to maintain your own machine state or work within your get checkout or all of these things.

A lot of the mess of that is the part that I think that really comes out from us just really focusing only on software engineering. If you had to put your most sci-fi lenses on for the future of technology in the world, what would you do?

What are some things that you maybe talk about with your team just for fun that don't seem feasible yet, but might be feasible in the future that get you the most excited? My co-founder Walden has this great line, which I've always really loved, which is we've been playing in Minecraft survival mode for so long, and now it's going to be time for Minecraft creative mode.

- Describe what that means for those that don't have 10 year old boys. - Yeah, yeah. And so basically, Minecraft is a sandbox open world game and it's meant to be a simulation of life. You can go and grow food, you can go and forage, you can mine, you can do all these different things, you can build stuff.

It's kind of funny because there's two major modes. There's one which is survival mode, which is you are actually just have to deal with the enemies that come at you. You have to go build your own shelter and make sure you have enough food to stay alive and all these things. And there's obviously creativity within that, things that you can do, but you are also constrained by how hungry your own character is or whether you're being attacked or something like that. Whereas Minecraft creative mode is basically a mode where

you're able to put together whatever you want. It's not like I didn't find any iron ore today, and so I'm not going to be able to build this thing. It's like, I want to build this thing. I want to try this new thing. I want to set up this really cool thing. There's folks who have set up entire computers and computer programs in Minecraft, which is insane. It's basically just focusing on the ideas and the imagination rather than the execution.

It's not about, okay, now I'm going to have to go spend another two hours going and collecting gold ore because I want to build this thing. It's like, I know exactly what I want to build. And now I have the freedom to just make that come to life. That's honestly, I think the big dream of AI as a whole is to be able to free up humans. The point of AI isn't to go and just solve all of our problems for us and leave us with nothing. I think it's to give us the power that, or the kinds of problems that we want to solve or the things that we want to build to give us the ability to actually just turn that into reality.

Yeah, it's so funny to imagine. I'll use my own experience. When I visited you, you asked, I think the demo is incredibly powerful. So you asked me to come up with an application idea. I think it took eight minutes to build and we watched it do it. Watching the iterative nature of it feels like a real aha moment because it really does look like it's in hyperspeed doing something that a person would do as it builds something. But maybe the more profound realization I had was it built the thing that I described in natural language.

But it wasn't until I saw the thing, the little stupid application that I made up, that I realized why that idea was bad. When I said the idea, it sounded good. And when I saw the thing, I knew why it was bad. And so I would say the thing that excites me is not that I can make my ideas sound

but that I can get better ideas, which I think is pretty underrated part of this whole thing that just like a computer program is going to be faster now, anyone with ideas is going to be better and faster because they can see the thing and see why it stinks. Does that resonate? That's a great point. It's very much the idea of turning 10% of time that you get to spend on your ideas into 100% of time, whereas before it

You got to get a team together. They're going to go and build the whole thing. And you spend a couple of days or something building this whole thing. And then you get to do your next step of idea iteration. And by the way, Devin's still honestly a lot of room to grow and it's going to take a lot of progress, honestly, to get to this future. But to get to a point where it's basically just you get to do the ideation work, you get to iterate against that, you get to do the new idea, you get to do the new idea and so on. And instead of these 90% implementation with the little 10% chunks in between, you just really just get to spend all your focus on the ideas. I'm super excited about that.

It seems like one of the defining aspects of cognition is the degree of friendship and trust between the leaders is extremely unusual. Maybe you can describe the origin of that. But when you're moving fast, trust is really valuable. And I'm just curious to hear about that side of your building experience so far. Pretty much all of us on the team have known each other for years and years and years. Steven, Walden, Russell. I've known these guys for a long time and the whole crew as well.

I had always wondered about, okay, is it going to be awkward to work with your friends? There's lots of things that come up. And if anything, it's honestly the other way around where it's easier to have tough conversations because you know, obviously...

We were friends before all this. All of this that we're doing is coming from a place of we are just trying to take care of each other and make something great together. When you both know that's true and when that undertone is always there, it's a lot easier to just call out stuff like, hey, I think this is totally wrong or we need to change this thing right now or things like that. What in your experience talking to investors, present company included,

What do people not ask you about the product, the company, or the vision that they should ask you about? Yeah, it's a good question. I think one thing that's definitely different talking about it versus feeling it is that folks sometimes talk about or ask about really is just kind of the experience with the team. I think the day-to-day, it's getting the product out there and seeing what folks are saying about it and talking with our users and things like that, obviously. But ourselves, I think...

just how we do it and how we build it as a team is truly special, I would say. I think we have some of the smartest and most ambitious and most capable people, and yet somehow they're all such low ego people as well. I think that is truly the, that's the soul of it for me is the 20 of us in a house together trying to do something meaningful. It's a big difference, I think, talking about it versus really seeing it live and so it

One of the things that we often do as we're really pushing with it is having folks at the house just come by and see what it's like. It's going to be 1 a.m. or 2 a.m. and stuff and people are still building and there's a lot of discussion and stuff that's moving quickly and everything. And that's probably the thing about building the company that I'm most grateful. What uncertainty?

interests you the most that's outside of your control? Interesting. Yeah. One of the uncertainties I actually think a lot about is broad adoption of

It seems very clear to me now that the technology will get there or maybe already has gotten there in a lot of cases to be extremely impactful. And I think the rate at which we see it happening in different industries is going to be pretty fascinating. We can both think of things like medicine or law or things like that, where you can certainly imagine it will take time to do that. And then there's obviously just the whole world of distribution.

I will call it a win where everyone in Baton Rouge, Louisiana is actually having a significantly better life because of the work that AI is doing. It's a different curve everywhere. Obviously, I think we're lucky in software that developers are just kind of inherently curious and always want to try to do it. But I do think the rates of adoption, it's going to be a real question with this technology because I think the pace of progress is going to be such that technology in some places is going to be way ahead of reality.

When we were first together, you did this insane card trick. Are you willing to do it for us? Yeah, let's do it. Tell us the origins of this trick. Yeah. So this is a card trick. So I wouldn't even call it a card trick. It's more of a card game. And so growing up, I always used to play this game called 24. It's a little math game. And I think it actually helps a lot with just getting the fluency in math. I think it's one thing to learn, okay, here's addition, here's multiplication and whatever, which is obviously super building blocks.

But also a lot of it is just this creativity and this ability to reason with numbers and experiment with numbers. And so I'll just show what the game looks like. And so the idea is you have four cards and there are four numbers. By the way, Jack is 11, Queen is 12, King is 13. So it'd be 4, 7, 8, and 12 are the four numbers that we're dealing with here. And the idea with this is you can use the four numbers in any order, in any combination. You got to add, subtract, multiply, divide any of these. And your goal is to

put together an equation that uses all four of these and makes the number 24. And so here, what I would do is 12 minus eight is four, four times seven is 28, and 28 minus four is 24. And so you can see how it uses a lot of this creativity where it's like, hmm, messing around this number four,

24 is a multiple of four. And so if I could just make a six from the other three, then maybe I could do something. It turns out that you can't quite do that, but you can make a seven and subtract another four out, things like that, where you're playing with these ideas. And so at these math camps and programming camps, we used to play this game all the time. And eventually we got to the point where you learn all the combinations of 24. There's only about 1300 total combinations that you have to know. Not all of them are possible, but for the ones that are possible, you know them. And so what we did next was we did a thing where you take six cards and you're making 163 and

And so I'll just steal another six cards. And so in this case, for example, what I would do is 11 times two is 22, 22 minus three is 19, six plus 10, 16, 16 times nine is 144 and 144 plus 19 is 163.

And so as you can imagine, it's a lot of fun. We would play this at the math camp during the team selection training for the US team. Basically in between contests, we would just be playing this game all the time and doing this. And eventually you get to the point where 163, you have a lot of the patterns down and you see things. And so then the next thing that we did was we do eight cards and we make the current year. And so now it's 2024. And so let's just go ahead and do that.

So 13 minus 12 is one. And this is actually relatively easy. I'm just gonna take one times one times one and we still got a one. And 13 plus 10 is 23. 23 times 11 is 253. 253 times eight is 2024, which times one is still 2024. Incredible. And eventually you learn all the patterns with that. And so obviously the next fun thing that we have to do is we have to do a custom number. How many digits? Four digits? Yeah, yeah. Let's do seven, five, three, two.

Seven, five, three, two. Okay, so it's a little bigger, so we're going to do nine cards, okay? Okay, so 11 times seven is 77. 77 minus two is 75. We got two tens here, which is a nice hundred. And so that makes it 7,500.

And then from here, nine times four is 36. Three plus one is four. 36 minus four is 32. So 7,500 plus 32 is 7,532. Pretty amazing. I don't know how you do it. What are you doing? What is the method behind calculation? Or is it multiple methods? A lot of it is intuition, I guess, that you build over time.

With 163, for example, a lot of these little things where I know, okay, 11 times 15 is 165. So if I can make an 11 and a 15, and then I can make a two left over, then I could do 165 minus two, and then I'm searching for that. And so based on the numbers I see, I'm solving for something like that.

And it depends a lot on the patterns that you get and everything. It's actually, I think, a great game for kids. This is my favorite game as a three-year-old kid or whatever, playing 24 with the four cards, because a lot of it really pushes the number sense. It's not just a question of can you add two numbers, but can you really reason around that? And like, you know what that means and be creative with the numbers that you have, which I think is the core skill that you really want to learn, not just the mechanical part.

I think what you're building is so cool to me because effectively what you're doing downstream is unlocking the creativity of a lot more people than currently have the ability to build things through computer science and programming. And I just think that's an incredibly cool and amazing thing for the world. I hope you and others like you are successful in doing so. When I do these interviews, I ask everyone the same traditional closing question. What is the kindest thing that anyone's ever done for you?

The number one thing that I come to throughout my life is honestly just all the mentors that I've had and how much effort they truly put in. Even as a kid, as you can imagine, like training for the US team for the international thing. It was funny because I never had a coach. I never had formal this is the person. But I did have a couple people and one in particular who just really invested in me and believed in me. And you know, he was the one who really just pushed. I'm the high school kid. I had

I had a lot of other things that I wanted to do besides just doing math all the time. And he would always push me and be like, you got to practice. You got to do this thing. Every time I would get something wrong, he'd be like, yeah, you can't make that mistake. Here's how we're going to do these things. And all the time that he put in is everything, honestly, comes from that. I feel like I'm really only the sum of all the mentorship that I've received and

I think the same is true in startups. I can think of a few people. And again, I can think of one in particular who's really just coached me through a lot of this and guided me through a lot of these things and been the one to give me the harsh criticism and the feedback and hit me with the things that matter and push me to do better and to do more. And people often think of kindness as sacrifice. And I do think actually that often a lot of the greatest kindness is about building more together and doing more together. Scott, thank you so much for your time. Thank you so much, man. Thank you.

If you enjoyed this episode, check out joincolossus.com. There you'll find every episode of this podcast complete with transcripts, show notes, and resources to keep learning. You can also sign up for our newsletter, Colossus Weekly, where we condense episodes to the big ideas, quotations, and more, as well as share the best content we find on the internet every week.

We hope you enjoyed the episode. Next, stay tuned for my conversation with Katie Ellenberg, Head of Investment Operations and Portfolio Administration at Geneva Capital Management. Katie gets into details about her experience with Ridgeline and how she benefits the most from their offering. To learn more about Ridgeline, make sure to click the link in the show notes.

Katie, begin by just describing what it is that you are focused on at Geneva to make things work as well as they possibly can on the investment side. I am the head of investment operations and portfolio administration here at Geneva Capital. And my focus is on providing the best support for the firm, for the investment team. Can you just describe what Geneva does? We are an independent investment advisor currently about over

over $6 billion in assets under management. We specialize in U.S. small and mid-cap growth stocks. So you've got some investors at the high end, they want to buy and sell stuff. And you've got all sorts of investors whose money you've collected in different ways, I'm sure. Everything in between, I'm interested in. What are the eras of how you solve this challenge of building the infrastructure for the investors? We are using our previous provider for over 30 years. They've done very well for us. We had the entire suite of products.

From the portfolio accounting to trade order management, reporting, the reconciliation features. With being on our current system for 30 years, I didn't think that we would ever be able to switch to anything else. So it wasn't even in my mind. Andy, our head trader, suggested that I meet with Ridgeline. He got a call from Nick Shea, who

who works with Ridgeline. And neither Andy or I heard of Ridgeline. And I really did it more as a favor to Andy, not because I was really interested in meeting them. We just moved into our office. We didn't have any furniture because we just moved locations. And so I agreed to meet with them in the downstairs cafeteria. And I thought, okay, this will be perfect for a short meeting. Honestly, Patrick, I didn't even dress up. I was in jeans.

I had my hair thrown up. I completely was doing this as a favor. I go downstairs in the cafeteria and I think I'm meeting with Nick and in walks two other people with him, Jack and Allie. And I'm like...

Now there's three of them. What am I getting myself into? Really, my intention was to make it quick. And they started off right away by introducing their company, but who they were hiring. And that caught my attention. They were pretty much putting in place a dream team of technical experts to develop this whole software system, bringing in people from Charles River and Faxit, Bloomberg. And I thought, how brilliant is that to bring in the best of the best?

So then they started talking about this single source of data. And I was like, what in the world? I couldn't even conceptualize that because I'm so used to all of these different systems and these different modules that sit on top of each other. And so I wanted to hear more about that. As I was meeting with a lot of the other vendors, they always gave me this very high level sales pitch. Oh, transition to our company, it's going to be so easy, etc.,

Well, I knew 30 years of data was not going to be an easy transition. And so I like to give them challenging questions right away, which oftentimes in most cases, the other vendors couldn't even answer those details.

So I thought, okay, I'm going to try the same approach with Ridgeline. And I asked them a question about our security master file. And it was Allie right away who answered my question with such expertise. And she knew right away that I was talking about these dot old securities and told me how they would solve for that. So for the first time, when I met Ridgeline, it was the first company that I walked back to my office and I made a note and I said, now this is a company to watch for.

So we did go ahead and we renewed our contract for a couple of years with our vendor. When they had merged in with a larger company, we had noticed a decrease in our service. I knew that we wanted better service.

At the same time, Nick was keeping in touch with me and telling me updates with Ridgeline. So they invited me to Basecamp. And I'll tell you that that is where I really made up my mind with which direction I wanted to go. And it was then after I left that conference where I felt that comfort and knowing that, okay, I think that these guys...

really could solve for something for the future. They were solving for all of the critical tasks that I needed, completely intrigued and impressed by everything that they had to offer. My three favorite aspects

Obviously, it is that single source data. I would have to mention the AI capabilities yet to come. Client portal, that's something that we haven't had before. That's going to just further make things efficient for our quarter-end processing. But on the other side of it, it's the fact that we've built these relationships with the Ridgeline team. I mean, they're experts. We're no longer just a number. When we call service, they know who we are. They completely have our backs.

I knew that they were not going to let us fail in this transition. We're able to now wish further than what we've ever been able to do before. Now we can really start thinking out of the box with where can we take this? Ridgeline is the entire package. So when I was looking at other companies, they could only solve for part of what we had and part of what we needed.

Ridgeline is the entire package. And it's more than that, in that, again, it's built for the entire firm and not just operational. The Ridgeline team has become family to us.