How AI is changing national security w/ Kathleen Fisher
55:01 Share

Hey, the level here before we start to show, I have a quick favorite ask if you're enjoying the dead eye show, please take a moment to rate and leave a comment in your podcast APP wish episodes have you loved and what topics do you want to hear? More of your feedback helps us shape the show to satisfy your curiosity, bring amazing gas and give you the best experience possible.

Back in google, we use a framework called the helmet caister, a set of probe in questions designed to help you think both big and clearly. Questions like, what are you trying to do using absolutely no jargon? How is IT done today? What's knew about your approach and what's IT think these questions help access which risks are worth taking.

But this framework for bold strategic thinking didn't originate a google. IT began with darpa, the defense advanced research projects agency. In a world where you think governments and imagine a massive bureaucracy, darpa is the exception to the norm, a nimble agency known for its relentless pursuit of the impossible. Darpa has expired the most unconventional pats s from robotics to biotechnology, always willing to chase even a glimmer of possibility.

And that same appetite for risk is what LED to ground breaking innovations that shape our modern lives, things like GPS, the internet and even the predecessor to thei today, the need for that kind of thinking is more critical than ever as we stand on the frontier of a new chAllenge, cyber security in the age of A I. While most AI models come with built in safeguards, bad actors are constantly finding ways to circumstance them. And as A I becomes more sophisticated, so do the tactics used to attack our digital infrastructure.

We're not just facing more convincing A I deep fakes, spreading misinformation and enabling ransome where worsening disruptions to global supply chains with entire cargo ships hobbled nowhere, government sponsored hackers are infiltrating electricity grids and sensitive personal data is being ked with alarming regularity. So now we find ourselves asking the same kind of questions darpa has always asked. How do we stay ahead? How do we build systems that are nearly impossible to breach because patching vulnerabilities isn't enough anymore, not in a world where a is accelerating th Epace o f a ttacks f aster t han w e c an d efend a gainst t hem.

We need to look beyond today's threats and methods and ask ourselves the bigger question, how do we secure the systems of the future? I'm belval to do. And this is the tedy issue where we figure out how to live and thrive in a world where A I is changing everything.

Add a little curiosity into your routine with ted talk daily, the podcast that brings you a new ted talk every weekday in less than fifteen minutes. Today, you'll go beyond the headlines and learn about the big ideas shaping your future coming up. How A I will change the way we communicate how to be a Better leader and more listen to detox daily wherever you get your podcast.

In this episode, i'm joined by doctor kathleen Fisher, director of the information innovation office at darpa. Cataline has been at the forefront of ground breaking programs aimed securing our digital future. From unhackneyed t are to deep fake detection tools. SHE offers a glimpse into the cutting edge work being done to protect our way of life in the rapidly evolving world of cyber security. Kathline, welcome to the show.

My pleasure. Thanks for having me are right?

So a lot of people talk about the A I race as this new space race, but there was the actual space race that LED to the genesis of the organization you work for. So for the uninitiated, can you talk a little bit about the origin story of darpa? sure.

So the late one thousand nine hundred and fifties, the soviet union launched the sputnik satellite, and this took people by surprise. There's a little bit of debate of how much IT took various people by surprise. But in the interest of not letting a crisis go to waste, the U.

S. Decided that, uh, we didn't want to be taken by surprise. And to that and created darpa. The creation of darpa is a two page, one and a half page document that launched the agency.

And what that document does is IT gives darpa the authority to write contracts with pretty much any kind of organization, big companies, companies, government organizations and universities to go out and invent new technology, international security. And that LED to the order to the organization ation, that became darpa. And the organization has a couple of organizing principles that has LED IT to be amazingly successful since, like one thousand hundred and fifty nine.

So one of the things is that IT IT does all of its work by contracts. So darpa has very has no labs. IT has no permanent technical staff.

IT creates projects that go out to explore technical apology that something might be possible for national security. And IT goes and hires on contract people to explore whether that technical apotheosis possible or not. And so that allows you to be very nimble um because we don't have to have long term commitment to a particular lab infrastructure to particular people. And with that, we've done things like create the initial technology for the internet, create the initial technology to mini ze GPS, create the technology that allowed that LED to mra vaccines that was very instrumental in the response to the COVID pandemic.

Yeah, even syria in our pockets, right? Most people don't know the theory assistant they use every single day. I know and love the origin story also is a darpa exactly.

So right? So part of the I like we're created technology for national security, but IT turns out a lot of technology that is created for national security turns out to be super .

useful for consumers as well. A N dual use technology seems to be a phrase that's thrown around a lot, lady, special especially when he comes to A I and some kind of curious like um dark has been described as working on a lot of these high risk, high reward projects in the last decade. What are some of the highlights and maybe even low lights that come to mind when IT comes to your domain in darpa?

Yeah, so high risk, high reward. So dark has been involved in A I since the very beginning of A I and there's a ton of a high risk, high reward research in A I. But another area of high risk review research is in how do we build systems that are much harder to hack into.

We kind of come to the mindset that software, of course, is going to be hacked, and we just adjust when it's packed, we clean up the mass and we patch the vulnerabilities that we find. But dark invested in two thousand and eleven in a program called hackers about using formal methods to produce where that would be much harder to hack into. And people have been trying to use formal methods to build software for a long time, and i'd been basically only been useful for toy problems.

Before doctor Fisher continues, let me explain formal methods. So bugs and code are very common and usually fixed over time. Think about apple's IOS updates.

While minor bugs might just crash a browser in critical systems like medical devices or airplanes, they can be catastrophic. Formal methods rely on mathematical techniques and proves to guarantee that these systems don't fail. Now, formal methods have been used for decades, but mostly in simple low level code, or what doctor Fisher referred to as toy problems. With the advent of hackers, however, doctor Fishery demonstrated that formal methods could be applied to far more complex systems, making them nearly on hacker.

But in two thousand and eleven, with the hackers program, darpa demonstrated that, in fact, formal methods were ready for prime time. So around that time, researchers at the university of washington and the university, santiago, so you kono is washington and seven and savage at the universe, california and ago, showed that you could remotely hack into and take over control of an automobile, which is clearly a bad thing, right? And an OTA BIOS kind of a stand in for all sorts of different kinds of cyberspace sc systems, systems that impact the real world but are controlled by software.

There's lots of good reasons why you want to have a car, uh have soft ign n you can do things like anti like breaking. You can do things like unlock the car remotely if you lock your keys in the cars. Like there's lots of good reasons why software is in control of cars now, but they're downsides, right? If you can hack in and take over control, there's all sorts of bad implications of that.

And so the hackers program was like, can we use formal methods based approaches to make software for vehicles and other related kinds of systems much harder to hacking into? So at the beginning of the hackers program, we had a professional red team try to break into a quad option and boeing and man little bird, so a quad oper commercial off the shelf, open source system, bones am a little bird, a helicopter military, relevant, big enough to fly with two pilots on board, but also can fly autonomously. And with red team showed was that they could break in and take over control of both of those systems to be able to basically fly both of those systems.

So like what you see and stefan had shown on the automobiles and then the formal methods, researchers got to work and analyzed what the security of the quiet option was and what parts of the system were important in what parts were not so important for security, and basically read a lot of the system on a lot of the software. And the quicker the red team had been watching the whole time. So they knew basically everything there was to know about that quick ter, and they were charged with, like, do IT again, like take over control of the quote option from office system and they couldn't do IT.

So like that kind of shatters the myth that like, no matter what, red teams will be able to get in A P. M at darpa, who was an expert in penetration testing in his assessment of that system, was that that was the most secure u AV. On the planet at that time.

And the the director of darpa thought that the program was going to fail miserably. That kind of shows how darpa does high risk, higher reward research, right? The director of the agency thought the program was gonna fail, but that IT was worth trying anyway, right?

So after that result, that was only the phase one, phase two. The performers on the program got to work on bowings. M, A little bird. Now, boeing. S, M, A little bird, much bigger system.

much more complex. So for the uninitiated, ica african helicopter.

helicopter, right? Helicopter is big enough to have two people on board. And there's also another complications that bones, i'm a little bird, is proprietary and I tar restricted.

So like the formal methods researchers who are like professors, they can't look at the source code for boeing. S I am a little bird, right? So they have to talk to the boeing engineers, the aviation engineer.

So not formal methods engineers, researchers, aviation engineers. So smart people, for sure, but trained and completely different discipline. And the formal methods, engineers, researchers have to teach the aviation engineers how to use the relevant formal methods, techniques and learn from what they had seen done on the icopal.

They invited the red team to, again, attack, but this time instead of making the red team attack the boy am a little bird from off the system they let the red team put whatever code they wanted on a particular partition on the boeing am a little bird what a partition does is keep code, like its only all sand boxes right and so what they did was they let the red team, but whatever code they wanted in the camera partition as a proxy for a bad guy could get in to this practice because IT wasn't entrusted and that the red team was chAllenged to disrupt the Operation of the helicopter while I was on the ground. But Operating on the ground, the red team could not disrupt Operation of helicopter. All they did was crash their own partition.

So they could, like, bring down the camera proposition, bring down their own partition by essentially fork bottoms ing themselves, like creating like lots and lot of copies of their own processes, which which would crash that partition. The rest of the system would be like, hey, wait, the camera partisans gone down. Okay, less we started.

So not particularly disruptive to the Operation of the helicopter. Of course, this who's on the ground at the end of face three, they redid that experiment while the helicopter was in flight with two test pilots on board. So they trusted the formal methods so much that they were willing to put the lives of those test pilots at risk.

With the same results, the red team could crash the camera partisan, but IT could not disrupt Operation of the helicopters to the pilots survived. And they reported that they couldn't tell the, they were flying the high assurance version of the helicopter instead of the Normal version, the helicopter, the helicopter to them, they were perceptable to them. And IT changed the conversation about formal methods.

IT clearly demonstrated that we know how to build systems that are dramatically harder to hack into. And so it's not the case that we have to live with software that's always going to be easily hacked into, right? So the fact that we are not, you know, rapidly rolling that out is a choice, not a technical limitation.

You know, I think it's what the example that you've given an obviously hat comes as a program that I believe you started as a program manager at least ran for the first half of its of its life cycle. I'm clearly i'm seeing darpa has very long phased life cycles for everything, though I hear darpa is the nimblest of the many other D O, D organizations .

yeah indeed. And so like how is that the darpa can do such high risk, high reward research? Um so one of IT is that all these programs like like hackers that yeah wasn't deed, the program manager who started hacking, program managers are responsible for starting programs. We we have to answer the hil arcadia ism questions.

What are you trying to do? How is he done today? What's knew about your approach? Why do you think that will be successful? That why do you think IT be successful is really important because it's really easy to answer the high marcato ism questions for things like transporters or time travel, but like nobody has any idea how to do that.

In fact, we really, you can do this, and you know how many resources will IT take, who cares? And how do you measure progress? Like those questions are like deceptively simple, but actually really, really hard to answer in a compelling way.

Then you go up to tech council, which is the leadership of the agency, and then if you get a yes, you go and you you publish a broad agency announcement that puts out into the world. This is what we're trying to do. Please write a proposal um describing your approach.

The program manager and an army of um experts in the government then reviews all of the responses and puts together the set of responses that they think maximizes the chance of satisfying the the proposal to darpa that we think we can accomplish. What was described in the higher ma caisse, those proposals, the ones that get selected going into contracting. And then the program manager tracks very closely the progress is made and you have the periodically evaluations, those program managers, everybody in the agency who has decision authority who can make decisions about this team is going to get selected.

That team is not going to get selected. This performers going to get cut or this whole program going to cut has an expiration date on their badge. Program managers, office director or the head of the agency, you get hired for like two years, get renewed for two years, maybe for one or two more years, and then you have to leave.

Everybody here who has a decision authority is a temporary employee. And that's absolutely critical. And why is that a critical?

All sacred cows and group think right like there's institutional inertia that builds up and you've tt a get fresh bullet in to almost think about the problem space in a fresher away one hundred percent.

So I think there's like at least two reasons why it's critically important. One is that like when you start up and effort, like you're like, I think this is going to be this could be successful when the program is actually ending. The people who are responsible for winding IT down are almost always different people.

They're not so invested in the successor failure of the program. So like the sting of failure doesn't land that hard. So I think that's one reason. A second reason is dark is all about creating strategic surprise.

And kind of once you've been here for a while, the agency is sucked, all of the strategic surprise that you're good for out of you, which is like you are leading, right? Like the group thing, right? Like once that sucked, the stories surprised out of you new blood comes in and it's suck strategic surprise out of them, right? So I know this is like I was here is a program manager and now i'm back is an office director, but I was gone for seven years so I had a chance to like, accumulate no different perspectives.

New strategic surprise a so IT does um mitigate against sort of group thing know it's true that darpa has done like sixty years more or less of investment and artificial intelligence, but that's not one person investing in artificial intelligence for sixty years. That is hundreds of people deciding that we should be investing in artificial intelligence over and over and over again. And it's not you know, one program and artificial intelligence. It's hundreds of programs, each one with a technical hypothesis and something that we're trying to get to in different metrics. So it's many, many different takes on what we should be doing and why we should be doing and how we should be measuring IT.

It's really inspiring to think of just like this massive path finding Operation to just like you explore something as nebulous as as as artificial intelligence. Right now, everyone just equate A I would generate of A I. But of course, it's a lot more complicated than that. But IT also seems a huge strength with darpa is what you said earlier, where you can basically tap the best in the public and private sector a to work with the also, are there any notable private sector AI companies that darpa is working with right now?

Oh, for sure. So as I said, you know, dark is contracting authority. Let us work with like almost everybody.

And so one of the things that we're working on right now that is a notable partnership with private sector companies is the A I C C. Cyber competition. That is a partnership with google, A I anthropic and microsoft.

The technical hypothesis there is that we can build cyber reasoning systems that are combinations of state of the art A I foundation models and cyber reasoning systems to automatically find and mood, more importantly, fix vulnerabilities in open source software. It's also a competition we've basically thrown down the gartley to anybody who's eligible to compete according to the amErica competes act of the U. S.

Citizen as the lead on on the team. But of the nuts, pretty much open. And we had the semi finals at defcon this year.

We had more than forty teams actually submitted cya reasoning systems to find and fix vulnerabilities in five different open source systems. Those systems were the linux cornel Jenkins engine x equal like three and apache tika. And those are like the real systems that people really are using in in the real world, like everywhere, like super wide used.

The competitors got to run their tool for four hours on each of those five systems. So no human engagement completely automatically. And the competitors tools found twenty two different of the synthetic vulnerabilities.

And they were able to patch fifteen of them, which is like super exciting. When we were talking about organizations that are responsible for U. S.

Infrastructure, critical infrastructure, they were super, super excited about the ability to pat vulnerabilities. They're less excited about the ability to find them automatically. They're like we know we have so many hlt abilities, but like.

don't try to light on our dirty laundry innocence well.

so much that it's like we know we have so many vulnerabilities, but the ability to fix them, they were over the moon with help fixing the vulnerability. The fact, tell how, and i'd like to reinforce that they only had four hours. So like if they had more time, they could probably do much Better than that. And and those bugs were not easy bugs, like we were planting really hard bugs that were um pattern after really hard bugs that had been found in the wild. Ah so really excited about what that says for the future in these kind of A I enabled cyber reasoning systems for finding and fixing bugs and helping us kind of really paid down the technical debt, perhaps its speed and scale, which could be really critically important for future national security chAllenges.

A quick aside, listeners doctor Fishery is about to mention volt typhoon, a hacker group believed to be linked to the chinese government. They're suspected of infiltrating critical computer systems, such as those controlling electricity grids to identify vne abilities. It's a bit like somebody planning a bank robbery. But first gathering intel and how many staff on duty, what security measures are in place and how alarm systems work.

We've seen with with world typhoon and with testimony before the the house select committee on the ccp that you know our adversaries are planting implants in our critical infrastructure that they would likely use to cause disruption in the event that, for example, china were to invite taiwan, do you know, cause disruption in both civilian and military infrastructure in the us, which would be horrible for national security. And so like the the ability to use something like the cyber reasoning systems that were creating IT could be game changing.

Yeah think how do people when they hear about open stores and technical that they don't understand just how much of like critical infrastructure and just like the services people use every single day, rely on this, right? And so maybe the examples that come most top of mind for folks are like, obviously crowd strike, right? Like leading cybersecurity firm, which had acknowleged that cost companies an estimated five point five billion.

We've got the upcoming elections and you're talking about geopolitical concerns to like in a situation where, let's say, china does decide to invade taiwan in that scenario, are they already going in planting a bunch of vulnerabilities that are just ready to be kicked off and create seed chaos in amErica to hinder a response? You said something that says we like living the best of times and the worst of times. Can you just elaborate on why you think that is.

well, the best of times, right? I mean, we have like so many, you know, really cool capabilities and tools, right? Like think about the fact that we're having this conversation where I don't let me know where you are, but I can see you and I can hear you like you're in the next room, right?

We have cell phones. Like, know, we have all of these wonders of technology. But these wonders of technology are built on an infrastructure that is kind of riddled with vulnerabilities.

IT assumes that most kind of assumes that everybody is well intentioned and is a good actor, a good actor, right? And sadly, that's not really a good assumption. Turns out the .

world has bad actors. IT turns out the world .

has bad actors IT turns out uh people have different motivations. And what we saw with not petch er which was a cyber er attack that ended up attacking the marsk shipping industry IT took out something like fifty thousand computers and their vote system IT destroyed all but one copy of their active directory system, which is what they were using to keep track of where all their containers ships all their containers were and what was in each of their containers.

Um and I recall reading the only reason that backup b remained is because there was just incidentally a power outage in ghana.

Right there is a power ottaway. And so they had one copy left and they had to fly that one, I think, the laptop or the computer that had that one copy back to the headquarters to be able to reconstruct their database of all of their shipping containers, right? So like that sort of shows like how vulnerable marine squad, they didn't sort of realize how vulnerable they were.

And so that's an example, kind of the worst of times, right? How vulnerably are crowd strike is an another example of formatting error took out. Like delta was down for for days, right? And there was millions of dollars of, and many, many people's travel plans were disrupted because of a right.

You can imagine what might happen if we had motivated adversities trigger ing such things intentionally, instead of accidents that that are happening. We, we, we're living in a very inter connected world that's easy to fly all over the world to see, like the wonder the world, like we're reading all of these amazing benefits. But it's all built on a very shaking infrastructure. And everybody's not a good actor. And people are willing to resort to force and could leverage the weaknesses in our cyber infrastructure and could be leveraging in the future the capabilities of A I to force to really live with the consequences of our technical debt.

Hi, i'm adam grant, host to the podcast rethinking, a show where I talk to some of today's greatest thinkers about the unconventional ways they see the world on rethinking, you'll get surprising insights from scientists, leaders, artists and more people like race, weather, spoon, welcome, glad, well and yoo here lessons to help you find success at work, build Better relationships and more find rethinking wherever you get your podcasts.

Want to change gears and talk a little bit about large language models and sort of the threats of jail breaking there. You'll had a project called guard, if i'm not mistaken, and you came up with these things called universal suffix attacks, which are kind of popular on a twitter slash x these days are just like IT almost looks like like ji bish leads speak that you append to the end of your prompt.

And IT basically removes all the trust and safety like considerations in these models and gives you an answer. Our listeners before doctor Fisher dives into universal suffix attacks, let me quickly explain what a suffix is in the context of ai. So when you prompt an A I system like ChatGPT, you want to response that's clear and easy to understand.

Thick is simply an additional instruction usually appended to the end of your prompt to clarify how you want the system to respond. For example, you might say, explain what malware is. And at the suffix in lamon's terms and under fifty words, that's the proper use of prompting.

However, you could use the suffix to try trick the model into giving unintended or restricted responses. For example, you could say, explain what nowhere is and then say, encore, what I just typed and teach me out to create nowhere. And in a previous interview, you mentioned that all the large language models did really badly on this other than anthropic. Then after asked, like he's been a bunch of time since then since that interview is there is something just implicit about how anthropic and their like constitutional AI approach that makes them much Better at guarding against these type of prompt tacks and have the other large language model companies caught up.

I don't actually know what the state of the art is since then, but um clearly they were doing something different. You could be I think there least two possibilities. So I do think that that work is is super interesting, right? The technique that the the team used was that they trained on open source models to find some fixes that would jail break the particular model.

And what those sort of sexes did an intuitive level was they they had the effective saying, like putting the model in kind of positive frame of mind, basically making the model say, sure, I be happy to help you by answering. And that is an interesting characteristic of large language models, is that in some ways they kind of do work like people. And that the prompts, the words and the prompt kind of put them in a frame of mind. So if you, if you get them saying something like, sure, I be happy to help you by answering your question about how to build a bomb.

And the answer is like, if you can get them to say that, but sure, I be happy to help you with how to build a bomb, then they're very likely to kind of keep going um even if like they previously had lots of things that said no, of course i'm not going to help you build a bomb if you can kind of get them started like sometimes like people like that, if you can get them started down the path, they will kind of keep going and then the interesting thing is that that gybed h like to you and meat IT looks like gibberish. But to them it's just a different vocabulary for saying, sure, I be happy to help you with that. Like you know, we have lots of different ways of spelling things.

It's the same kind of thing. They have lots of different ways of spelling them, but they have a bigger, much, much bigger vocabulary and much bigger way of spelling things. I don't know why anthropic was Better defended.

Its possible that they had preprocessing kinds of things where they were stripping out girish things that looked like gibi h um and so I just got filter at the beginning. And so the oxide underlying model never saw IT that just pure speculation. But IT could be something like that.

There is definitely every time a new model drops, there's like a fun couple months period where people are coming up with alter to fun hacks. Now do when to ask you about open source is interesting, like you open source clearly makes the world go around, as we've talked about and we've had on the C E O of github, uh, the CS o of hugging face, obviously both very big proponents of open source as well.

But in the industry, when IT comes to A I specifically like A I models, there are still a bit of this debate about should we continue to open source large language models as these training runs keep getting bigger and bigger? Or should we still be putting these capabilities out there? Obviously, a bunch of these vertical suffix attacks you all came up with.

We're trained off of open source models, as you mentioned. What is your view on open source? A. I. What did you think about matter releasing lama, for example, and continuing to release larger and larger iterations of IT?

Yeah I mean, I don't have like it's it's a real puzzle, right? On the one hand, right, as we released opens source models, we are releasing more and more powerful capabilities that anyone in the world can get access to. And you know to the extent to which those models become nation state level capabilities, that a potential massive threat to national security.

On the other hand, to the extent to which we don't release the open source models, then the companies that have that capability in source, that capability is available only to that rare small set of companies, which is a massive amount of power to a very small number of companies. So that's a threat, right? So like both of those things are really significant threats.

So I think that issue that we are going to need to continue to pay attention to. I think that's a question for policymakers more than technologists, but I I think that that's something that technologists will need to be informing policymakers very carefully on. I think that like what's gna happen with the next round of, uh, large language models that will see drop one, one GPT five comes out, and then the next round that is gonna trained after that is like, super interesting, right?

Microsoft making the deal to bring female island back on, but that's not for powering large language models. And I don't think that's for the next round of language models is for the one after that, like how capable that is. I think it's it's fine that lama three was released.

Open source, that's roughly the GPT four level of capability. GPT five is IT OK to release that level. Open source, I don't know.

We'll see what GPT five has to do like GPT six. Is that okay to release that level open source? I I don't know.

I think there's the advantage that the open source models are no generation behind so that we have like a generation to assess how capable is the model and how danger would IT be to release that. That level of model gives us a year roughly. I think the open source models are year, year and a half behind the close source models. Um I think the the cybersecurity of the foundation model companies is another really important thing to think about so that we the models aren't getting .

stolen terribly. It's like, hey, if we're not open sourcing this but some nation state gets access to the model weights anyway or other hacker groups.

right? Like we need to make sure that the electuary property remains where the intellectual property should be and so that we can assess that the how dangerous the capability is and how much IT needs to be controlled before IT. You know, that just gets out there to anybody who has the capability of of stealing IT or accessing IT. Otherwise.

you can ask you a question here, just like on a personal level, it's interesting, right? Like history, ally, they've been things like the manhattan project, which is like a run by the public sector and pulls in the best of academy, the best of the private sector to make stuff happen this time round. IT feels different, right?

Like we're sort of the innovations between like the transformer stuff happening in google and then of course, OpenAI kind of hey, here's a fun like research released ChatGPT moments that you know and here we are many years later. Do you think that there is going to be more and more of an a convergence between public and private sector efforts as IT comes to AI? Um especially as these capabilities like the emerging capabilities of these models as we threw more data and computer and just like keep increasing and go from the realm of like helping me write my youtube video script to yeah .

coming up with bioweapons yeah I think IT will depend on how good the capabilities become and how much resources they require. Um I I think that you know the racing dynamics are really, really interesting, right? IT costs a fortune um and those costs mean that the companies have to be able to make a profit to be able to pay for IT um at least have to have a promise of a profit so that investors will invest.

Where are those investors coming from and what kind of divided loyalties to those investors produce? Um and that the racing dynamics to those racing dynamics mean that the companies can't afford to invest in appropriate safety research because the safety research slows them down too much. But then you know can we do appropriate public private partnerships where there is public research in the safety pieces?

Can you do that in a way where you don't have access to the actual city of the art model? Is the safety research then become not really valid? Um well, maybe you do the safety research on those large open source models.

That's a reason to open source the model so that academic researchers, people who are on the outside, can get access to a meaningful model. Then can you transfer the research to the the state of the art models? How fast is that research? Can you get the foundation models companies to pay attention?

Think those are really interesting questions that we rustle with kind of on a daily basis. Do you do you understand what the real safety issues are if you're not on the inside? Can the people on the inside communicate what the real issues are when they have lots of intellect? Al property issues they don't want to share with their competitors. Um how will the international like the the national security issues related how do you not like leak the shift to adverse nations is a really interesting time yeah sounds like I mean.

not to see these are very hard questions and no clear answers, you know but there is another aspect to all of this, which is sort of like now that we have systems that can create content that I sort of shattered the audio visual turing test. How I do you tell if some content is actually human generated, authentic or manipulated? In fact, that's a program that your la darpa been working on even before the term deep fakes was popular eyes. So can you tell us a little bit about that?

Yeah sure. So darpa has run actually two programs in this topic metaphor. And then sam. So I met the sam for technology is recently successful at detecting manipulated media.

Right now, when you generate the media, it's not detectable by people, but it's not super hard to detect because the systems are working to fool people but not working so hard to fool detectors. When i've been talking to experts, they're not super clear on how long that will be. Uh, that will be the case.

It's still even now though, like the fact that we might be able to have a system that can detect IT to the human eye or to the human ear. It's not detectable. You fall for like immediately you know somebody calls up and and it's your loved one saying that they've been kidnapped like you you enter a panic mode right away and you know, thinking through a week.

This could be an audio O I need to like, have a conversation and asked them about something that would be not knowable by a random person like where do we go on vacation ten years ago that isn't on social media, like it's a whole mindset change like we have to adjust as a society. Accountants and companies have transferred large sums of money before they can video, deep fake and zoom with multiple people saying things. So, you know, that will require a kind of mindset shifts, even though we have developed technology that can detect those kinds of things. And according to experts, that technology probably will last for a while.

So you are saying like while this content is you know perhaps cannot elude a machine just yet, humans, especially as you're kind of going through a feet of social media or you're getting a call, especially if it's like targeted directly towards you, we still have these cognitive vulnerabilities that we need to deal with. One of the first episodes we did for the show was all about AI literacy.

Like how do we just get people aware of the fact that people can generate this type of synthetic content that is in distinguish from reality? And of course, twitter and acts came around and speaking A A open source release in open source image generation model flux, which is like just as good as my journey, the previous close source AI model. And then I I would say we saw a many weeks are very unhindered, uh, kala.

And thump means on twitter. So when you see stuff like that, do you think that actually helps our hurts? Like, is IT helping build anti bodies and people for people like, great. Next time I seen images like this, I may not actually be real .

or does IT hurt. I mean, all first, what commonly didn't really look like cmp was a lot like trump.

This is true um which you know I .

has to do with you know bias and A I right in the fact that there's well, first of all there's way more images of trump um but also in general AI is much Better at White men than women and much Better at White people than than black um that's just like really in your facing example of that um so I think that will inoculate some people and I won to oculina other people I don't know there will be a single single response to that. I think a really well place deep fake with appropriate supporting material will catch anybody by surprise and um could have potentially disastrous consequences. So I don't think we will know the full consequences for a long time.

Have fun, have fun. And sort of related to that, when figuring out what is authentic and especially proof of person hood has become this huge topic. And of course there's like approaches like world coin there's like ID dot me apple with face ID to figure like is this truly you um or did a human actually generate this content?

And one of the things that keeps coming up in the show again and again, which is sort of roughly this notion of the solution to one thousand nine hundred eighty four, often ends up sounding like one thousand and eighty four, which is sort of this idea, in this case, where it's like weights we need, like all the big tech companies, to train, like this person of interest models, so they can figure out if this is truly you or not. Like most people don't know this, what the iphone has, this feature called attention awareness, where it's constantly is taking a very like low raz I R photo to see. Are you looking at the phone in order to unlock IT? And so there's proposals here in the future. May be your phones just locally on devices doing that to make sure this is actually you using IT. I'm kind of curious, do you run into these type of countries or catch twenty tools or like to in order to defend the very thing you have to create technology that can kind of offended all the time.

So um when we're starting a new program, we often think about, you know what are we trying to accomplish with this program? We have an an effort darpa called L C. So ethical, legal, societal implications. So we do our best as we are starting the program or or thinking through whether we should start a program to think about not just what is the intended consequence of this program, what are the potential unintended consequences of this program and how do we anticipate them mitigate the negatives.

Is this something that we shouldn't do IT all? I don't know that there are cases where we don't do something because of the unintended consequences, but IT does often shape how we do something and often IT makes the program much Better as a result. One of the examples is that just the bianco gy is like seat belt, allow you to drive faster, right by thinking through the consequences, unintended consequences, that lets you be more robust and stronger.

When we do cyber, we often like there's a defensive application, there's often an offensive application, and we often kind of do both at the same time. By by doing both, we can really think through both sides and do both sides, both sides Better. There's like just so many thorny consequences.

I think one of the examples is with A I and and data privacy, right? With A I you need data foundation models. Basically, you're consuming all of the data in the world.

And maybe A I models could have even more data if they could convincingly explain or deal with privacy issues from a national security perspective. And we have top secret data, secret data, sap data, like all like. I would be fantastic if we can kind of feed all this data into this kind of technology.

How do we do this in a way that keeps IT all separate? The same technology could be like, could we have like a you all of my data, like all of my data, but I don't want like you to have all of my data. I don't want like, you know, my mother to have all of my data.

But you know, we could have all of this data. We can maybe build even more amazing things, but people wont be willing to share that data if they could keep IT all separates. So like these are more examples of conflicting goals. And if we could solve the keep data separate problem, then maybe we could have even more advantages.

So it's it's very I mean, this particular one kind of hits close to home because there seems to be a huge advantage then to the large tech companies that perhaps already have your data in order for you to be useful, right? Like like google, apple, meta, microsoft comes to mind.

And then of course, I have to imagine, like all the tokens in the world, probably the nsa has, I would guess, but even in a company right, like you don't want some random m person going in querying hr records and getting detailed hr records back. And they've already been examples of people putting l lambs as this orchestration layer and ending up with these sort of outcomes. Now let's change gears to the future a little bit.

What do you you have opinions on what the future of AI is? Is he going to be like, like a, like a very few large models and like a long tail of small ones? Is he going to be something difference? I know you cited uh, like the elephant example by gary Marcus who we've had on the show. He obviously thinks that like generate vi sucking the air of real progress and like you know like the transformer based, like the transformer to fusion, prepared to me is a dead end.

What's you're that I spent a lot of time thinking about this. Um so I mean I mean I think one question is how long a time horizon are you thinking? I'm going be really curious about what happens with GPT five and how how much improved that is.

I think the the strap ry model is really interesting development. I think it's very early days on that path. Be really interesting to see kind of what strawberry prime looks like in how much of an improvement the next version is to sort of see what the scaling laws are for that approach.

I do think we're going to see a deliberation of different kinds of models because the cost profile of the foundation models is so gobs macks high that is not affordable for many applications, but smaller models are much more affordable and much more find tunable for really interesting applications. So I think that there will be a tale of a long tail of smaller applications customize for various purposes. I think you can customize smaller models to particular domains to get much Better performance and accuracy, much lower hallucination rates and in much Better performance.

So I think that we will continue to see that. I do think like strap ry is pointing in a really interesting direction where it's not just the transformer based approach, it's transformer plus other kinds of algorithms and other kinds of approaches. I don't think pure transformer is gonna get us to appropriate levels of trust.

But transformer plus calling out to tools plus like the reinforcement learning kind of approach that is a little bit not to in strawberry or uh you know kind of ebola c approaches to like you know humans clearly use multiple different kinds of thought patterns, right, fast learning, fast thoughts. So thought like I think we're going to get to a more hybrid kind of approach that is going to get us to a range of different kinds of trade, ffs, of speed versus accuracy y and that we're going to get dramatically improved capabilities over the next x number of years. I don't have knowledge, so I don't know how good GPT five is going to be.

I don't know how good GPT six is going to be. I am I going to remember the source um about like do we have enough data? Do we have enough powered? We have enough compute to kind of keep going. And the analysis was, yes, we do have enough of those things, which I think the power and and power was the most limiting factor.

But the the the real question was, do we have enough money to pay for those things? Will the companies be able to keep going? And I think that's the real question, will open the eye and be able to find enough investors clearly that the move to be a for profit company is probably leaning towards finding investors who are willing to an invest um because like the costs are astronomical.

And so you know there's a question of like is that investment predicated on though they really are close to agi, which would be super game changing? Or is IT you know they just have to keep going because of the the narrative they told so far. You know I think you know chat bots have a certain a degree of flexibility because they're talking to people and people are willing to like fill in the gaps.

And oh, you must have meant this and auto correct for the chatbot that is talking to when you talk to agents that are interacting in the physical world, which is the obvious. Like next step, like, don't just talk to me. Go plan my wedding, go plan my vacation.

go. Like, do this physical thing for me, way less room for air. Like when you have to go, do multiple steps in a row if you make a mistake. So a hycy ation in that setting is an actual mistake, right?

Like just compounds IT, just compounds, right?

So like that compounding error is like fatal for those kinds of applications like you have to be able to do much Better than they're currently doing in jail. Breaking in that situation is catastrophic. So we have to do Better with respect to keeping the models aligned with human interest, and we have to reduce the the illustration rate. We're clearly going to converge how much time that takes is you know months to to GPT five, six months a year or is IT a decade? No, I think that that difference will um predict the future of those foundation companies.

And as you say, there is a very much a race dynamic going on between all these companies. Then again, all the companies are far more cognition about the trust and safety concerns. So they are working more closely with the public sector. And so I will be interesting to see, I think like one quick note on on strawberry or or o one as it's been released in public, I think is very interesting where just having a model like yes, they did some reinforcement learning and then search stuff, like just have a model sort of think over stuff and due chain of thought, kind of like a human does, makes IT way Better by throwing more like inference time, computer adverse training time compute kinder blew my mind.

But I also like made me have like a weird religious spiritual experience looking had IT sort of going through like thinking through very these various steps and you know, reminded me of one of the other projects that came across the you'll worked on, called in the moment where the goal was about in viewing a moral compass into machines. And so I think like that's a question people always talk about when IT comes to alignment is like can we actually have these machines totally understand morality and sort of our value system and then add red to IT? Can you talk a little bit about that?

Yeah sure. So in the moment is about, uh, can we align algorithms with human decision makers? And what influence does that have on the willingness of human decision makers to delegate to algorithms? And the sort of motivating h examples are places where decisions have to be made too fast for humans to be able to make this decision. So one example was the mass shooting in vata, where there were no hundreds and hundreds of casualty is overwhelming the hospital system. And so they basically had to throw out the Normal playbook and and do the best they could with the the casualty rate.

So that was one of one of the motivating examples for in the moment and one of the other interesting things about in the moment, there isn't a single human um value system, right? Like there are many, many, many different human value systems like you put two people in the room and you have two different value systems, right? And so what in the moment is looking at is can you develop algorithms that are tunable to different decisions makers? And you know, some algorithms s are tunable, some algorithms are not tunable.

So what extent can you tune different algorithms to different decision makers? And then what difference does that make? Know our decision makers than more willing to delegate and less willing to delegate. And IT turns out that different algorithms are more less tunable. And yes, decision makers are much more willing to delegate when you can show them that the algorithm is more, you know, is aligned with their decision making.

Are IT to rap things up or ask you what makes you hopeful for the future of ai and what worries you the most right now?

In terms of like my biggest fears, I think we might see a lot of societal disruption because of job displacement. I don't know quite on what time frame I think that that disruption can cause massive society appeal. Al, which has ramifications for the political system, is that a, that can be quite unexpected and can manifest in all sorts of weird ways. I think that there could be significant consequences for national security in terms of A, I paired with offence of cyber weapons. Things like that could be quite disruptive.

Yeah, it's it's funny. It's like speaking of people ducked taping stuff together. A video that went viral yesterday was like a harvard student hack together, like the new meta A R glasses with a bunch of public A P S, basically recreating clear view A I like on campus.

And of course, as we're seeing in ukraine, people ducked taping together, drones, thermal sensors in doing alters of other kind of like hacky science projects. But in the battle field, it's interesting. IT feels like more than ever, kind of the technology and capabilities that were delegated to the intelligence community is now accessible to like the crime syndicate next door. And there is something really scary about that. But at the same time, i'm actually going to see a lot Better knowing the yellow, thinking about something as crucial as like the underlying software infrastructure that runs most of our world.

So like we're making technology that makes IT easier, easier for smaller and smaller groups of people to accomplish that, which in general is fantastic. But when you talk about people who want to just cause chaos, taking that same technology, the ability of small number of people to cause massive bad things is terrifying. O, of course.

what makes you feel, I mean, I spent an amazing .

time for the progress we've seen. An A, I just lean ChatGPT. Three came out the fact that you could converse with computer system in human native terms, completely game changing, right? I worked on program synthesis as part of, like, my background is in programing languages, and had done some technical work on programmes. And this is, and I would have said were decades away from being able to do programme synthesis and then kind of turnaround and how this completely different technology be able to start writing programs, just shocking.

And I think that domains where you have A A checker that can check the output of the the AI system and to confirm like, yes, this was good or no, what was that? What was that thinking? Our places where we can have a feedback loop on the output of the general eye system, our places where we will be able to really very rapidly be able to innovate and have amazing new capabilities based on general vi, independent on whether we get to agi in two years or or ten years or twenty years.

Going into the research prep for this interview, I wasn't sure how I fill by the end. Now we've touched on exciting duced topics on the show before, but speaking with someone in as series of a role as doctor, Fisher takes IT to a whole new level and SHE brought up some sobbing realities about the chAllenges we face in cybersecurity and national defense. But all the enough, I came away from this interview feeling hopeful.

It's so easy to be cynical about the government when you consider the average congress person often seems out of touch with modern technology, as we've seen in the numerous big tech congress's hearings. But knowing that agencies like darpa are actively collaborating with big tech companies while leveraging and improving opens, so a software to create a safe digital environment is all very reassuring. IT shows that there are people within the government who are not only aware of these chAllenges, but also thinking protectively about solutions.

Even more encouraging is that major tech companies are now willing to work with the public sector on these issues. And this collaboration isn't about building something the series like skype, it's about safeguarding our increasingly fragile way of life as technology continues to orchestrate more and more of IT. There's a real sense of responsibility and ambition in these efforts.

And i'm certainly to sleep more soundly ly tonight, knowing darpa is on the chAllenge at hand, both creating and preventing strategic surprise. The teddy eyes show is a part of the ted audio collective and is produced by ted with cosmic standard. Our producers are dominic jard and alex higgins.

Our editor is bang bang chang. Our show runner is iona tucker, and our engineer is asia polar simpson. Our researcher in fact, checker is Christian apart to our technical director is Jacobean ic and our executive producer is a lizer math. And i'm beloved s to do. Don't forget to rate and comments and i'll see you in the next one.