Sean McGregor on the AI Incident Database and the AI XPRIZE
34:26 Share

Hello, and welcome to Skynet Today's Let's Talk AI podcast, where you can hear from AI researchers about what's actually going on with AI. We release weekly AI news coverage and also interviews with people in AI. I'm your host, Sharon Jo. In this special interview episode, we'll get to hear from Sean McGregor.

Sean is the ML architect at the neural accelerator startup, Syntient, works with the XPRIZE Foundation to structure the IBM Watson AI XPRIZE, which just awarded its $5 million prize in AI for Good this past week. And he is also the creator and maintainer of the Partnership in AI's AI Incident Database, which will be the topic of our discussion today. Thank you so much, Sean, for joining us for this episode.

Thanks, Sharon. It's great to be here. I'm very excited about the AI Incident Database. We chatted about it in our weekly AI News podcast. Could you give us a quick high-level overview of what it does? Sure. So the AI Incident Database is inspired by similar databases in sectors like aviation, where you...

collect all the failures, accidents, incidents into one place. Then you use those collections of failures to actually inform industrial practices and design to make it so that those systems are much safer and better for the world. So in the case of AI, this seemed to be a natural fit because, you know, in contrast to

which you just need to make the designs better and you don't really need to apply your imagination. In the case of intelligent systems, you very often don't know how things can go wrong until you're provided with an example and then you kind of slap your forehead and say like, oh, I've

Of course, that's a thing that we need to solve. And so it just makes a natural intuitive sense that we should collect all these failures into one place and use them to tell the heads of products at companies, the machine learning engineers at companies, the general public what AI is and is not good at and what problems need to be solved before we're really at the state of truly robust AI for the real world.

Right. And you span a lot of different types of incidents that either are causing near harm or do cause harm to people. And I see that you have a definition of harm and near harm specifically on your site. And some really cool examples, just to give the audience a sense of what there is, is

Google's YouTube kids app criticized for inappropriate content is one of them. If you remember, uh, the inappropriate content, uh, that was served to YouTube kids, uh, a few years ago. Um, and there's also, when it comes to gorillas at Google photos remain blind, uh, and

And this is about the news story that broke out around Google's computer vision algorithm, basically classifying a lot of people who are black into gorillas, which is very inappropriate.

And finally, a Tesla driver killed while using autopilot was watching Harry Potter when it says so spanning from, you know, things that are just in the software world to to Tesla crashes that very much are are killing people. Could you give me a sense of how you came up with some of the definitions for harm and also how how you've been collecting all of these incidents?

Sure. So initially, and still, the definition of harm is intentionally broad, and we generally err towards inclusion of things so long as there is a harm or a near harm. And we came up with the definition that is presently on the database in collaboration with Georgetown Center of Security and Emerging Technology. And

We've been collaborating with them extensively. There's actually a new feature that's due to be rolled out shortly, probably around the time that this podcast episode posts, including a taxonomy that goes through and classifies all the incidents that are in the database according to a great many tabular entries, a lot of types of

types of harms, uh, uh, entities associated with it. Uh, I'm quite excited to roll that out into the world. Uh, so the definition of harm, it's, it's basically, uh, if you were to capture it into a single sentence would be something along the lines of, uh, someone saying that there was a harm. Uh, and, um, that, uh, that's really, uh, enough to, to get in here and, uh, to, uh,

be sufficient for people to want to learn from those incidents and to prevent that from happening in the future. Right. And for each of these incidents, I see that you list a ton of news articles associated with it. So for a lot of them, they've received quite a bit of attention.

When you released the AI incident database, what was the reception like? Because I know that, you know, corporate names are there and there's data collected of, you know, who who has the most incidents or or at least people have reported on that. What was reception like both from, I guess, corporations, but also the public? Because I feel like this is long overdue.

Yeah, I agree completely. And I think that it's probably a useful binning that you did there of talking about corporations and the general public. I think on the case of the general public, it's very much the kind of spectacle of AI. And I think that the database is serving a

important need for them where it is collecting all these different viewpoints into one place so they can sort through and kind of allow for the ground truth to emerge out through synthesizing the different viewpoints that are represented.

But by and large, you can expect more people to read an article in Wired than reading the instant database itself. So from the general public perspective, the...

the things that are incorporated into the database are probably more impactful than the database itself. The database is generally pointed towards trying to prevent the recurrence of these things happening in the future. It's trying to inform the machine learning engineer or researcher like myself that's in the company and make it so that...

you know, I really have that ammunition, uh, when I need to go to the, uh, leadership at a company and say, uh, Hey, we need to spend X amount of time and, uh, why billions of dollars additional to solve this problem. Otherwise you're going to get a story and, and wired or the MIT technology review or something like that, talking about, uh, how terrible the system is. And, um,

So it's really a useful corporate tool and something that people and companies are going to want to be able to use. You can actually already see that to a degree in a lot of the incidents that are in the database, which in a lot of cases are sourced from people that are or were working at major tech companies. They're very interested in bringing together resources like these.

And, um, you know, uh, there's a lot of different, uh, viewpoints within companies towards things like the instant database. And I think the final group to cover there and bet it in your question is probably the, uh, corporate communications officers and the, uh, messaging and, and everything that they're trying to put out into the world. And, um,

I don't think that they have too much to fear from a collection of news articles that are generally lower profile than the news articles themselves. And,

There may in fact actually be a little bit of a normalizing effect of just letting people know how hard these things are and that could probably actually be a little bit useful on the corporate messaging side of things. I don't think that we should normalize failures, but I think...

coming to an understanding of what is the current state of the art and AI is quite useful for ensuring we do that responsibly.

Right. And there might be an additional category of ML researchers and engineers who are working on this actively. And, you know, I see each news article as it comes and then it's fleeting and you don't you don't have it all together in your head at once. And I think this database makes it so that, you know, I can return to it. I can see where we're at for all of these things, maybe even if we've progressed from some of those incidents.

and change what I do, right? So maybe that's yet another category. Right.

Yeah, the part of the origin story after the database is that it became clear that a lot of people were maintaining these informal lists and developing their own catalog of failures and using those for their rhetorical purposes to explain to management what the systems they are building can and can't do. And yeah, I agree.

And that, I mean, this was born out of your own work in a sense too, right? The origin story. Yeah. Yeah. And, um, the kind of going back, uh, probably 11 years at this point, uh, I remember, um,

a moment very clearly of just kind of coming to a realization and, uh, my PhD program that, uh, machine learning was simultaneously incredibly powerful, but just immensely brittle and just really lacking a lot of the, uh, uh, solid theory and foundation that, uh, you build a lot of, uh,

econometrics work onto where you're looking very hard at the relationship between variables. Just the behavior of so many machine learning systems emerged through data and that's a

dangerous thing. And as a result, I started moonlighting during the course of my PhD on a lot of what others have termed technological activism. I actually spent a good period of time figuring out how to

apply usable cryptography onto things like Google emails or Gmail messages, Facebook chats, really the rest of the web had some success there. But I ultimately ran into a cat and mouse game that the technologies we were developing looked a lot like spammers and people that were scraping data off of systems and

Um, but then on the, on the other side of my, um, of my work in the grad school work, I was working on, um,

reinforcement learning is applied to a wildfire suppression policy. So deciding what you should do in response to a wildfire, should you suppress it? Should you let it burn and simulating forests for century time spans in order to arrive at those decisions and came to a very close understanding of the weird things that can result at the intersection of

simulators, reinforcement learning, and the values that you bring onto the reward functions in those simulators. And I found in that setting that if you applied different reward functions, like if you valued the ecology of the forest more so than the timber or less so than smoke inhalation, which is also a major factor in forestry policy,

The policy that you would apply to it would shoot from suppressing all wildfires to letting all wildfires burn. And it really highlighted to me the extent to which the intelligence systems we're developing are at this perfect storm of society, technological capability, and the values that different constituencies bring to it. And I...

As a result, have developed a good number of systems around trying to surface the technologies and the kind of democracy of the code and make it so that it's accessible to people that aren't spending a few years hacking together a solution that gets deployed to the real world.

Yeah, I think that also speaks to how brittle some of these systems might be or how easily adaptable they could be going from, you know, one action or one policy to the next. And I yeah, it feels like we want I think in an ideal world, we want our AI system to somehow find this perfect and nuanced solution.

Right. And it seems to have fallen into maybe based on training data, maybe based on a lot of other things that we put into our algorithms, fallen into the camps that we know too familiarly that we already have created for ourselves. Yeah. And this is why representation in the field of AI is so important because

People in AI find themselves in a position that very often they didn't ask for or didn't want, which is making these really giant decisions that are replicated millions or billions of times and have immense impacts across the world and change.

in order for that, that person to, I think, feel, you know, calm and okay with the, uh, decisions that they're making engineering wise, there has to be a lot more that, um, exposes the decisions to the world and makes it so that, uh, uh, it's a little bit more democratic or, or available to it. Um, because without that, we're, um, we're not going to find ourselves in a, in a very good state. Um,

Right. And hopefully the AI incident database, just to bring us back a little bit, will help with, I don't know, some decision making or at least becoming conscious of some of the things that we've been doing. Yeah. The AI incident database in many ways is a...

developing a checklist of things that you need to solve in order to deploy to the problems that are associated with incidents. And that's an immensely useful thing for your engineering processes. Great.

Right. And I love how it's not just a, you know, just a list. You also have data on your list, like you have leaderboards and those leaderboards, you know, of the submitters, the top submitters, the top authors of various articles, the top domains. I thought that was really interesting, especially just giving almost giving credit to certain media outlets for reporting it. Right.

Yeah. And you can see which media outlets have people that are covering this issue area and doing so in depth and where the expertise and the fourth estate lie. And that is a very useful thing as well. In our next update, which comes out shortly, it's going to have even more statistics and things that...

go beyond the unstructured full text of all the articles. Exciting. Yeah, quite exciting to show on that. And one thing that was striking for me on that is, you know, you have the usual suspects of companies in there that are very prominent and

developing of AI systems and bringing them to the world. They're the ones that have the multi-million or multi-billion dollar budgets to bring things. But there's also a very large number of just

companies that fit into the other category, ones that are kind of downstream from that leading research. And it shows that it is spreading out through society at this point. And there's a lot more parties involved in the deployment of these systems now.

Oh, that's so interesting. So it's almost like, I mean, we have these incidents, but if we could also trace the lineage of that incident, because they're not necessarily coming, they're not actually doing that research in-house, they're just applying something that has already been published, that would be super interesting to see just, you know, what is the lineage of this? And what is also from a researcher's perspective, especially those in academia, what is the impact of your work that you've been putting out, right? Yeah.

So the...

So the impact of the Anson database so far, I've been quite happy to see. We've actually had users from now 157 different countries visiting it, and this is where we need to do a lot more to make it representative of the Anson's that are occurring around the world. We need those people to come in and contribute their insights because that's...

something will happen in China, for instance, that we can learn quite a lot about in the application in the US of similar systems.

And there's a lot of knowledge sharing that can occur there. And I'm hoping to add on more capacities for applying a translation on incident reports sourced from non-English outlets. Right now, it's all entirely English. And despite that, among the

top countries. There's China and India actually have a good number of users coming into the system. Also, a huge number of Finns. I don't know what's going on in Finland for us to get so many users as compared to the number of people that are in Finland. But if someone could explain that to me, I'd...

I'm curious to learn why they're all coming in. So our VPNs are routing there, that's why. Yeah, that's actually been suggested to me, and I am wondering if that is a very real possibility, in which case we have a lot of VPN users. That's weird, too. And then on the kind of...

uh, PhD, uh, works on things of doing the, um, uh, uh, wildfire work. I think, uh,

The biggest takeaway on that one is the degree to which that whole policy area is trapped in political morass. And we need to come to grips with that as a problem of what it means to have people that are building in the wildland and urban interface, the WUI and so on.

You know, who's bearing what costs of fire suppression and whether it's a socially borne cost or an individually borne cost of those that were deploying millions of dollars on stamping out fires in an area that historically had fires and it's protecting a hundred thousand dollar structure.

And so a lot of my work there was building the visual analytic systems and the like that allowed for understanding the optimizations that are produced on the basis of changing reward functions. And I'm seeing some impacts on that, but ultimately, I did not feel that the system was appropriate for

arriving at suppression decisions for a live wildfire, it's good for the ones figuring out policies and like why people are yelling at them when they're writing the forest management plans that incorporate these optimizations. - Right.

And maybe as part of the database, there could be another section of what is gridlocked or what AI can't really solve right now or shouldn't be applied to right now. Yeah. The database itself, so let me go a little bit into the design there because I think it's important to explain the voice brought by the database and what its position is.

Positioning is the way that we've architected the databases is meant to be multiple perspective and the data presented and the characterization of that of that data.

What that means is we're presenting multiple publications, multiple reports about the incident. So you can have a list of, I think we top out somewhere in the 30s of a number of publications that are associated with an incident. And each of those bring their own perspective to it. They each have their own voice and they can be as biased or as unbiased as they want to be on that in whatever direction they're

Makes sense. What the database itself does, though, is it doesn't have something that says this is the findings of the database. This is the panel has met and decided X company is at fault and the impact was $10 billion. That's really...

The people reading it and distilling all these different viewpoints can distill it on the basis of this infrastructure we're developing that incorporates all these different viewpoints into it. And so we have the perspectives of

of people getting incorporated in incident reports. We also have a new feature coming out for taxonomies, which is allowing third parties to go through the database and apply their own coding set or qualitative analysis of the data set and then incorporate that into the database, either in full text or also in

tabular or categorical form, numerical form, depending on what it is they're trying to do. And

This is where I'm really excited as a machine learning researcher of just how rich this data set is going to get and how you can build systems that monitor the entire Internet for incidents that have been reported on but not committed to the database yet. So have a continual dashboard of what is happening with AI in the world and where things are going wrong.

So there's really a great opportunity and NLP research here that I hope people will take up. That's really exciting. I love it. And as you expand to different languages, too, I can imagine there are there will be cultural differences and different cultural viewpoints as well around a single incident that you'll pick up on and maybe add to and then have these diverse perspectives for for each for each incident.

Indeed. And maybe if it's all translated in every language, you know, everyone could just read all of those points of views. And that would be that would be very exciting. As I know, I'm multilingual and and I know the news sounds very different or perspectives change a lot of language language or maybe just culture to culture, even language language.

Yeah. And I think that's particularly important since so many systems are global in nature. So something that is an incident in the U.S. might not be an incident in India or vice versa. And it takes a lot of cultural appreciation to produce a global model that's making decisions in all contexts.

Right. Agreed. Agreed. And now shifting from the AI incident database to your work with the X Prize for a little bit. I'd love to chat about this prize, what it was rewarded for and the overall process and how long you've been been involved with that.

Sure. So I first signed on to the XPRIZE effort in 2017. It was actually shortly before defending my PhD. And the XPRIZE Foundation had just announced a prize in AI for good. So improve the world in some form, use AI to do it. This is very different from what the XPRIZE Foundation typically does. It's best known for

the Ansari XPRIZE, which was awarded, I think, something like 20 years ago. And that was in space access. So can you get to space, pass the von Karman line twice in the space of two weeks with the same platform? It was really launching the

effort for reusability and space access. And after that, it's done a series of grand challenge competitions in various areas, carbon capture, education. And each of them, it's largely been a challenge-specific technology agnostic. So you can apply any technology you want to solve the problem, but there's a definition of the problem you need to solve.

In the case of AI, though, the approach was you will improve the world. You will use AI. So it's technology specific, challenge agnostic. And then you'll be qualitatively judged on how much you're improving the world via your advancements. And so about four years ago, we had 150-ish proposals for AI.

how teams would compete towards the $5 million prize purse generously funded by IBM. And a lot of the work in the intervening time was figuring out how to appropriately judge them, enable them to be successful and ensure that their work was maximally benefiting society. Like the

From the earlier conversation we've had today, there's ample reasons to believe that even things with the best intentions can go wrong. And we had to figure out ways of ensuring that would not happen. And so over the course of the years, we've we've leveled it down from initially we were cutting things on time.

on the basis of the problem that they're approaching. We recruited a group of largely AI academics for the first few rounds of the competition, and they applied largely academic review processes of the team's reporting and the judges selecting the most deserving ones to move on. And we arrived then just this last week with,

at the finals for the competition, which pitted three teams against each other for the prize purse. The top team got $3 million, second place, $1 million, third place, $500,000. And then the remainder, $500,000, was awarded to two other teams outside this group. And to step you from third to first,

The third place team was Marinus Analytics. They actually work to try and find and protect people that have been sex trafficked. And they actually crawl a lot of people.

a lot of the internet actually looking for someone that's disappeared or been sex trafficked. And then they use that to try and get them out of a bad situation. Um, uh, second place was, uh, AI Fred, which is, um, uh, a startup that, uh, does, uh,

clinical depression treatment. So one of the problems that we have in mental health treatment is that different people respond differently to mental health medications and standards of practice. And it's very, it's,

It's very difficult and time consuming and on both the mental health industry and also particularly on the person to find the best treatment. And so what this team has done is they've developed systems that generalize and suggest treatments.

treatments on the basis of past patient data and helps systematize better standards of practice across the mental health industry.

Finally, the grand prize winner here is one in malaria eradication. So there's a lot of mosquito abatement work that happens throughout the world trying to eradicate particularly the mosquitoes that produce malaria.

um, uh, malaria, uh, outbreaks in areas. And they've been combining several different elements that help identify, for instance, where standing water is. Uh, if you know where the water is, you can go and you can spray that water to eliminate the mosquitoes in it. And, um,

They've done things like develop simulators for mosquito spread and effectiveness of those activities. And they've put a lot of this intelligence into the hands of people in particularly several African countries and into smartphones and help them understand where the most efficient application of mosquito abatement practices are. And yeah,

So this is just a sampling of three of 150 plus teams that were all vying towards winning the XPRIZE and was really a great laboratory and good and how we can improve the world with AI. We've talked a lot today about the negative things that AI can do and

I didn't go into AI to be a critic. I went into it because I think it's immensely powerful and it's something that we should absolutely build and bring into the world. We just have to put through the time, attention, and effort to make sure it's beneficial.

I love it. So AI for good. It could also be good. It's a tool. And of course, it could be a weapon as a tool, but it can be for both. And these are large projects. I'm very impressed with the scale of each of these and how much they've built.

And now I guess ending on more of a fun note of, uh, are there any, you know, hobbies or habits outside of work? Uh, this is just a common question we ask, um, anything beyond all of this that you do, maybe all of these are your hobbies already. Uh, you're right that, uh, I do consider, um, working on things like the instant database to be a, uh, a passion project. Um, uh, that's, that's close enough to, um,

my corporate profession that those two become a little bit mixed. I think stepping away from AI, I can say that I enjoy running and I did track and field for more than a decade and try and play the distance runner from time to time. And I was always a sprinter and I

Yeah, it's cooking and the like as well. Netflix. It's a little bit harder to talk about hobbies right at the end of the pandemic because the number of things one could do in public spaces went away for a while. I was rock climbing and all that, but it was hard to go to rock gyms for a while. Oh, for sure. Yes. I was also a sprinter, so that's cool. Cool.

Cool. Well, thank you so much for being on the podcast, Sean. Well, thank you for having me. It's been a pleasure talking. Awesome. Thank you so much for listening to this episode of Skynet Today's Let's Talk AI podcast. You can find articles on similar topics to today's and subscribe to our weekly newsletter with similar ones at skynettoday.com. Subscribe to us wherever you get your podcasts and don't forget to leave us a rating if you like the show. Be sure to tune in to our future episodes.