AI Setting Grades, ICE Pays Clearview, and Much More
30:10 Share

Hello and welcome to SkyNet Today's Let's Talk AI podcast. We can hear from AI researchers about what's actually going on with AI and what is just clickbait headlines. I am Andrey Kronikov, a third-year PhD student at the Stanford Vision and Learning Lab, where I do research on learning algorithms for robotic manipulation. And with me is my co-host...

I'm Sharon, a third year PhD student in the Machine Learning group working with Andrew Ng. I do research on generative models, improving generalization of neural networks, and applying machine learning to tackling the climate crisis. And as usual this week, we are going to discuss last week's big AI news. But before that, we actually want to ask you, our listeners, to tell us a bit about why you listen.

We've been doing this podcast for a little bit, and it seems we have some consistent listeners. And so we have created a survey at bit.ly slash LTA survey.

where you can fill in some easy questions about what you like about the show, what you don't like, some things we might want to change, etc. We'd be very curious to hear what you, our listeners, think about the show and why you listen. So please go to bit.ly slash LTA survey to give us some quick feedback.

That being said, let's go ahead and start on discussing this week's news. And there's a lot of it this week, so we're going to go pretty quick.

Starting with the article, ICE just signed a contract with facial recognition company Clearview AI, which was reported in The Verge. And the title is pretty much all there is to the article. It's about how the ICE, Immigration Enforcement Agency, signed a contract with Clearview, which is a company that sells their ability to detect who is in a given photo using facial recognition technology.

And the contract was for mission support. And there was a purchase order of $224,000 worth of Clearview licenses and ICE mission support tickets.

Dallas as the contracting office there. So pretty much a fairly large order of theory technology for ICE to presumably enforce immigration law in the United States. Well, this is slightly concerning.

We've covered many articles on Clearview AI before. And in May, they said that they would stop selling their app to private companies and would avoid transacting with non-governmental customers, which I guess they're still complying with with ICE. But they have previously worked with the NBA, Bank of America, Macy's, Walmart, etc., as well as private individuals.

ICE, however, it has been very controversial in terms of their treatment of people at the U.S. southern border with Mexico, which has included separating immigrant children from their families and detaining refugees.

I guess it makes sense that the federal government would be able to use this technology and facial recognition. But on the other hand, many people quite dislike what ICE does and really oppose it. And I suppose this might be an example for many people of facial recognition and AI broadly being used for negative outcomes. And so it's definitely worth being aware of and just

you know, continue to see what Cleaver is doing and where facial recognition is going. On the topic of flawed AI, our next article is from the Wall Street Journal. The quiet growth of race detection software sparks concerns over bias.

So companies such as Revlon are using race detection software from companies such as Miami-based Kairos to find out what lipstick women of different races or in different countries are wearing. And more than a dozen companies now offer race detection software on the market. And this is useful for photoshopping.

folks in marketing and also seemingly innocuous applications such as finding the right lipstick color for you. But some researchers and vendors are saying that the tech shouldn't exist because it has some, some consequences on undesirable consequences and side effects.

So governments have barred using race to decide which patients to treat, of course, and which job applicants to hire. But race detection software could introduce the possibility that corporations can make decisions based on a person's race, either intentionally or not intentionally. But it's really hard to detect and tease out from the software because it's hidden within these black box applications.

Yeah, I think this is very interesting. Personally, as this article notes,

I find that this whole thing of facial analysis has kind of flown under the radar. We've talked a lot about facial recognition and how it's been shown to be flawed, etc. But personally, I had no idea that there were already commercial solutions for classifying which race people were and that, you know, actually companies were using that in software. Sharon, were you at all aware of these things existing?

No. And I think, I think the benefits don't quite outweigh the risks here though. I, I definitely empathize with the benefits where it's very hard sometimes for me as an Asian American woman to find the right color, lip color and makeup colors since a lot of the stuff on the market in the Western world is, is not tailored to me. But yeah,

But I don't think that benefit outweighs the risks whatsoever. And it is a little bit concerning or a lot of bit concerning that there is a good number of offerings on the market for this software and that it could go past under the radar and be seeping in many ways to various decisions. Yeah.

Yeah, exactly. I think, I mean, there are possible ways to make this a way to improve personalization and actually benefit people, I guess, in your case, set to being example.

But the article does note that, for instance, race detection software poses the disconcerting possibility that institutions could intentionally or not make decisions based on a person's ethnic background in ways that are harder to detect because they occur as complex or opaque algorithms. And there could be various problems, for instance, for peoples of mixed race,

And if ethnicity recognition is used to push marginalized people towards specific products or offer discriminatory pricing, these are all things that seem a little bit problematic. So maybe there are kind of ways to have it both ways where you opt in and you say, yes, please use this.

I don't mind my ethnicity being used to help tailor the product suggestions. That could be a way, but it does seem to me and it sounds also to you that if this is being done in the background without any explanation, possibly in a flawed way, sounds a little bit problematic.

Right. And continuing the trend of problematic tech, our next article comes from Wired UK. Police built an AI to predict violent crime. It was seriously flawed. So the police, the UK police, has admitted that a flagship AI system to detect gun and knife violence had serious flaws and this made it unusable.

So the prediction system called Most Serious Violence, MSV, was part of the National Analytics Solution Project, which is part of the Home Office and funded with over 10 million euros to create machine learning systems for use across England and Wales.

And no data was really provided about the issue. Besides that, quote, a coding error was found in the definition of the training data set, which has rendered the current problem statement of MSV unviable. Something we probably haven't encountered before in research. I think that sounds kind of relatable. But this is kind of an interesting article, I think, because...

This is reporting on essentially something that didn't happen. So they were developing this prediction system. It was never deployed. And we just found out that it was in development and it was flawed and kind of sparked some discussions and some coverage there.

to illustrate where it could have gone. And questions have been raised about the potential to be biased towards minority groups and whether it would ever be useful for policing, even if it did work.

What's really interesting about the system is that they actually involve an ethics committee and the ethics committee is actually able to ask some pressing questions and get answers from them. And this is really just very much unheard of in the development of AI systems within policing in particular, but also more broadly in the development of AI systems.

So that is a great step forward, I think, in that case. The transparency around that and the involvement of folks in ethics is really important.

Yes, I agree. And also the article notes that the current thinking was that this predictive violence tool could be used to augment existing decision-making processes used by police officers, but not to make any decisions or bigger accommodation on its own. So it looks like, as you say, at least in this case, this was being done pretty mindfully and

And also, as we know from the results, they were at least careful enough to find this flaw and realized that the accuracy was very low and that this was unusable. But they're also considering how it could be used with these FX configurations. So perhaps actually a good example for how

this kind of pretty sensitive and ethically tricky kind of AI could be developed in the future. I would say it's pretty tricky even as a decision augmenter. AI is

Oftentimes, even if it's augmenting decisions, people will start to over rely on them as decision makers for them because it's just easier. It's like the lazy path. It's not that the person's actually lazy. It's just you already have someone who kind of gave you an answer. And if that person is or if that system is lazy,

right most of the time or only wrong when it's something where you have to be very, very attentive and careful, then you're probably not going to question it as much. And it's easier to just go with it. So I think, I think there, there isn't as much talk about how to use AI, build AI systems and design AI systems to,

be better at getting people to not over rely on them. I don't know exactly how that should be designed, but I would be very curious to see where that could go. If there's a way to better augment people where the AI system is more

Yeah, would be less relied upon. That's a great point. Yeah. But even if the AI system itself is maybe not biased and maybe it's meant to be used in some way that is good, even still, you need to be mindful and careful about how it actually ends up being used in practice.

And actually, we can go on to our next article, which is somewhat related to that. And it's titled Problematic Study on Indiana Parolees Seeks to Predict Recidivism with AI. And this is from VentureBeat. So quite similar in that there is some predictive kind of policing almost going on here, except here it's about whether people currently in prison would be recidivists and be in prison again.

This was funded by a grant from the Justice Department and was done with a collaboration between Florida State University, the University of Alabama Huntsville, and also the Tippecanoe County Sheriff's Department.

And also researchers at Purdue University Polytechnic were planning to use AI to algorithmically identify recidivism in released prisoners. So a big kind of collaboration between different departments and groups to try and apply AI to predicting recidivism. And the claim goal here was also not necessarily bad. It was to ID opportunities for intervention to help release rejoined society.

So the idea was to be used to avoid bad outcomes for people. But it seems like at least this article is making a point, and this is often true of AI, is that researchers maybe didn't fully address the potential for the AI perpetuating existing biases in society and basically

capturing all the existing biases and statistics of certain minorities being more likely to be recidivists. And then even if that's accurate, if put in practice with SAI and not questioned enough and, you know, over-lied upon, it could definitely be problematic. Something that I found pretty interesting in this article was the note that

within large data sets, historic biases become compounded. So basically as the more historical data you use, actually the more biases there are probably contained in those, uh, in that history, in those historical reams of data. And that definitely makes me think, wow, well, what data do we use? You know? Um,

especially as trends change over time or something like that. So that definitely got me thinking a little bit from this article. Yeah, and that's an interesting point because this article notes that the researchers' plan was to actually recruit 250 Paralees as they were released.

and then collect data, actually use bracelets that could collect real-time information like stress biomarkers and heart rate, while the smartphones also record a lot of personal data ranging from location to photos. So it's not clear if they primarily wanted to use historical data and augmented it with this additional collected data, but either way, it sounds like they really intended to

apply and do this in a data-driven matter as is kind of common today. The article also notes, and this is definitely worth noting, that other ill-fated experiments to predict things like GPA, grit, eviction, job training layoffs,

and material hardship reveal the prejudicial nature of AI algorithms. So as Sharon, you pointed out, even in large datasets, historical biases become compounded.

So yeah, there's kind of a whole category of things dealing with life outcome and kind of human nature and stuff like that, where it seems like maybe we can try to apply AI, but it's very hard to root out any problematic bias. And as you say, even if the intention is to only have AI help out, it really needs to be used very carefully to not just make things worse.

And speaking of grades in school, our next article from Harvard Business Review is titled, What Happens When AI is Used to Set Grades? So AI is used to predict grades for A-level or GCSE students, much the same as before with IB exams.

And this predicted based on previous performance, so school performance and teacher assessments. And there are, of course, issues with this AI that are entrenched in existing inequalities. Yeah.

Yeah, this was a big story. I saw it kind of blow up on Twitter. A lot of people were really affected because these are things that kind of, you know, determine if you can go to a good university in the UK, I believe. So it's a big deal for people's lives. And at least to some degree, this algorithm was now setting the grade instead of actually being able to take the test since COVID made it impossible to take this A-level exam.

And the idea, I think, was to make a normal statistical spread. So some people had to fail and some people had to succeed. And of course, if you're one of the people who failed based on your historical prior performance or whatever,

feels unfair and very problematic. And certainly I would feel like it's not fine for this algorithm to just say I would have failed without me ever having taken the test. How about you, Sharon? What was your take? I mean, I was really concerned about the people, the students in the poorest areas were marked down the most.

because they probably didn't have resources to study necessarily in the beginning, or it was just compounding a bias from the data. And so, of course, the results actually tied the fortunes of individual students to pre-existing inequalities of outcome. So exacerbating and reinforcing the bias

the pay gap essentially, and the, um, the gap in socioeconomic status. And so that's really concerning. I would be pretty upset while I would have been happy that I wouldn't have to take the test. I would have been upset that,

No matter what they would have given for me. Unless I got a good grade. So yeah, I mean, obviously people are going to be upset at this. This is kind of ridiculous. You can't expect this to... I'm actually surprised they thought people would be down for this. Yeah. Yeah, I agree. I think it's surprising that they went through with it the way they did. And kind of...

Not necessarily expected all of the disagreement they got over it, all of the anger they got over it, I think, rightly. And we don't have an article here yet, but...

I think this is a very rapidly evolving situation. So they're definitely currently reviewing the plan and it seems likely that they will change the outcome here probably soon. Because again, if you're one of the people who the algorithm said would have gotten a bad grade, it doesn't seem fair to do it that way, right? If there's no recourse, if there's no way to challenge the algorithm, I don't think people would be okay with that.

Right. And it's also hard to say that, you know, if you've been doing well this whole time, actually you get this free hall pass of not having to take it anymore. I'm not even sure if that's fair either. So why can't they do an online exam? I guess an online exam, you can cheat more easily and stuff. Yeah. I suppose it's, it's a infrastructure issue and they thought this was the easiest fix, but yeah,

You can also, as you said, definitely see why people have an issue with this. And it's surprising that this was kind of done the way it was given. It was very predictable. People would just not agree with it.

And on that note, maybe we can go on to a slightly more typical topic. We see an AI that we've talked a lot about, which is deepfakes. And there's a whole bunch of stories going on this week. So we're going to kind of jump through them and quickly go through them without going too in-depth.

The first story being "Pro-China Propaganda Act Used Fake Followers Made With AI-Generated Images From PCMag.com". And the title pretty much sums it up. There was a pro-China propaganda campaign dubbed "Spamoflage Dragon", which posted English language videos critical of Trump administration on Facebook, Twitter, and YouTube since June. And it was found that the

these videos were using AI-generated pictures and fake names so that they could not be found. So this...

Case had some pretty kind of easy to tell propaganda. These were awkward videos. They were not particularly well made, but it's still another example of usage of GANs and fake images and kind of pretending a future where this is super common and we see this happen all the time.

Andre and I were talking about this prior to our recording, but essentially what's interesting if you follow the link is that

The GAN they seem to use only generates people who they don't look Asian necessarily. So they used Russian names for the avatars of these fake profiles that post awkward Chinese phrases, I suppose. So it seems very...

poorly executed, but I think it is largely the idea that, hey, this is democratized so that anyone could kind of throw this up without any quality checks or thoughts at all. And this could definitely be refined if the person put in a little bit more effort. So that is very concerning. Indeed. And on the note of it being concerning, actually, we can go ahead and go to our next article.

which is stopping deep fake news with an AI algorithm that can tell when a face doesn't fit. So I guess the good news is, as you know, while sharing, people are working on various ways in which we can maybe mitigate this problem and actually detect when they are used properly.

detect when there is usage of GANs and deepfakes and so on and make it easier to combat. Yes, and the article highlights that

essentially that while deepfakes can damage reputations and spread misinformation, there are also methods being simultaneously being developed for the opposite, for essentially trying to be on the defensive as opposed to offensive. And it can also be used to prevent these quote unquote infodemics, which are an extra side effect for,

During during the COVID-19 pandemic and continuing the trend of deep fakes. Our next article and technology review is titled a college, a college kids. Can you move your cursor? A college kids fake AI generated blog fooled tens of thousands. This is how he made it.

So Liam Poor, a UC Berkeley student, asked a PhD student friend with access to the GPT-3 model to run a script that gave GPT-3, which is a recent natural language processing model that's been able to generate very realistic text.

So he gave this friend a script with a headline and intro for a blog post. And then GPD3, given that input, was able to spit out several completed versions of blog posts. Liam Poor then copied and pasted those outputs with little to no editing. And this first post actually reached the top of Hacker News.

And I don't know about you, Andre. I think I saw it there too. I think I saw the discussion when he also had a post that kind of revealed that, uh, that happened. And there was a lot of discussion about that being the case. Yeah. Right. And I think it made it much more realistic. What was interesting was I think, uh,

Yeah, definitely. You could tell in the comments who read the entire blog post, who read just the headline, and how far people had gotten in the post based on people's comments. But overall, this experiment shows that people could use GPD-3 and similar tools to generate a lot of clickbait content, and that could be pretty dangerous in the age of misinformation. Yeah.

Yeah, I think Liam Poor, who did this, actually mentioned that he went after topics that tend to get a lot of clicks, like productivity and stuff like that, and used those topics because you could write fairly, I don't know, wishy-washy kind of recommendations, etc. Pretty short, not super in-depth kind of posts and get a lot of attention. And so he himself actually says that

he has he's troubled that maybe people could use this to generate a lot of clickbait content and not necessarily create any like super deep fake text not pretend for people but still if the internet is full of these sorts of clickbait things it'll make it even more

full of useless garbage, which I guess we do not want. Exactly. And that brings us to the defensive side of our last article titled Photoshop will help ID images that have been photoshopped.

This is an article in Wired. So Photoshop, which is often used for digital fakery, either deepfakes using, typically deepfakes using graphics,

But Adobe that makes Photoshop and puts out Photoshop intends to add technology to tag these images, essentially with metadata or some kind of watermark to help ID these deep fakes or manipulated images. But this authenticity tagging will require...

obviously other vendors and other platforms to support the standard as well. And this is a way to basically identify when something is coming from Photoshop. So the system can only be applied to a small fraction of online content, of course. A lot of deep fakes are coming not from Photoshop necessarily, but perhaps this could help combat it. And I'm glad, I'm actually very happy to see Adobe doing this, even if it won't cover everything.

What are your thoughts, André? Yeah, definitely. I think, as you note, it's to require a lot of different players to agree on a standard. And also this article notes this. And this approach we have, this content authenticity initiative,

seems like a good push and having such a big company as Adobe pushing it seems like maybe they will actually succeed in doing it. I'm also finding it interesting that there is an article, I mentioned that the first live test of the content authenticity initiative in the news business will most likely come from the New York Times because the newspapers had a R&D, Mark Lavelle,

wanted to test this already. So I think, yeah, this has long been one of the suggestions of deepfakes. You add more media data, you make it easier to trace media and actually say that it hasn't been tampered with. And it's coming together maybe with this push from Adobe, and I'm looking forward to hearing more and hopefully this actually happening and preventing deepfakes from taking over our Twitter feeds.

And with that, thank you so much for listening to this week's episode of SkyNet Today's Let's Talk AI podcast. You can find the articles we discussed here today and subscribe to our weekly newsletter with similar ones at skynetoday.com. Subscribe to us wherever you get your podcasts and don't forget to leave us a rating if you like the show. And be sure to tune in next week.