More Ethical Quagmires, More Surveillance, and more Academia Talk
27:03 Share

Hello and welcome to Skynet today's Let's Talk AI podcast. We can hear from AI researchers about what's actually going on with AI and what is just clickbait headlines. I am Andrey Korenkov, a third year PhD student at the Stanford Vision and Learning Lab. I focus mostly on learning algorithms for robotic manipulation in my research.

And with me is my co-host. I'm Sharon, a third year PhD student in the machine learning group working with Andrew Ng. I do research on generative models, improving generalization of neural networks and applying machine learning to tackling the climate crisis. And Sharon, we were just chatting about how Veneerub's deadline is coming up on Wednesday this next week. And I believe you're trying to submit. So how has that research been going for you lately?

I spend many sleepless nights and working very, very hard towards the Wednesday 1:00 PM PST deadline, which has been ingrained in my head.

I'm sure, yeah, at least for us at Stanford, it tends to be the case that every deadline comes with a real few weeks of grind and working very hard to make it. You always try to be ready early, but it never seems to work out that way. So hopefully this admission goes well and you can get some rest after that.

Thank you. Yes. And invariably the cluster always goes down a couple of weeks before. So that's, that's always funny. There's always some exciting surprises and problems to deal with when you're trying to finish things up. So you just have to kind of keep going and hopefully things work out. Yeah. Wish me luck.

Yeah. But for now, let's go ahead and dive in and chat about last week's AI news. And you can now think about research for a bit, hopefully. And we can start with this first article, which comes from Aeon, an online magazine titled Algorithms Associating Appearance and Criminality Have a Dark Past.

And this is not really news related. It's more of an overview of a topic, which is the general idea of using AI to predict if someone would be likely to commit a crime or has committed a crime based just on a photo of their face.

So in particular, this article focuses on a paper from a few years ago by the authors Xiaolongwu and Shi Zheng that claimed to have trained an algorithm to identify criminals based on the shape of their faces, meaning a photo of their face, with an accuracy of 89.5%. So it could tell you if someone is a criminal or not looking in a photo with an accuracy of 90%.

And this article goes through not just saying that obviously this is probably not true, right? You cannot tell if someone is a criminal based on look at their face. It also talks about how there's a long history of this kind of idea of talking about people's personality based on their appearance and their face, which is now called phrenology. And basically highlights that

Obviously, this is problematic and there's different

methodological problems with this research, but also there's moral problems. So basically there's a general problem with doing this kind of research, which is inherently flawed. And the researchers said that they were doing research for purely academic discussion, just for curiosity. And the article makes a point that you really can't use that excuse that you're doing it just for research. That's not enough. And it concludes with a

saying that for scientists to take their moral responsibilities seriously, they need to be aware of the harms that might result from their research. Spelling out more clearly what's wrong with the work labeled phrenology will hopefully have more of an impact than simply throwing out the name around as a result. What's interesting is that the author said they were, quote, deeply baffled at the public outrage in reaction to their paper and that it was intended, quote, for pure academic discussions.

And also that some commentators argue that facial recognition should be regulated as tightly as plutonium because it has so few non-harmful uses. So those are kind of the two different sides at war here. Yeah, and again, there's some pretty stark problems with research in the first place. So in the study of AI for criminality or in a recent study,

The data we're taking from two very different sources, mock shots of convicts and pictures from work websites for non-convicts. And the algorithm could easily differentiate two different types of photos. So when you see this kind of research, you should A, be aware that if they are claiming a lot of accuracy, probably there's methodological issues, very basic problems in how the data was created that allowed that high success rate, not that it actually works.

But secondly, you should also be aware of that in general, there's a long history of kind of colonialism, class structure, racism to as a basis for this kind of research. And that's sort of the underlying reason.

belief system that this kind of research promotes. And you should just be very skeptical, I think. And I would assume, Sharon, that you would agree that this is very problematic and probably shouldn't be accepted even in the paper reviewing for conferences like NeurIPS. Yes, it's very worrying. And I'm glad that NeurIPS has added the broader impact statement as a requirement for all paper submissions. So

As I've been writing my papers, I actually have been, I've been writing those sections and it has been actually a really good way to think about my research a bit more and reflect on it and think, continue to step back at different levels, you know, like initially, oh, who does this benefit societally? Who might this harm? And then stepping back even further, you know, like not just, oh, this impacts the research community, but,

society at large and there are many many different levels of society at large as well depending on who ends up applying your research in some way or or reading your research and using it to some degree so I think it's I think it's been really helpful to have that section and I hope that something like this would also enable the authors to to self-reflect a little bit more as well

And I do hope that conferences will be taking such things into consideration much more strongly. Yeah, exactly. And that also goes back to this conclusion of this article that says that for scientists to take their moral responsibilities seriously, they need to be aware of the harms that might result from their research, which is exactly what the broader impact statement of the NeurIPS conference is meant to kind of promote, that researchers think more about the broader impacts

And hopefully avoid research of this sort that seems to be almost certain to do more harm than good, at least in the way that it has been done. Well, enough on research. We'll come back to that in a later article. The next article that we have is titled Americans Don't Know Why They Don't Trust Self-Driving Cars.

And it's published in Gadget. And so self-driving cars have been a pretty hot topic in technology for the last few years. We've seen companies from Tesla to Google's Waymo to Argo AI enter that race to develop safe, functional self-driving vehicles.

But the road has been pretty bumpy. So we've seen issues with Tesla's autopilot mode in an accident in Tempe, Arizona, involving a self-driving Uber vehicle. And so with all that in mind, it seems obvious that Americans who have been close to all that action might be less than enthused about the prospect of self-driving cars.

And mistrust definitely abounds, but a study conducted in February and March this year on behalf of Partners for Automated Vehicle Education, or PAVE, found that this mistrust actually comes from somewhere else. And so for most respondents, the more relevant issue to their mistrust is that they hadn't had the chance to experience a self-driving technology firsthand.

In fact, quote, 58% said they would have greater confidence in autonomous cars if they could ride in one. So this is really interesting. So mistrust I think is largely right now derived from the unknown of this uncertainty of what it actually feels like. So maybe if you experience it, you get a sense of where it might be faulty. Um, and I think that definitely makes sense in people, uh,

definitely don't trust things or trust things that they don't know as much. I wonder, have you actually gotten to ride in a self-driving vehicle? I don't think I have yet. I have not. You're right. I have not yet, but I've seen many.

Yeah, they're all over Palo Alto, all over New Stanford, San Francisco. So I sort of feel like I know them, sort of, but still they're in this tasting phase. And I agree with you that it kind of makes a lot of sense that a lot of Americans, they may have heard some things about it, seen some articles, but because you haven't had that firsthand experience, you don't really know what to think about it.

which makes sense. And in fact, this article says that from the survey, 48% of people said they would never get into a self-driving taxi and almost yet 20% think that technology will never be safe. And if there was the ability to test it out, maybe have a person in the vehicle, you know, uh,

being there as a safety driver, probably, hopefully, that opinion would be changed. Yeah, it's kind of a catch-22 because it needs to be super, super safe for people to trust it and have it be out there. But we might need some kind of testing for it to be safe. So it may be kind of an anti-fragile kind of situation where...

The more it breaks, the more it gets better. But we might not be able to afford it breaking.

Yeah, it's actually, it's also interesting that speaking of braking, that majority or 87% in the survey said that they knew nothing at all or just a little about the Tesla autopilot incidents. And actually, if you go to Wikipedia, there's a page for fatalities from self-driving cars. And there's a list of all the ones that have happened so far. And there's, I think, about half a dozen.

already over the past few years as people have been testing, with the majority being from Tesla cars. We've actually, I think, talked about one or two. So it seems like there's a mix of people are hesitant because they haven't experienced it, but at the same time, maybe we should know

Also, that there is this real cause for concern in that there are some incidents that have happened. And so when you use advanced driver assistance systems, which is what test lighter pilot actually is, it's basically advanced driver assistance. It's not full self-driving. You need to know about its limitations and not trust it too much as more advanced.

companies roll out things like Tesla's features. Are you are you fond of the idea of getting self-driving capabilities for your car, Sharon, or do you prefer to be the driver? I definitely don't prefer it to be the driver, but it's also an asterisk there because I do not have a driver's license.

I've actually been holding out for self-driving cars for quite some time since the promise 10 years ago was that it'd be in five years and yeah. And still we are waiting and it seems like you'll be waiting for a while more.

But I guess for both of us who don't drive, we kind of like the idea of self-driving cars coming here so that we can rely on that technology and continue to not buy into cars anymore.

But moving on from that topic, let's go ahead and move our next article, which is how a Chinese AI giant made chatting and surveillance easy, which is from Wired. And it's all about how Shenzhen-based iFlytech has received a lot of press about how it has made a breakthrough in AI-powered voice recognition.

but was also placed among the companies on the U.S. trade blacklist because it is a major Chinese company doing AI. Now, we've heard a lot about facial recognition-based surveillance from China, but this article makes a point that this company is also seemingly involved in efforts to do surveillance based on audio and voice recognition.

So iFlytech released a major consumer app, iFlytech Input, now 10 years ago, in 2010, that allowed people to dictate text anywhere on their phones. So a lot of people are using this, and iFlytech obviously invested a lot in being able to recognize what people are saying and to dictate that sort of thing. And this...

Article makes a point that iFlytech's technology has helped the Chinese government to integrate audio signals into its digital surveillance network. More specifically, in 2017, Human Rights Watch published a report detailing iFlytech's government work. And in that report, a researcher of the right group said that the company's tools are an essential part of the Chinese party's plans to build a digital totalitarian state.

and that their voice biometric technologies made tracking and identifying individuals possible.

iFlytech also enables pretty clear security work. So in 2012, the Ministry of Public Security in China purchased machines from iFlytech focused on intelligent voice technology. And the ministry chose the Anhui province where iFlytech is headquartered as one of the pilot locations for compiling a voice pattern database. So this is basically a catalog of people's unique speech patterns that would enable authorities to identify those speakers by merely the sound of their voice.

Yep. And according to Human Rights Watch, off-light technology also appears to enable surveillance in Xinjiang region in northwest China populated by predominantly Muslim Uyghur minority group, which has been the subject of a lot of discussion.

oppression in recent years. So obviously a pretty downer article and more on surveillance, which seems to be a major trend in the news over the last few months. We keep discussing more and more facial recognition, now apparently also voice recognition. I guess

One positive is that this doesn't seem to be the case in more Western states like the US and Europe. So here we haven't heard anything about this sort of system so far, at least. But I wonder, Sharon, reading all these news and discussing all these news articles, has your kind of

Overall apprehension of surveillance and kind of the amount of AI being built just to recognize us been increased. Have you been thinking about it more? Yes, I think when there's a big entity like a government that wants to impose something using AI, it can become more effective because they have control over perhaps a lot of people. And that does become concerning.

So the Uyghurs, where the Chinese government was tightening its grip, was using iFly tech to make the residents install these apps on their phone that give biometric data at regular security checkpoints. Okay.

and have cultural inspectors in their homes. And so that's a little bit concerning, I would say, because that starts to make me think, oh, isn't that how, you know, I just watched The Pianist, the movie. Isn't that how kind of the Holocaust started in the very beginning stages? And so that really concerns me there. And that when surveillance could be done in a very widespread way, especially with now speech on top of text,

It's hard to essentially figure out what you should be, how you can have any privacy at all. And yeah. Yeah, I think personally, I found it pretty concerning to keep seeing this pop up and be reminded of how much effort is going into building these sorts of systems.

I think this is part of why we should be a little bit careful about worrying too much about far-off AI, you know, AGI taking over and Skynet and so on, because there's a lot to be worried about already, and we should already be trying to avoid the negative implications of a lot of AI being built today. And this article is just providing yet more details on that.

how real this is, how it's already happening and how we should be aware of it and probably at least aware of it if not using kind of our position as researchers, as people with some knowledge of AI to make people aware that this is a concern and that this is really something we should try and hopefully prevent happening at least in the US.

Yes, and I think it's actually particularly concerning that this article also talks about how iFlytech has an agreement with MIT's CSAIL, which is MIT's AI lab. And essentially, iFlytech gave them an undisclosed sum of money in return for the prestige of the MIT brand. And I really think that institutions should think about when they should partner with such institutions

entities, such companies, and really think about the ethics of that. And of course, this is me reading this after knowing that MIT had a huge thing with Jeffrey Epstein. So I'm starting to question how they filter and decide on who's allowed to give money to MIT.

Exactly. Yeah. So as industry and academia get closer and closer, as governments get more and more investment in AI, us researchers will probably have to become more and more aware of these ethical boundaries. And, you know, sometimes probably you have to reject money and funding and stand for principles. No doubt, partially for surveillance, partially for military. This is likely to just keep growing.

Switching gears back to academia research, there was an article posted called Research Summary, Troubling Trends in Machine Learning Scholarship. And a few of the articles we've covered in the past have examined issues in machine learning scholarship. For example, Yasho Avengeo discussed problems with our conference publication system. And the authors of this paper said,

"draw attention to some common issues that have become exacerbated in the field due to a thinning experience reviewer pool who are increasingly burdened with large numbers of papers to review and might need to default to checklist type of patterns in evaluating papers." And so the authors identify four main areas of concern. The first one is a failure to distinguish between speculation and explanation.

Two, a failure to identify the source of empirical gains. So maybe your model does well, but you don't know why it's doing well. Three, the use of mathematics that obfuscates or impresses rather than clarifies. I've definitely seen this. You just throw in a math equation to make it seem more complex.

4. The misuse of language such that terms with other connotations are used, or by overloading terms with existing technical definitions.

And so from these, the authors do actually provide some recommendations on how we can improve this by sharing that we should consider why we're getting certain results as authors of these papers and what they mean more so than focusing on just how people get these results.

And then for reviewers, the guidance is to cut through the jargon, the unnecessary use of math, the even anthropomorphization,

That's used to exaggerate results, which is basically applying human properties to a lot of these models and instead critically asking the question why the authors arrived at results in evaluating arguments for strength and cohesion rather than just looking at the empirical findings that compete in producing better state-of-the-art results. Absolutely, yes. Yeah.

I think it's really challenging because the number of papers coming in is exponentially growing. And, of course, the reviewer pool, as I said, is diluted essentially with people who are not as experienced. But also reviewers are overloaded with papers to review.

Yeah, and I think actually NeurIPS itself instituted some new policies this year to help with reviewing burden. So some of them were like desk rejections or some of the papers were not even reviewed. They were just looked over. And if there was some strong signal that they would be not accepted, they were just kind of cut off early in the process and things like that, which also spawned a lot of discussion when it was announced.

I wonder if you recall those changes, Sharon, and if you thought they seemed like a good idea given these problems. Yeah, I think it definitely is a good idea. I think how to filter and sift through papers, which ones to desk reject, can be quite a challenging task since often the best, most impactful research is

are those that are the most polarizing and that might actually be rejected first before it's accepted because it is so groundbreaking, right? So that's the nature of good research. So I think it could definitely be challenging, though having reviewed lots of papers, there are definitely some that should be desk rejected because they're just kind of incomplete. Yeah. Yeah.

Yeah, I've sort of been fortunate in the thing that most of my reviewing hasn't been of outright terrible papers. Although there's been a lot of sort of middling work I've seen that hasn't been particularly exciting, but, you know, sort of useful.

And yeah, maybe for people outside of academia, this paints kind of a reasonable picture of where the field is at, where we've had this huge growth phase and now there's just a ton of work coming out. A lot of it is very incremental. A lot of it is pretty sloppy, honestly, as far as research goes. And so this paper argues that we have had these trends and we should be trying to do something about them. And I guess the positive side is that

Things like the changes in the neuro-previewing system, things like adding the public impact statement, things like that do show that we as a field are trying to deal with our growth and all these trends and hopefully kind of figure out how we can have a balance of scale and actual good scholarship.

And with that, thank you so much for listening to this week's episode of Skynet Today's Let's Talk AI podcast. You can find the articles we just discussed here today and subscribe to our weekly newsletter with similar ones at skynettoday.com. Subscribe to us wherever you get your podcasts and don't forget to leave us a rating if you like the show. Be sure to tune in next week.