Mini Episode: ICE/Clearview, Race Detection and Schools
06:03 Share

Hello and welcome. This is Daniel Bashir here with Skyden today's Week in AI. This week, we'll look at facial recognition and schools.

If peanut butter and cream cheese is a match made in heaven, then what we have this week is a match made in hell. Immigration and Customs Enforcement, or ICE, signed a contract with facial recognition company Clearview AI for mission support. The Verge reports that the purchase order of $224,000 describes Clearview licenses and lists ICE Mission Support Dallas as the contracting office.

Clearview has become infamous in the past few months for its aggressive scraping of images from social media sites, the resulting season desist orders levied against it, and its worryingly large list of customers exposed by BuzzFeed News. But its partner is hardly better. ICE is known to use facial recognition technology. The Washington Post reported that last month, ICE and the FBI had accessed state driver's licenses without the knowledge and consent of drivers.

ICE has also been criticized for its practices at the southern US border, such as separating immigrant children from their families and detaining refugees indefinitely. One can only imagine with a little fear what might come of this partnership. While general facial recognition technology is becoming more in vogue, more than a dozen companies also offer some form of race or ethnicity detection.

As the Wall Street Journal reports, companies have started using such software to understand how certain customers use their products, who looks at their ads, or what people of different racial groups like, among other uses.

While the field is still developing and it is an open question how the technology will be used in the future, researchers and companies have recognized the technology's potential to drive discrimination. While some companies might feel as though race detection software is not harmful when used for retail purposes, some researchers and vendors say the technology should not exist.

Race detection software poses the risk that institutions could, intentionally or otherwise, make decisions based on a person's ethnic background that are difficult to detect because they occur in the context of opaque algorithms. For example, companies could use ethnicity recognition to push marginalized people towards specific products or discriminatory pricing.

Thus, even seemingly innocent uses, such as Revlon Inc.'s hiring of Miami-based Kairos Inc. to use a facial analysis algorithm to determine what lipstick women of different races were wearing could manifest in discriminatory behavior.

Next up, how would you feel if instead of having to take a test, an AI gave you a grade? Now how would you feel if that grade was the main factor in deciding where you spent the next four years? The Guardian reports that as it became clear that A-levels and GCSE examinations could not happen as usual, this is precisely what happened to the students scheduled to take them.

But instead of taking into account information about individual students, regulators prioritized achieving a normal statistical spread of results and avoiding grade inflation by combining teacher assessments of likely grades with pupils' previous performance and the past record of their school as a whole.

But tying predicted grades to past records and school records merely entrenches pre-existing educational inequalities that bar talented students from achieving the same results as their equally talented peers from more fortunate backgrounds. The likely outrage such results will inspire has already been presaged by the very similar steps taken by the International Baccalaureate Organization.

Students whose schools participated in the prestigious diploma program also had their final grades decided by an AI that based decisions on their past work and other historic data. The result was a fiasco.

The IBO's mishandling of the situation was unfortunate, allowing AI to make life-altering decisions without adding checks and employing a careful appeals process will inevitably result in chaos. Such irresponsible use of AI has the potential to erode public trust, despite the fact that when used properly and paired with human expertise, AI can help dramatically improve the quality of decision making.

Finally, the reopening of schools this fall has been a contentious issue, with teachers uncertain about their safety while the federal government issues threats to schools that choose not to reopen. The reopening of schools would undoubtedly be good for almost everyone involved in a number of ways, but it also poses health risks that cannot be taken lightly.

Schools that do intend to open will have to follow strict precautions, and the private K-12 Meadows School of Las Vegas has an interesting plan to use an AI-powered thermal screening system to check students' temperatures.

Vox reports that Remark Holdings, the AI company supplying the technology, has been providing a similar system to over 100 schools in China for over a year, and is repurposing its tech to assist semi-public places reopening during the pandemic. Remark Holdings is just one of the number of firms that already sold surveillance products and are adjusting their technology to the pandemic, offering tools that detect whether students are wearing masks, to measuring how well people are social distancing.

It is hard to know whether any of this technology works as intended, but schools seem ready to give it a try. That's all for this week. Thanks so much for listening. If you enjoyed the podcast, be sure to rate and share. If you'd like to hear more news like this, please check out Skynetoday.com, where you can find our weekly news digests with similar articles.