What happened when AI went after welfare fraud
47:14 Share

can say to my new Samsung Galaxy S25 Ultra, hey, find a keto-friendly restaurant nearby and text it to Beth and Steve. And it does without me lifting a finger, so I can get in more squats anywhere I can. One, two, three. Um, will that be cash or credit? Credit.

Galaxy S25 Ultra, the AI companion that does the heavy lifting so you can do you. Get yours at Samsung.com. Compatible with select apps requires Google Gemini account results may vary based on input check responses for accuracy. Support for this podcast comes from Is Business Broken? A podcast from BU Questrom School of Business. Stick around until the end of this podcast for a preview of a recent episode about how to fairly compensate workers.

WBUR Podcasts, Boston. This is On Point. I'm Meghna Chakrabarty. Elon Musk does not like Social Security, Medicare or Medicaid. He was on Fox Business just this week, plainly saying what he'd like to do with those programs. Most of the federal spending is entitlements. So that's like the big one to eliminate.

Musk's Department of Government Efficiency, or DOJ, is reportedly using artificial intelligence to aid in those government-slashing endeavors.

Last month, the New York Times reported on Thomas Shedd, a former Tesla engineer who's now head of the technology services department at the Federal General Services Administration. Shedd reportedly told staffers that AI would be used as a tool to, quote, detect fraud and waste.

Asked just last week if AI was indeed crawling through government databases, communications, and contracts, Musk told reporters, quote, right now, we're not using that much AI, end quote.

But that hasn't stopped congressional Republicans from celebrating Musk's use of AI or his goals. Here's House Speaker Mike Johnson. He's created these algorithms that are constantly crawling through the data. And as he told me in his office, the data doesn't lie. We're going to be able to get the information. We're going to be able to transform the way the federal government works at the end of this. And that is a very exciting prospect. It is truly a revolutionary moment for the nation. It's true.

The data doesn't lie, or the data as I prefer to call it. But the conclusions made by an algorithm based on what it finds in those data, well, that's a little different. Because AI is extraordinarily good at completing program tasks rapidly. It can analyze massive mountains of data very fast. But should it then be making decisions about federal programs? Which ones to keep? Which ones to cut? Who gets benefits and who does not?

Well, globally, there are already examples on how this can go wrong without tight oversight. In the Netherlands, Prime Minister Mark Rutte's government resigned in 2021 after a major AI scandal there. An investigation found that 20,000 families were falsely accused of committing child welfare fraud. A court ordered the Dutch government to repay approximately $32,000 to each affected family.

In Australia, a system called Robodebt accused 400,000 welfare recipients in Australia of misreporting their income, and those recipients were fined significantly. Robodebt was implemented in 2016, and by 2019, a court ruled that the program was unlawful, and the government was forced to repay $1.2 billion, according to Reuters.

And here in the U.S., a lawsuit was filed and settled in 2024 against the state of Michigan when an anti-fraud algorithm indicated widespread unemployment insurance fraud. The algorithm was wrong, and 3,000 plaintiffs were reimbursed to the tune of $20 million.

Well, Amos To is senior counsel of AI and national security for the Brennan Center for Justice. He previously worked as the senior researcher of AI at Human Rights at Human Rights Watch. And while he was there, To wrote extensively about the use of AI in welfare systems.

And he says the pattern of predictive algorithms and AI cutting services is something that should concern everyone, not just the people receiving benefits from welfare systems. It's like no surprise that they became like they were like the first playgrounds for AI experimentation. Right. And like something we have worn for a long time is that like what is being experimented on when it comes to people experiencing

of low income, who have less means of fighting back, will invariably be rolled out to the broader population. So this hour, we want to take a much closer look at how AI could potentially help states and the federal government when it comes to welfare systems, what works, and also what doesn't work.

And what impact could AI technology have on the future of social services? So Kevin DeLiban joins us. He's the founder of Tectonic Justice. That's T-E-C-H-T-O-N-I-C. It's an organization dedicated to helping low-income families confronting the use of AI in welfare systems. And he joins us today. Kevin, welcome to On Pointe.

Hello, Meghna. How are you doing? I'm doing well. So you have had some firsthand experience. Obviously, that's why we've invited you on the show today with a very particular case of how AI had an impact on benefits received by folks. Can you tell us a little bit about that story?

Sure. So, Medicaid is the state's only health insurance program for low-income people, and it will pay for in-home care that people who otherwise would have to be institutionalized in places like nursing facilities would receive in order so that they can stay in their homes and out of those facilities. And it's much better for their independence and much better for, generally, much cheaper.

And so what happened in 2016, the clients on this program who had been receiving a set number of hours for many years suddenly started calling with news of devastating cuts seeking help to fight them. These are people who have cerebral palsy, quadriplegia, conditions that don't get better. And the state was cutting their hours from, say, seven or eight hours a day of care, which was really not enough, but...

folks would try their best to get by to something like four or five hours a day of care. Kevin, can I just jump in here? Sure. This is the state of Arkansas, correct? Oh, yes. Pardon me. The state of Arkansas. Yes. It's okay. Yeah. No. And you were working at Legal Aid in Arkansas at that time? Correct. Okay. So go ahead.

Sure. So clients would be calling reporting this devastating loss of hours that meant people were lying in their own waste, it meant people were getting bedsores from not being turned, just intolerable human suffering. The only clue that they had that something funny was up is when they would ask the nurse, the same nurse who came to assess them last year and the year before, the same nurse who gave them six, seven, eight hours day of care, "Why are you cutting my care?"

And universally, the nurse would say, it's not me, it's the computer. And I heard that story from a couple dozen people, figured, oh, there's something funny going on here, not in the ha-ha way. And we'd look into it and find that the state of Arkansas had implemented an algorithmic decision-making system to determine how much care to give these folks. Okay. So when was this algorithmic decision-making system implemented? Yeah.

January of 2016. 2016. And what was the original purpose for it?

The state waffled a lot on what they said the justification was. Ultimately, it was to cut benefits. That was the purpose of it. They would give various justifications about wanting to standardize or objectify the process, things like this, that we ended up proving were lies. Okay. I want to know exactly how this played out because I've seen some stories here that said

This algorithmic decision-making system was implemented, but were people who received the benefits, were they asked about how much care did they need? How much did they receive? How often did they need specific types of care around the house? And then that stuff was inputted into this algorithm? Yeah.

No, not exactly. So the first part of the process was a 286-question assessment. So the state nurse would come out and ask you 286 questions, usually taking an hour and a half to two hours to do so, which is an exhausting process. At the end of that, the nurse would push a button, and the responses from those 286 questions would be run through an algorithm that would group

It couldn't be deviated from at all.

I want to know more about that. Yeah. Sure, sure. So the way the algorithm works is of those 286 questions, it turns out only around 60, give or take a few, ever matter. But they don't all matter in the same way. They only matter sometimes when aligned with other factors. So the best way I've learned to describe it, and this is almost 10 years on, is as constellations, right? Sometimes the stars...

form something when you see them with other stars, but other times they're just a star, right? That's sort of how the algorithm worked with those, with those, uh, questions from the assessment that actually mattered. Some of them mattered some of the time, and there was never any explanation about which mattered, which at the time, and the state couldn't explain it. Um, let alone, um,

expecting beneficiaries who are on the program to suddenly understand how come their care is being cut drastically when their conditions haven't been approved. Now, I'm seeing that at least at one point in time, the state of Arkansas had said that

The AI system was supposed to eliminate what we know is a potential problem, which is human bias, right? That like people could be, you know, could have favorites. They could increase the amount of benefits someone's getting for sometimes arbitrary decisions, right? Yeah.

That's what they said, and that was one of the lies that we proved. So if you thought that that was actually an issue, that somehow a nurse in one part of the state was giving more or less hours than a nurse in another part of the state, you would think there'd be a trail of that. There'd be some paper identifying that this was a problem. Some nurse supervisor or some program supervisor would have talked to a nurse to talk to them about the issue. There would have been a little study, something. None of that had ever happened.

No nurse supervisor had ever talked to any nurse about giving too few or too many hours. There was no paperwork showing that. So the state was really lying. What they were trying to do is cut benefits. And that's what the algorithm did. And it did it drastically. And, um...

in a way that was just abjectly cruel and that devastated the lives of several thousand disabled people. Wait, so this is the thing about AI, right? It does what it's told to do, and it does it really, really well. So did you present evidence in court? Because this, of course, did go to trial, right?

and you and the people you're representing won that trial, did you find evidence that you presented in court that the purpose of the AI, the AI was told to cost cut? Yeah, I mean, that's what it was programmed to do. So the best case scenario, so under the previous system where nurses decided how much care to give people within guidelines established by the state, the maximum was eight hours a day of care. And again,

That is not enough for most people with cerebral palsy or quadriplegia. Most other states actually provide significantly more than eight hours a day of care as a maximum. When the algorithm came in, the best case scenario for most people was five and a half hours a day of care. So there was no scenario in which you got subjected to the algorithm and somehow won an increase. And that's where the shell game is of this, right? Is your whole possibilities are structured differently.

to decide how much of a cut you get, not whether there's any chance of actually getting an increase in hours, at least for the people at the highest levels of need. Okay. Kevin Dillibond, stand by for just a moment. We are talking about the use of artificial intelligence in social service programs, not just at the federal level, but at the state level as well. So lots more in just a moment. This is On Point. ♪

Support for On Point comes from Indeed. You just realized that your business needed to hire someone yesterday. How can you find amazing candidates fast? Easy, just use Indeed. There's no need to wait. You can speed up your hiring with Indeed.

and On Point listeners will get a $75 sponsored job credit to get your jobs more visibility at Indeed.com slash On Point. Just go to Indeed.com slash On Point right now and support the show by saying you heard about Indeed on this podcast. Indeed.com slash On Point. Terms and conditions apply. Hiring? Indeed is all you need.

Support for this podcast comes from Is Business Broken? A podcast from BU Questrom School of Business, which asks in a recent episode, how do we build trust in employee pay? Employees really want to know, is the process by which they are paid fair? Follow Is Business Broken wherever you get your podcasts and stick around until the end of this podcast for a preview of the episode.

At Radiolab, we love nothing more than nerding out about science, neuroscience, chemistry. But, but, we do also like to get into other kinds of stories. Stories about policing, or politics, country music, hockey, sex.

This is On Point. I'm Meghna Chakrabarty. And today we are looking at the use of artificial intelligence in social service programs and how it at the federal and state level, perhaps more importantly,

is increasingly being used to determine who receives benefits and who does not. Kevin DeLiban joins us today. He's founder of Tectonic Justice. It's an organization which helps low-income people who've had their benefits changed by the use of artificial intelligence. And he was also...

in a case that unfolded in Arkansas when the state there started using AI to determine certain welfare benefits. And Kevin's been telling us that story. And Kevin, hang on here for just a second because we actually also spoke to the person, the man behind the company, Kevin.

that developed the tool that the state of Arkansas used. Now, that company is called InterEye, and Professor Brant Fries designed the algorithm that InterEye uses, and used in Arkansas. So we spoke with him, and he tells us that statistical modeling, broadly, is very different from full-blown AI. Right.

The conclusions AI draws are sometimes shrouded in mystery, Professor Fries told us, but statistical algorithms like the ones he uses are reviewed, he says, by other scientists. We say, here's the statistical methods we used. Here's the data that we used. Here's what the variables that we considered. Here's why we considered these variables. What's the conceptual framework of why we think these variables are important to be considered?

we put all that into a scientific paper, it gets peer reviewed, people look at it, other scientists may say, "Well, why didn't you consider X or Y?" And then we say, "Well, we did and it didn't work." Or we can say, "Well, good question." We reanalyze the data and

Uh-oh, you know, we didn't consider this and we put it in the model and now it's a model really changed. It really looks different. We've got a better model than we had before. So essentially he's saying there that the kind of model that he uses, a statistical algorithm, is more transparent than full-blown predictive AI. But Fries also told us that the scientific review process doesn't completely...

insulate the algorithms from human error. We sort of put it out there and say, here it is. If you think it's good, you're welcome to use it. We can't tell a state that they have to use it. We can tell a state it's available. It does such and such. Here's the scientific article about it. If you want to evaluate it, we'll ask for any questions.

it's up to you to use it, and they can use it or they can misuse it. Now, they can use it for the wrong thing. We say you have to use our whole instrument. Don't change it because then the science goes away of the instrument and its reliability and validity. Okay, this is a really interesting point because Professor Friese tells us InterEye's algorithm wasn't the problem.

He says the problem was that the software company that implemented it changed some of the algorithm's functions to the detriment of the outcomes. It only came out after the trial when we started to go back and say, well, when I calculate it, I get this answer. And you're telling me that your software tells you that answer.

And those are not the same. And I am correct. And this sort of clouded what happened in the case. Professor Fries continues to stand by InterEye's work. InterEye's algorithms, or instruments, have been used in many states as well as governments in New Zealand, Hong Kong, Finland, and more. And he says the algorithms help improve how welfare systems function overall, which then leads to the benefit of the greatest number of people.

It's very complicated when you bring a poor person with a wheelchair who is horribly disabled and so forth, and someone saying, well, you're not giving them enough services. And it's really hard to not be very sympathetic. And judges and juries are very sympathetic to the individual and say, yeah, they should get more services. Yeah, they should.

Except everyone should get more services. And why is this person eligible for more services and not all those other people who are not getting service? In fact, you're going to get less services if you give this person services. So that's Brant Fries, professor emeritus of health management and policy at the University of Michigan and president of the research group Inter-I. Kevin DeLibon, what do you think? What's your response to what he's saying there?

Yeah, I mean, I guess there's a few things. The first thing I think for people who aren't in the weeds of this stuff to understand is that AI isn't a term that has a set meaning, right? Oftentimes, it's a hype marketing term that can encompass a lot of different technologies. And from the perspective of somebody whose life is being judged by it,

It doesn't necessarily matter whether it's the latest generation of AI that just came out yesterday or some advanced statistical modeling system that came out a dozen years ago or two dozen years ago. The purpose is the same. Every time these systems are introduced to decide the lives of low-income people, they result in benefits cuts.

every time. We don't have a single example where one of these systems was introduced and somehow people magically got more care or got better opportunities in life. Never. Now Mr. Fries, or Dr. Fries, pardon me, also said something else interesting. There was an error in the software but that wasn't the only problem. If the algorithm had worked perfectly as designed, the best case scenario was still this five and a half hours a day of care more or less for most people.

So the best case scenario for the way the algorithm worked was a huge cut that left people still lying in their own waste or getting bed sores or missing out on physical therapy appointments or doctor's appointments or going out into the world. Now, the fact is the way Arkansas had implemented that algorithm was flawed and made it even worse. Right. But the best case scenario was still horrific.

So was there any way that then this outcome could have been avoided, right? Like if how it had been implemented in Arkansas, if the goals were different, if the AI was told to seek a different sort of optimization, could it have actually helped people? Or was the AI fundamentally flawed, as you're saying?

No, no, it's fundamentally flawed because its purpose is to make benefit cuts. Now, one thing Dr. Friese highlights that I agree with is that these systems generally are underfunded, right? And lawmakers putting more resources and funding into social services generally, and healthcare, Medicaid, SNAP, public benefits, disability, the whole thing, unemployment, would help, right? It would give people a little bit more stability in life and, you know,

you know, get above these very meager, barely subsistence level benefits that currently exist. But even so, it is not strictly a policy problem because every time you see these tools implemented, AI-based tools, statistical modeling, whatever you want to call them, it leads to cuts. It leads to lost services. It leads to false accusations of fraud. It leads to horrific things for low-income people. Okay.

Well, let me just bring another person in here. Sorry, if you heard a little pause in my voice there, it's because I'm just still trying to think. Like, there seems to be a—well, I'll bring in the other guest here for—

For a second, Kevin, because there's almost like the question is almost more philosophical rather than technological. Right. And so to that point, let me bring in John Maynard. He's the principal industry consultant at SAS. It's one of the world's largest data analytics and AI companies. And he's also former state Medicaid program integrity director for the state of Ohio. And he served in that position from 2015 to 2018. John Maynard, welcome to On Point. Thank you.

Hello, Megha. So first of all, just answer Kevin DeLibon's assertion there that there isn't yet an implementation of AI when it comes to welfare benefits that isn't designed specifically to cut costs or cut benefits. That's just how it's been deployed out there at states and in federal governments around the world. Well, I couldn't answer that in terms of everything that's out there because I'm not sure what

everything that is out there. What I would say is when you, we say keep the human in human services and we look at this as an opportunity to use analytics and AI to help automate processes to basically augment the human, to bring information to their fingertips so that they can make a decision. When you allow an algorithm to make the decision on behalf of the person, I do think that they're correct.

They're not perfect. An AI model is not perfect as a person is not perfect. And so we think that the humans should make that decision.

Well, OK, so tell me at what point in the development of a instrument or system that a company like SAS would want to sell to, let's say, you know, a state welfare department. At what point are the goals of that program sort of built into the instrument? Is it do you go to the state first and say, what would you like to accomplish? Or do you do you create the instrument first and then say, here's what it can do?

It can be both. Sometimes we will create something on behalf of a customer that they would like to see get done. And then there are some things that we sort of sell out of the box. So we do have fraud solutions. We have different solutions to do like AI medical record review. But everything that we do is really to help sort of augment the human and

If a customer said, we want you to build a model and do intelligent decisioning and run it across this data set and make decisions on behalf of a person, we won't do it. Our founders, Dr. Goodnight, doesn't allow us to do that. I'm glad that he doesn't allow us to do that. Personally, I think that that's a good decision because when it,

When a model can make a mistake because it's not perfect, it can make that mistake in volume. And Kevin, I think, is making a really good point here. So I actually used to be an eligibility worker for cash assistance and food assistance and Medicaid and child care and things like that, workforce training. And what Kevin is saying is right, is that when you make these types of decisions, sort of as a blanket, there's a policy.

you know, there's this chance that you could, you know, you could hurt the people that you're intending to potentially try to help there. And I think for, you know, the welfare population, they don't recover from those kinds of mistakes very easily or very well because they're sort of on the edge. And that's why, you know, we sort of say keep the human in this. And I think you want an eligibility worker to make an eligibility decision. You want a nurse to make a health care decision. Kevin, go ahead. I'd love to hear your response.

Well, no, I'm excited to hear that a vendor the size of SAS wouldn't use

AI for decision-making purposes. And I think that highlights the danger of this. You know, I'm focused on AI-based decision-making, right? Where it's making decision about a vulnerable person's lives. And in those contexts, the danger is just too high. And you also have to understand the practical consequences or practical circumstances into which this stuff is implemented. A lot of states have been cut, right? There aren't enough state eligibility workers like Mr. Maynard was.

They are overworked. They don't necessarily have the resources. They've lost a lot of expertise over the years as positions open and people leave or, you know, they're eliminated cuts to services. And so what you have is really inadequate oversight of these systems in the government context. So in the state of Arkansas, for the algorithm we've discussed,

For example, nobody on the state staff, nobody, not a single person, could explain how the algorithm worked, either mechanically in terms of like what you put into it leads to how you get, what you get out of it, or statistically why the factors that matter are the ones that matter and why others don't. Not a single person on the state could do it. They didn't test it prior to deployment, which is how you get the errors that intensified the injustices that were already baked into the system.

and they didn't care to fix it when the issues were raised to them. And that's not unique to Arkansas. That's true of very, very many states. So I think there's a lot there. There's also something that Mr. Maynard referred to

that I'd like to explore a little bit, which is the notion that somehow these can go wrong at scale. And actually some states have weaponized it to make benefits intentionally harder to get. Yeah. Kevin, hang on here for just a second. I'm Meghna Chakrabarty. This is On Point.

So tell me more that you're saying there's deliberate weaponization of the scaling ability of AI. Yeah. So one recent example of this is with what's called the Medicaid unwinding. During the pandemic or the height of the pandemic, there were protections put in place so that states could not terminate people from Medicaid who became eligible. And the whole idea was there, hey, we have a pandemic going on. People need access to health care. We don't need to be cutting people off when there are vital health care needs. At the

In 2023, 2024, those protections ended and states started this process of reviewing eligibility of everybody. At that point, over 90 million people. You had more than 20 million people lose Medicaid coverage due to AI-based decision making in many states, most of which were for paperwork reasons. They weren't because folks were actually ineligible. They were because AI made decisions

incorrect inferences from information they had sent out more paperwork to people than needed to be happening, confused folks, and a lot of people lost coverage. Good. John Maynard, go ahead.

And then that's a policy decision about how you're going to implement that. So I think you can always run some model that's going to say these people are likely to stay, these people are likely to leave. But at the end of the day, you still need an eligibility worker to look at that on a case-by-case basis and make decisions about it.

It sounds like in some cases it was not the AI or the technology as much as it was a communication problem with the actual recipients of the benefit. And I've seen that sometimes happen because, and especially with COVID, they can be transient. They can be difficult to get a hold of. So you're sending notifications because that's what the law requires to an address where that person no longer lives. So they don't know that you're trying to contact them.

They don't know that you're trying to send them paperwork to fill out in order to maintain their benefits. So that's an age old problem.

before AI, you know, just of how things work in the system and how people kind of come and go. Well, John, let me ask you this. I definitely take your point that human oversight is critical and shouldn't be eliminated after a new AI tool is implemented in a welfare organization. But if you need continued human oversight, if there's the opportunity for, you know,

Let's put it that way, to be inserted into the tool at implementation, etc. What exactly is AI helping with? What is it making more efficient or better?

So for me, I wouldn't, you know, and we say don't use it to sort of make those kinds of eligibility decisions. But there's a lot of things that it can do to help make it easier for caseworkers and people who make those decisions to do better. So one of the things I always remember, one of my toughest days as a caseworker is I had someone who, a woman who had cancer.

and she wasn't eligible for Medicaid because she needed a disability determination to come from Social Security. As it turned out, the day after she passed away, I got a letter saying she was eligible, and her cancer would have been treatable. So the policy's been changed, and people like her can now get Medicaid faster. But

Things like we're doing today is, you know, we're working with an AI medical record review on disability. We put it in a few years ago and it's looking at 16 million records a night. All it's doing is it's organizing them, it's analyzing those, it's indexing them, handwriting them.

And it's then putting that together in a quick, easy way for decision makers. And so that's produced 400 hours, FTE hours of productivity. Well, John Maynard, hang on here for just a second. It's only because I have to take a quick break and Kevin DeLibon as well. We'll be back. This is On Point. On Point.

Support for AI coverage in On Point comes from MathWorks, creator of MATLAB and Simulink software for technical computing and model-based design. MathWorks, accelerating the pace of discovery in engineering and science. Learn more at mathworks.com.

and from Olin College of Engineering, committed to introducing students to the ethical implications of artificial intelligence in engineering through classes like AI and Society, olin.edu. Still getting around to that fix on your car? You got this. On eBay, you'll find millions of parts guaranteed to fit. Doesn't matter if it's a major engine repair or your first time swapping your windshield wipers.

Are your alterative colitis symptoms proving difficult to manage?

Tremphia, Gacelcomab, can help you manage the cycle of UC symptoms. At one year, many patients taking Tremphia achieved clinical remission, and some patients also achieved endoscopic remission. Individual results may vary. Tremphia is a prescription medicine used to treat adults with moderately to severely active ulcerative colitis. Serious allergic reactions and increased risk of infections may occur.

Before treatment, your doctor should check you for infections and TB. Tell your doctor if you have an infection, flu-like symptoms, or if you need a vaccine. Ask your doctor if Tramfiya can help you manage the cycle of UC symptoms. Call 1-800-526-7736 to learn more or visit tramfiyradio.com.

This is On Point. I'm Meghna Chakrabarty. And before we get back to our conversation about the use of artificial intelligence in welfare benefits and social services, I definitely want to turn your attention to a series that we're working on for next month about boys and education. So are you the parent or family member of a teenaged boy?

Are you a teacher or a mental health professional who works with teen boys? We want to hear your stories about teen boys and their mental health or yours if you're listening and you're a teen boy. What struggles are going on? What symptoms have you seen? What kind of help do boys need? And is it talked about enough? So get on your On Point VoxPop app. If you don't already have it, look for On Point VoxPop.com.

wherever you get your apps, and leave us a message about teenage boys and mental health. You can also leave us a voicemail at 617-353-0683. That's for a special series we're working on regarding boys and schools in America. Today we're talking again about welfare benefits or social services and the use of AI. Kevin DeLiban joins us. He's founder of Tectonic Justice. And John Maynard is with us as well. He's principal industry consultant at

SAS and former state Medicaid program integrity director for the state of Ohio. And by the way, I just wanted to say quickly, when I said SAS is one of the biggest out there, according to their 2023 corporate report, more than $3 billion in global annual sales and 90%

of Fortune 100 companies or their affiliates are SaaS customers. So this is a major company here. And John, you were telling us a little bit more about that example of the woman who had passed away before she got her benefits and why was that relevant?

Well, I think it's relevant because when you have a decisioning process that's very manual and human-based, like a disability determination, generally there's backlogs. But when you have someone like her who's waiting for that decision, and her life was on the line.

That needs to be done faster. That needs to be done more efficiently. But Gavin's right. One of the things that we're seeing now is the boomers are leaving government. They're leaving these programs. It's harder to recruit younger people into them. So those staffing shortages aren't going to go away. And that's why I really feel like social benefit agencies globally, they need AI to keep up with the demand.

And so I think that's and also to help fight fraud, fraud in the program and also fraud against beneficiaries. So we've seen, you know, criminal acts against applicants and beneficiaries as well. How much fraud is there on the path of beneficiaries in these programs? This is something that gets said all the time. But I mean, have AI systems like actually found masses of fraud?

John? Well, you can find fraud. So generally, the general rule is 10 to 15 percent is going to be fraud, waste and abuse. And fraud is probably two to three percent of that. There's actually three to four times the number of errors than there is actual fraud.

But you mentioned, but you said fraud. Here's why I'm asking this, okay? Because, you know, I was looking at your bio, John, and I have become, lately, I've become a believer in the importance of org charts. And here's why. No, here's why. Seriously, because, I mean, your experience is very much valued in this conversation. And I was noticing that, at least at one point in time, your position was under SAS's global fraud practice. Is that correct? Yeah.

Right. Yes, it was. We've just been rebranded. I still work on the fraud stuff all the time. It's something I really like to do. Don't tell anybody, but I have a real propensity for figuring out how to steal money from a system. So that's kind of my white hat for that. Don't tell anybody except for the millions of people who just heard this. The reason why I'm asking is this.

is that, and I'm just going to use you as an example, but if a company is creating an AI tool that it's trying to sell to, you know, a state social services organization, but that company falls under the org chart of global fraud practice, again, to use you as an example, that is

Success in reducing fraud is the metric that that group is going to be measured by within their corporation. So they're going to be building tools that are really good at that. They're going to cut costs by finding, quote unquote, finding fraud. That is not a tool that is designed to help make a social service department more efficient and provide better services. It's just not baked into the goal of the team that's designing the tool, John.

Yeah. And so what I would say to that is a fraud solution is a fraud solution. That's really what it's designed to do. But we also do a lot of other things on top of that. And there's a lot of SMEs, even though we're because we're in the fraud risk and compliance piece. So we're just more than fraud. And so I worked on things like total cost of care, episodes of care, value based payments.

I'm very familiar with home health. I'm familiar with a lot of those things that we're talking about. And because I come through the system and from the system, I don't code. I'm not a data scientist. And so what I do is I talk to people inside our company about here's the struggles that I see in these places. We need a solution to help them do this. We need a solution to help them do that. Things like a medical record review. We also created a model called

That is helping with SNAP. And it's basically just looking for high risk error cases. And, you know, it's something that, you know, the state can have to help get their error rates down. And what I would say is you're losing hundreds of millions of dollars in these overpayments. But what you heard Kevin say is somebody else needs more health care benefits.

guess where the funding for those health care benefits they need can be coming from. And it's these errors and the waste that we have in other programs or even the same program. You know, there's a lot of things from a policy standpoint you might want to do that you don't have the money for. But if you're wasting money in other areas that's not really providing any value or any good to anybody, then you need to find that money and reallocate it to where it needs to go. Kevin, are you crawling the walls yet?

I might need to start.

But no, I mean, I guess the first point is like, look, instead of turning necessarily to automated solutions, particularly the more close they get to a decision that affects a vulnerable member of the public, we need to just think about investing in government capacity. Right. And that's a universal thing, both at the state levels and at the federal level. Social Security, for example, is has been running a massive staffing deficit from where they used to be and where they need to be.

for I think it's two decades now. So the first thing is not to go to like FastSeal or kind of nice, shiny, middle dangling solutions like AI, but actually invest in government capacity, human capacity to do these decisions. The other thing is I want to really focus on the dangers of this focus on fraud. So statistically in the SNAP program, which many people know as food stamps,

of every hundred dollars of SNAP benefits paid out to recipients, only 10 cents

are ever improperly paid due to recipients like intentional fraud. So there isn't really fraud on the part of beneficiaries. But this whole notion of fraud, waste and abuse ends up justifying right now the destruction of the federal government, what we see happening at Doge, not an objective thing, but using fraud as a cudgel to identify anything that the president decides he's opposed to and then destroy it. And we hear it in state benefit programs.

as ways that end up going after beneficiaries, right? Going after people getting the programs. And these fraud detection systems don't...

don't work when they come to individuals. The example in Michigan that you stated earlier, Meghna, is actually an understatement. Over 40,000 people were accused of fraud. When an auditor general, the state's auditor general, went in and did a review, 93% of those were false positives. 93%. Or during the pandemic era of expanded unemployment benefits,

there were routinely flagging people for possible identity fraud at huge scales, right? So, and at the end of the day,

It was established that most of those were false flags. Right. And so what you're doing with these fraud systems is if you're not destroying government outright, you're making benefits harder to get, putting up barriers that frustrate access. And you're doing it in a way that ensnares a whole bunch of people who are eligible. Right. So that's why this isn't you know what I mean? Like.

It's a dangerous concept because it justifies doing things that ultimately are harmful and frustrate people's ability to get benefits during really desperate times of life. But what you're saying here then, let me put it this way, because originally I had asked John, like, hey, well, if you're just building tools underneath the aegis of the group in the company that's supposed to pump out fraud detection tools, you're going to build fraud detection tools. But what you're saying, Kevin, is...

is actually, I think, more important. And that is the people who are deciding in government, obviously right now at the federal level, but also at the state level, you're saying that there are policy decisions that are being made specifically because they believe there's a lot of fraud going on. And so therefore, of course, they're going to use these particular AI tools whose programmatic strength is fraud detection, Kevin, right? So the problem is the people, right?

No, no, no, no. I want to push back there because I don't think in many cases it's a sincere belief that there is fraud. Right. I think fraud is a convenient cudgel to allow people who are ideologically opposed in many cases to any sort of safety net benefits or systems to go in and attack those systems. I want to be really clear about this. A lot of this is not good faith. Agreed. But then the problem isn't the AI. The problem is the people.

Well, it's both. It's both, right? The problem is the people in the sense they're out to destroy things or make benefits are harder to get. But AI is the perfect weapon to do that, right? For some of the reasons that John pointed out. It's scalable. You put that in and suddenly you're affecting the benefits of everyone in a way that you couldn't if you had a caseworker have to manually review or make decisions about

about whether or not somebody intended to deceive the government. And that's another key point here.

is there is a definition to fraud, right? It is intending to deceive the government in order to receive benefits or in the case of government contractors, contracts that you wouldn't otherwise. Most of what's being attacked by this isn't intentional fraud. People are getting caught up in it, but it's not demonstrating any intent. And the pernicious danger, and to your question of is this a technology problem, here's where it is.

Once somebody is flagged by AI or any sort of automated process as being fraudulent, that ends up serving as evidence of the intent. That ends up being, in many cases, the judge, jury, and executioner is the automated decision because it has this veneer of being objective. And it isn't. And human reviewers are oftentimes not able to...

meaningfully and adeptly challenge it or review that, you get tagged as fraudulent by the algorithm, you might as well lose your benefits. It's science sealed delivery. I think that point is very well taken about the complexity of the technology makes it hard to appeal or to push back or to get a human reviewer unless the system is built to

The office, let's say the organization is built to say, yes, a human reviewer can automatically just reject the AI's conclusion. I totally take that point. But, you know, we've only got about four-ish, less than that, minutes left, gentlemen. Look, AI is here to stay. There is no doubt about it. With every second of every day, more and more, it's sort of worming its way more and more to every aspect of our lives.

And I think to potentially huge positive benefit. But we have to keep being, you know, we have to keep scrutinizing it as well. So with that in mind, John, you know, is there a better way? Like what could companies like SAS do to create tools that actually don't lead to the high number of examples that I started off with of AI actually hurting people rather than helping them?

Well, I think one is the decision and question we've talked about, but, you know, SAS has a data ethics practice. And so we're focusing on human centric design. You know, we're focused on inclusivity, accountability, transparency, understanding what a model's doing, why it's doing it, robustness, which means once it's placed in operation, you're constantly monitoring it to make sure it's doing what you intended and it's not having unintended consequences. And then also working on privacy and security. I think what Kevin said there is, you know,

The data itself is not an indication of fraud. And as a human being, I've fallen in that trap through the years where I got something and I said, oh, this must be fraud, only to investigate it and find out, no, it wasn't. So you always have to be careful. The data itself and what the model's telling you doesn't mean it's fraud. It means it's an unusual anomaly. You need to look at it. You need to prioritize that and give it a review. Well, but then don't sell algorithms that are designed as decision-making algorithms.

software, right? Because ultimately, it's not just analyzing the data, it's pumping out a decision on benefits. Not the ones that we do, and I wouldn't recommend that. And I say that because as a caseworker, as an auditor, as an investigator, every situation is a one-on-one decision. There is something about every one of those patients. There's something about every one of those families which would make a decision go one way or the other, and an AI model can't account for all of

that. And that's why you want a human to make that decision. And I know that there's human bias, but there can also be bias inside those models because human beings make them and human beings use them. So let the human make the decision. Let AI augment the processes around that to help them make better decisions faster.

Okay. So, Kevin, you get the last word today on how we make it better because I hear what John's saying, but we started out with you describing to us what happened in Arkansas and humans making the decision, you know, ultimately on how many hours of assistance these folks were getting. I don't recall you saying that that was how the system was being used. So how can we make it better?

Yeah, first you have to change the broken incentive structures that are here and the broken accountability structures. There's no political accountability usually for state government officials or elected officials when things go wrong for poor people. There's very little market accountability in

most of these situations because you have only a select number of very big vendors. And so you see vendors who have demonstrated histories of failure still get big contracts because states don't feel they can go anywhere else. And you have limited legal exposure because of some things we didn't get a chance to get into today. But what you need is you can't rely on kind of the goodwill of people who design these tools

You have to legislate around them. You have to ban certain decision making uses. You have to require really heavy duty vetting and deployment and monitoring. You have to have a mechanism for meaningful oversight by the people who are affected by the decisions that are to be made. And you have to make sure that there are disincentives, legal disincentives

to doing anything that's going to be massively harmful to really vulnerable people. And the only way to do that is through significant legislative and regulatory interventions. Well, Kevin DeLivon, founder of Tectonic Justice, an organization that helps low-income people whose benefits have been impacted by AI. He worked at Legal Aid in Arkansas for 12 years and was the former director of advocacy there. Kevin, thank you so much for joining us today. Thank you.

Thank you so much for having me, Meghna. And John Maynard, principal industry consultant at SAS. It's one of the world's largest data analytics and AI companies and also former state Medicaid program integrity director in the state of Ohio. John, thank you so much for joining us. Thank you for having me today. I'm Meghna Chakrabarty. This is On Point.

Support for this podcast comes from Is Business Broken, a podcast from BU Questrom School of Business. A recent episode asks, how do employees feel about executive compensation? And how can companies balance rewarding top leaders while keeping employees engaged and valued? I think if you see long-term success of a company and very attractive awards for executives and others aren't being brought along on that journey,

That, to me, is a real concern because I think we should live in an economy where you can make as much money as you want and work as hard as you want. But at the same time, there should be a path for others to also benefit. And if I were going to change the overall structure of compensation in American companies—

I would look for a way to get more ownership in the hands of all employees. And right now, a lot of investors don't like the dilution of giving too many shares to employees. And some of the accounting rules make that a little difficult from the profit and loss statements. But finding a way to make everyone in the company an owner.

Well, in addition to paying them fairly, but use sort of the Lincoln electric model where they have a very strong profit sharing. People can make $100,000 a year on profit sharing, but you can do that also through equity. You've seen what's happened to the stock market over the last decade. Executives benefit, employees don't.

Find the full episode by searching for Is Business Broken wherever you get your podcasts and learn more about the Mayrothra Institute for Business, Markets and Society at ibms.bu.edu.