AI in the US State and Federal Governments with the hosts of the AI Today Podcast
35:36 Share

Hello and welcome to Sky News Today's Let's Talk AI podcast, where you can hear from AI researchers about what's going on with AI. I'm your host, Andrey Kurenkov. And I'm your co-host, Dr. Sharon Zhou. Today we have a special crossover episode with the AI Today podcast, the hosts of which, Kathleen Walsh and Ron Schmelzer, discuss pressing topics around AI, interview guests and experts on the subject, and cut through the hype around AI.

So let me welcome the host of that podcast, Kathleen and Ron. Hi, and thanks for having us today. Yeah, thank you very much. Thrilled to be on the podcast and share our mutual audiences and interests. Yeah, and so are we. One of our early crossovers and, you know, good to diversify from the news we usually do.

So just to set up what we want to do, the AI Today podcast interviews a large variety of people in the industry and in various sectors. And so we picked just one subset of those specific to AI and government. And yeah, so we'll just chat with Kathleen and Ron about some of the interesting things they've learned from talking to government officials at various levels of the government.

Yeah, well, yeah, thanks for having us. Maybe you think we could start a little bit about by casting a little bit of a picture of the sorts of folks that we tend to spend time with on the AI Today podcast. So maybe Kathleen, maybe give the audience a little bit of an intro of like what we have been doing for the last four years and over 200 episodes. What have we been doing with all of our time?

Right. So as Ron mentioned, we have been hosting the AI Today podcast for four years now and over 200 episodes. So we have really dug deep into many areas of artificial intelligence. And we've been able to interview many thought leaders in both the public and private sector as well. So we interview people from, you know, large fortune 1000 corporations, but we also have a focus on government as well. And that's just not the US government. It's

International, federal, state, and local as well, which really helps give a very unique and balanced perspective on where we are with the adoption of artificial intelligence. And also the education around artificial intelligence, you know, what it is, what it's not, the range of understanding that people have about that.

So we have been fortunate enough to interview, as I mentioned, many people. And we can talk about that a little bit more, how things vary from that international level to a federal level, also state and local, budgets and resources, the knowledge that people have, skill sets, skill gaps, all of that.

Yeah. And just to add to that, I mean, part of the reason why we started the AI Today podcast, because this is not our business as it were. We have an analyst firm. Kathleen and I run an analyst firm called Cognolitica. And we spend all of our time trying to understand the market. And that market is all about basically who's buying what and who's implementing what.

And if you spend your time listening to the technology vendors, you know, we know what it's like. They like to spin a very positive picture. Everything is awesome. It's like living in Lego world all the time. But the thing is, is that you really want to know, well, what are people actually buying and implementing? So you have to kind of not listen to them so much and spend your time with these customers and say, okay,

are you really doing these AI things? Are you doing machine learning? Are you really able to adopt these things? And that's where we started AI today back in 2017 when sort of the market was starting to get really hot.

And we kind of want to do a bit of a sanity check or reality check, really. And it's really interesting because what you'll see is obviously there's not always a smooth connect between what, say, the technology vendors would like you to think about how much the AI space is progressing and, say, what both the companies, the enterprises side, as well as governments are doing when they're trying to put AI into practice.

Yeah, I think we can relate to that idea, right, Sharon? Yes. As researchers, part of why we started our podcast is we can be a little bit more informed of what AI is and then when news articles come about, give our takes as to what we think maybe is a little questionable or might make more sense and

Yeah, it sounds really interesting because as researchers, we don't often hear from these industry insiders and government people. So we'll be very interested to hear about some of the things you found out from these interviews.

There's a lot of hype when AI is deployed, and we would like to dispel some of that hype. That being said, there's a lot of hype around research around AI as well. But given what we know works and will maybe work realistically, we're trying to dispel that hype and also trying to understand what are people actually trying to do with AI and

Yeah, make sure that these headlines aren't just clickbait and can actually be somewhat informative or not. And so just diving into some of the interviews you've had, maybe we could start with the challenges in federal AI adoption interview with Justin Marcico.

Yeah. Well, that's a really good one. I think it's a good place to start because the interview that we had, I think it was a few months ago with Justin Marsico, who is the CDO, Chief Data Officer at the Bureau of the Fiscal Service, is interesting because

The Bureau of the Fiscal Service is not one of the government agencies that most people are even aware of. Yeah, I don't think I am. But it controls so much that it's like if you, if anybody, any of your listeners have gotten a check recently from the federal government, that's a payment, let's say an advance payment for the, for like a child tax credit, you'll actually see it's

signed by the Bureau of the Fiscal Service. The Bureau of the Fiscal Service manages basically the inflows and the outflows for the government. And one of the sites that they control is called usaspending.gov, which is a fascinating site. Because if you go to that site, you can see pretty much everything that the government is spending on.

From the highest levels to the lowest levels. If somebody you know got a PPP loan, for example, on these recent loans for the Patriot Protection, you could see that. You could see exactly how much they got and everything. And I think the thing is that it all really cues about data.

And the vast amounts of data that certain government agencies are handling, you know, in some cases rivaling some of the biggest social networks out there, just the amount and quantity of data and how vitally important that data is.

And so these government agencies are looking to AI and machine learning in particular to extract more value from that data and also tackle all these things, right, around fraud and efficiency and optimization and automation. So, you know, one of the things we talk about, Kathleen, and we talk about is a lot of these challenges around data and trying to make that work for AI and machine learning projects.

Exactly, because a lot of these government agencies, you know, they need to deal with some of the challenges. And we talked about this with Justin, about challenges with data governance, data security, data ownership as well, and then other related areas. Data ownership is a really hot topic right now because the government, the large majority of what they're doing is going to be buy versus build.

So when you are not building this in-house and you need to give your data to someone to build a machine learning model, for example, you need to really ask questions of those vendors that are working on this. Who owns the data? How is it going to be used?

And so we push heavily on methodology, data-specific and AI-specific methodology to make sure that the people who are working on these projects have a methodology in place and that they are pushing on the vendors to use a methodology as well. At Cognolitica, we always promote the best practices, CPM AI, which is Cognitive Project Management for AI methodologies.

And we're educating people saying, you know, if you're not building it, ask the vendors what methodology they're using. And if they're not using one, make sure they adopt it. Now, if you are building it in-house, we encourage you to have a methodology as well. As Ron had alluded to earlier, there's more companies than we'd like to admit that are doing things ad hoc, which is not a good use of anybody's time, resources, resources.

you know, and how to have a successful project. But we push, you know, especially around all these data issues, have a data, and if you're doing AI and machine learning, a data and AI-centric methodology.

Could you give a quick description of what this methodology is that you promote? And also, in general, what does an AI methodology really touch on across the whole pipeline? Yeah, it's a really good question. And I think it starts from actually a methodology that's been around for several decades. Your listeners might be familiar with CRISP-DM, which is the cross-industry standard process for data mining. And it really originated with data mining projects.

And the way that we like to explain it is like, you know, go back in time, try to remember kind of what the technology landscape was in the late 1990s. That is, if folks are old enough to remember that, I don't know the average age of the audiences. But back in the late 1990s, when you had banks and financial services companies and government agencies that were trying to extract value from data, it was not that easy to

to query large volumes of data, if you had access to large volumes of data, and generate these reports or analytics that needed to be generated very quickly because every iteration just took a lot of time. And a bunch of large companies and agencies, and it was IBM and NEC and I think SPSS and one or two others, came together to basically say, "Okay, guys, you can't just be doing things randomly. Create an analysis.

throw it at the database, wait, in their case, sometimes a day or two for the response, and then come back and then realize that you asked the wrong question or you had the wrong model or the data was bad, whatever. Why don't we do things in a logical way here? Start with the problem you're trying to solve. That's phase one of CRISP-DM. And then

Then figure out what data is needed to address that problem. That's phase two. And then phase three is prepare that data for the query, then build the model in phase four, then evaluate your model in phase five, and then operationalize, get the model out in production phase six.

It sounds very, very logical, doesn't it? But the vast majority of folks who are working in data warehousing and data mining just went right to phase four. Let's build a model because that's the fun part, right? Nobody likes doing data prep. I bet you everybody here in this phone call is like, oh, data cleansing, data prep is the bane of my existence. But without good data prep, all of it fails, right? So that was 20 years ago, right? Fast forward now and

you look at all these AI projects that are spending millions of dollars and they're failing, right? Gartner said something like 78% of AI projects are failing. Andrew Ng was just out there saying that these two to $5 million projects that they're doing and landing are failing because of data quality. And we're saying to ourselves, how the heck did you even get to two to $5 million and spend if you didn't even ask yourself what data you needed and what quality it was. So guess what? We got to go back to crisp DM and we got to use that methodology.

The challenge with CrispDM is that they never made a version two of it. It started with version one and it kind of stopped and it didn't address some of the AI specific needs, more specifically in the machine learning model specific needs. As you know, if you're building a machine learning model, you do need to

Do things like figure out what you need in your training data set. Do data preparation that's more AI specific. It's not just data cleansing, which is important, but you have to do things like feature augmentation. You may have to do some data augmentation. You may have to do some massaging of the data set, data transformation that's very specific to building particular kinds of models, right?

And then, of course, the actual iteration of building the model and model operationalization is different because we're using models for inference. We're not building a report, which is basically what CrispDM is, like generating a static report. We're trying to use these models in production environments.

So what we did and others did was we basically just iterated on CrispDM and generated something called CPM AI, which uses the same six phases of CrispDM, business understanding, data understanding, data prep, model development, model evaluation, and model operationalization, but it provides more specifics for the questions you need to answer first.

when you're building machine learning projects, something called the AI go, no go, the seven patterns of AI, you need to determine what pattern you're using and various other things around model evaluation. And all that's very, very well documented. You can go to the Cognitica site, you can read all about it. You know, what we're doing is, was we definitely spent some time doing education, do some training and certification on it.

But, you know, if you're not using that methodology, you got to use something. Because these projects are failing and they're mainly failing because people are just doing things, either they're skipping steps or they're doing them in the wrong order. Yeah, that's interesting. It sounds like in modern terminology, you're kind of adopting some of the basics or some of the, you know, foundational ideas from data science practitioners where, you know, you more so do some insights or some reports, you know,

And then adopting it to a machine learning setting where there's some other tweaks. And yeah, methodology is interesting. I think many PhD students like myself and Sharon previously had to sort of self-teach methodology. You make some mistakes and then you figure out, well, I need to understand my data. I need to...

you know, set up for learning and do some baselines and so on. So interesting to see how that gets transferred to training and industry. Yeah.

And so bringing this back to like the, this, you know, federal government and maybe even state governments. So you mentioned that, okay, so bringing this methodology there would be fantastic since they're kind of going at it ad hoc. Do you see areas within the government where they have adopted this methodology or similar or some kind of methodology and it's been successful or it's going well?

So, you know, we have put people through the methodology, different various government agencies. So the IRS folks from there have gone through this. Some folks from U.S. Postal Service, Department of Energy, in particular, the EIA. And then we've had, you know, a few folks that we've just educated on basic, you know, artificial intelligence knowledge and overview of methodology, but they haven't been certified yet.

I know that the Department of Defense is also doing a lot around artificial intelligence. They're a little bit more closed-lipped about everything that they're doing. So, you know, can't share too much of that. But in general...

The vast majority of, I think, anybody who's working on projects is not adopting this data-centric and AI-specific methodologies. So I think at a federal level, they have the budget right.

to invest in this and invest in projects. But they're still doing a lot of that buying as well. And they need to be pushing hard on vendors to be, you know, getting methodology certified and also to show and prove to them that they are certified and following a method so that they can, you know, have it documented and show it to them. At the state and local level, for the vast majority of them, they are going to be buying this. They don't have the talent in-house

to be building this. And a lot of times they don't have the runway either, even the federal government to be building machine learning models. So we always say, if you're not going to be building it yourself, make sure that the vendors are showing you that they're using a proven methodology. Because vendors have a lot of spin, a lot of marketing spin. It can be a lot of smoke and mirrors.

And they can act like they have more than they do. And when you actually look behind the curtain, let them prove to you that they can do what they say and that they have this documented, that you know where the data is coming from, that you know what algorithm they selected, that you know how they cleaned and prepped the data and how they're retraining your models.

Yeah, sounds a lot like reading a research paper. So real quick, we do want to get to the state level a bit more because you've had a couple interviews, but I wonder how this AI center of excellence, which you've also had an episode on, how that relates to these ideas you've had of

kind of making sure that there's a methodology in place for good adoption. Yeah, it's really interesting that government does have

a number of groups within federal agencies that serves as so-called centers of excellence, right? And particularly within the GSA, which is the General Services Administration, many folks might think, if you know of the GSA, you might think of them as sort of like the landlord for the government because they basically own all the buildings for the government and they kind of rent them back out to the government. It's kind of interesting, the model there, they own a lot of assets. But one of the big things that they do is they do a lot of handling contracts on behalf

with the government. And because they handle contracts, they can see when government agencies are kind of doing things the stupid way, I guess. And they can say, wait a second, wait a second. Why are you doing that? Why are you buying that? Why are you spending money on that? Didn't we solve this problem some other way, better way? At least that's the theory, right? And so the Center of Excellence, they have a Center of Excellence for Artificial Intelligence. Actually, they have a few Centers of Excellence. One of them is focused on AI and

We did an interview with Neil Chowdhury, who is at the Centers of Excellence, actually also with Krista Kennard and a few others. Krista's no longer there. She's now at the Department of Labor. And they basically shared with us that they spend a lot of their time actually on technology.

data-centric issues, which is what comes up most of the time, data availability and quality. And of course, a lot of that comes down to doing things in the right order. And they are definitely, there's actually a current initiative right now to establish the best practice methodology within the government. So we're hopeful to kind of see them progressing on that. But that's a really interesting interview and one of our AI today. I don't know which, remember which episode number it is, but it's worth listening to.

Yeah, we can link some of these in the episode description so our listeners can go and check those out. And I think I will also go back and listen to that one because good to know that there is an AI center of excellence. Someone trying to keep things going well.

I'm curious, what are some of the use cases of AI? If you could give some examples at this federal level or in the AI Center of Excellence that they handle. Since we've been talking a lot about methodology, they need to get the methodology right. They especially need to get it right if the impact is going to be huge, right? And so I'm just curious, what does that impact look like? What are some of these use cases?

Yeah, that's a great question. So, you know, they are specifically at the GSA Center of Excellence. They are working with other agencies as well to help them on their journey. A lot of folks in the federal government, you know, are on this AI journey, as they like to call it, and they start off with robotic process automation, RPA.

which is not AI at all. We say that it's automation and it's incredibly useful and can do a lot, but it's not intelligence. And so within our seven patterns of AI, we have an autonomous pattern. And the goal of that pattern is really to remove the humans from the loop.

That is not to be confused with automation. And so we always harp on that because I think sometimes people can just get confused. And back a few years ago, there was a lot of spin in the market with what is RPA and RPA is AI and it's not. So I think that we have thankfully gotten over that as an industry. So, you know, so we're seeing some areas there. We're also seeing a lot of natural language processing.

So we had an interview with Courtney Winship, who's the CDO at USCIS.

And that's U.S. Citizenship and Immigration Services. And they have a chatbot named Emma. And she's able to communicate with folks in both Spanish and English and answer a number of questions for them. We've also, you know, seen different use cases at USPTO, the Patent and Trademark Office, where they're using machine learning to help with patent search.

Yeah, and we've also seen some really interesting examples, a lot of NLP actually, whether they're using to do document analysis because the government's drowning in documents. So they'll do a lot of document search, document analysis. We actually did a really cool case study, Alex Measure at the Bureau of Labor Statistics, another one of those agencies you may or may not know of. They're the ones that compile the economic numbers. They also compile workplace injuries and

And they used to do all that in the form of surveys, where people would like fill out a survey, actually like filling out a survey. And someone had to go in and basically code them. We're basically like, oh, this injury was of this type and blah, blah, blah. It sounds like you can feel the pain on that one. So –

Actually, quite a few years ago, Alex was like, hey, I think we can do this. We could do classification with natural language processing with machine learning. And they started with like PyTorch and ended up with something, a TensorFlow thing. And it's a really great little thing. One person, one economist built it himself and deployed it. And it's basically being used all the time. That one example now we're seeing at the VA conference.

the veterans administration, they're using it there. We're so a lot of that stuff, we're starting to see more recognition, which is, so we have these seven patterns. The conversational pattern is one of those patterns, mostly NLP. A separate pattern is the recognition pattern, which is, uh, basically handling unstructured data, images and audio and that sort of stuff. We're starting to see a lot more of that, a lot of satellite data, uh,

which that's one of the lot of things we're seeing at the Department of Energy has to do with with with the imagery and image data. Of course, NOAA, the Weather Service, we're seeing it, of course, at NASA. We had some really great we had a great

interview with Chris Mattman, who, and it was the perfectly timed. That was the day that the, that the perseverance I think was landing on Mars. We had him do our AI and government event. So we had like a live, we actually do these live events to a virtual, of course, called AI and government. And he was doing it while that was live. And it was just fantastic for him to explain to us all the machine learning that's happening on these rovers, which are, you know, light years, not just say not light years,

hundreds of thousands of miles away, not quite a layer. So, uh, just, just really far. So it's lots of those kinds of examples and lots of like really kind of, I would like to say mundane uses of, of AI and machine learning that, you know, nobody could really appreciate unless you probably feel that particular pain. Yeah, that makes sense. And, uh,

Yeah, I think we discuss some of these things. Sometimes we see some news articles about using satellite data for weather prediction, but we do discuss a lot more on sort of a cutting edge. And it sounds like maybe there's a lot of interesting stuff that is adopting existing techniques isn't necessarily, you know,

something totally new, but in the organization of the government is making a large impact. Yeah, so it's really interesting to hear about these various things going on at the federal level. And next, I also saw you've had several interviews with officials at the state level, in particular,

the state that Sharon lives in currently and that I live in usually with Stanford, California. You had an interview with Joy Bunaguro, the CDO of California about data and AI at California. So I'm curious how that's different from the federal level and yeah, what is there maybe specifically about California?

Well, welcome to the nation state that is California because California is just gigantic. I mean, just in terms of not just geography, but the number of citizens and the number of systems. So one of the biggest things that differentiates a state government from a federal government is

is that obviously much more constrained by budget. Most states have to basically keep their budgets as balanced. They can't be out there spending a lot more than they're earning. So that's one big thing. The other thing is that the state governments usually only have so much control over what happens in the local jurisdictions, whether at the county and city level. And a lot of the data –

for better or for worse, is usually locked up in the state or local data, whether it's health information or whether it's utilities or whether it's anything about the workforce and economics. So the interview with Joy Bonagro was really very interesting because what they're trying to do, the biggest impact, obviously, over the last year of machine learning and AI and data systems in general has, of course, been health information.

You know, of course, the pandemic, anything related to COVID, COVID data, COVID reporting predictions. But it's not even just the direct impact of like, you know, knowing, you know, who how you've all been vaccinated, which is which is a good question to understand the answer to that one.

And just all that sort of stuff. But also it's all the side impacts, which are which businesses have been impacted, business shutdowns, unemployment, of course, a huge amount of deal about unemployment and unemployment insurance, state unemployment.

And of course, dealing with things like the work from home environment, especially in governments where you might have, you know, in a federal agency, you might have a government agency with like several thousand workers, right? Or tens of thousands of workers.

but you might have a city or a state government agency with only dozens of workers. And if like, let's say the seven people that are working in say building inspection, you know, working from home, you can imagine sort of the backlog, right? So there was a lot of discussion around things like trying to basically introduce more self-service. So we're starting to see a lot of like

chatbots, apps with self-service, automation systems, of course. So the needs at the state level are really broad and they can use a lot of help, right? I mean, Kathleen, if you want to chime in on that.

Yeah. You know, as Ron mentioned, I mean, the states just are more constrained in the resources that they have available and they're smaller. So, you know, they can't they really need to focus on their budget. They need to focus on on what they have at hand. And they don't always have the luxury of, you know, those longer term projects that maybe the federal government can can take on.

Yeah, we've had some other interesting interviews. We interviewed Carlos Rivero, who's the chief data officer of the state of Virginia. And they were talking about very similar things and also in North Dakota, the Dorman Brazell, who's the CDO there. And North Dakota is a really interesting case because the governor used to be the CEO of a software company. He was the CEO of Great Plains, I think it was. I've heard of this, yes. Very counterintuitive. I know, and you're like,

Wait a second here. This definitely understands the value. And North Dakota is probably the entire state of North Dakota is probably the population of maybe not even the largest city of California. I forget the comparison there. But yeah.

But the state's large, too, and it's geographically diverse. And so they're trying to really also step up and handle it. So it's kind of interesting. You can go from the largest state to not the largest state. And many of the problems are still there. One of the things that they bring up to us is they are open to working with

small companies, students, you know, colleges to basically, you know, tackle this whole idea of addressing the data and taking the data that they have and trying to extract more value. Kathleen, you wanted to chime in a little bit about the CDO panel?

Yeah. So, you know, we were also fortunate enough to, as Ron mentioned, we run a number of communities as well. And one of them is our AI in government community. So you can go to AI in government dot com to check that out. And we were fortunate enough to have a state level CDO panels.

So the moderator was the former chief data officer of the state of Connecticut. And then the panelists included Joy and Dorman and then Adita, who was with Arkansas at the time. So it was able to bring a really unique perspective from three to four because Tyler was

who was the moderator, used to be the chief data officer of Connecticut. So he was able to bring in perspectives there as well. So, you know, very geographically diverse from coast to coast and different, you know, population sizes, different budgets.

And it was really interesting to see, you know, some of their challenges are the same. Some of their data challenges are the same, but then some of them are a little bit more unique to maybe where they're situated and what they need to focus on.

That's really interesting. Yeah, I think it's easy to be cynical with respect to government and its ability to leverage technology. But I think hearing from these specific people, the chief data officers and some of these people at the federal level,

is maybe a good way to be informed and to actually understand that there are these efforts and while there may be challenges, our elected officials and other officials are working on it and are being diligent as much as they can.

Exactly. It's interesting, too, because, you know, we have 50 states, but there's only 27 chief data officers. So the states that we interviewed, it's great that they have thought that they need a chief data officer. So everybody's on their own journey in those states that haven't hired one yet yet.

It's interesting maybe to find out why they haven't. Are they just having a hard time filling that position? Do they not see a need for it? Is that in their pipeline? Is it in their budget? What will the chief data officer do to kind of help drive them forward, especially as we know data is powering a lot these days? Data is king in AI. That's true.

Yeah, it's actually kind of interesting now because it's sort of like everybody has become more data aware, I guess. I don't want to say data literate because I might be overstating the case here. But people are much more data aware. I'd be like, you know, we're in Maryland and you can go to coronavirus.maryland.gov. And what you're looking at when you go there is you're looking at these Power BI dashboards. And it's like,

In what sort of alternate reality would we be where it's like the typical grandmother or the average person, you know, of any kind would be looking at Power BI dashboards on a daily basis to know whether or not their kids are going to go back to school in, you know, in person in August, right? And so it's like...

we never would have been so data driven. It's like, okay, you know, the vaccination rate needs to go to this percent. The hospitalizations rate needs to go to this percent. You know, the seven day average, it's like, sometimes you have to explain to people really what people who don't really have a good foundation in statistics or probability be like,

no, no, people are arguing about things like death rates and like, well, you have to understand the data. It's kind of funny now that we're not talking about applications, we're not talking about sort of systems, we're really focusing on the data. And actually, we like that because the data sort of speaks, should speak to help us make much better decisions. And so,

We're hoping that the other 23 states that haven't gotten their act together on CDOs get that together quickly. Sounds like we should, yeah. Okay, so that was very interesting hearing about AI at the federal level and at the state level. You do have much more actually about government. You have some interviews at the city level, some international. And although it would be fascinating to keep talking to you, we could probably talk for hours.

Instead, maybe we should point out this is a crossover so our listeners can go to the AI Today podcast and check out all of these interviews themselves. It's really interesting. There's a lot more in terms of also industry applications and a lot more.

And with that, thank you so much for listening to this episode of Skynet Today's Let's Talk AI podcast. You can find articles that we discussed today as well as similar ones on our website. Please subscribe to our weekly newsletter there as well.

And yeah, just to close it out, thanks again, Kathleen and Ron for joining us for this episode. Yeah. Thank you so much for having us. We're really excited to get to talk to you guys and your listeners today. Yeah. Thank you guys so much. You're fantastic interviews. We are definitely going to recommend to all of our audience to listen to all of your episodes. Of course, not just the one that we're on. So thank you very much so much for sharing this, this fantastic experience with us.

Yes, and our listeners, of course, go check out AI Today as well and subscribe to both our podcasts and, as usual, give us reviews and ratings to help other listeners discover us.