Insight 2: Safer by Design: Nurses at the Helm of AI in Healthcare
04:50 Share

Welcome to See You Now Insights, where each week we go back to our library to bring you a piece of wisdom from the over 100 interviews with leading nurses from around the world. I'm Rebecca McEnroy, executive producer for See You Now. This month, we're focusing on safety from start to finish and what it takes to truly ensure safety in every aspect of healthcare. As artificial intelligence becomes more integrated into care delivery, keeping patients safe is

means involving nurses at every stage of AI development. From Episode 102, AI in Play: Smarter Care,

Chief Nursing Officer Amy McCarthy and Product Director Scott Collado of Hippocratic AI talk about why nurses must be part of the process from early design through real-time oversight. They explain how nurses bring the critical expertise and patient-centered perspective that make these tools not only functional, but truly safe and patient-centered. As Scott emphasizes, in this process, nurses aren't a nice to have, they're essential.

So we actually have two different groups of nurses. We have a group of 4,000 nurses who are constantly talking with the agent and making sure that it's learning how to speak with patients. But then I also have an internal team of nurses that's led by our director of nursing. There's about 30 to 40 of them who are supervising and evaluating and making sure that we do our own

internal test of what this product looks like. Because we've worked with these agents for so long and we've worked with different scripts, we know what we're looking for, right? And so there's that added safety feature. But we always have nurses on the line as we start to do these calls. And the reason for that is that they're supervising, they're making sure that again,

The agent is saying what they need to say, that there aren't any gaps in conversation. There aren't any weird things being said throughout. And so we do that for a while until we ourselves are comfortable and our healthcare system partners are also comfortable with it too.

And then once that period is done, we go into what we call a semi-supervised phase, which means we still have nurses eyes on them. Our internal team of nursing reviews every transcript that comes through that call, every summary to make sure that there wasn't anything that was out of the ordinary. There wasn't an abnormal, um,

situation, which is what we refer to as our escalation. If a patient is short of breath, who does it escalate immediately to a nurse, all of that is working. And we, we do that over the course. I mean, sometimes I have 800, 900 calls to make sure that again, that this is, is tested in all different areas and that even our eyes are on this product at all times.

And then after that, that's when we go into more of an autopilot phase. But even then, we in our healthcare systems have access to our transcripts. We have nurses who are a part of this system and who are checking to make sure that if there's an escalation that needs to be dealt with, whether it's immediately or within four to six hours, there is always a human element in that.

And so when you think about that process from end to end, you have a clinician from the very beginning of that agent being delivered out into the public realm, all the way to where we're starting to send it into an autopilot phase. And so that to me just builds on our safety component and our commitment to our patients and to clinicians. This is an unknown territory for so many of us. So we want to do it right.

The number one thing that I am most cautious about as we make product and engineering decisions is not listening to our clinicians in the room. That is really what I'm most cautious about. How do I make sure I'm listening to the experts in the room when I'm building a product, especially in a space like healthcare? Maybe even more so in a space like healthcare. And so that's an area where we're abundantly cautious. How do we make sure that we are not making diagnostic and clinical decisions that

To Amy's point earlier, we're leaving those decisions to the experts, the folks who know their patients and know how to navigate those situations. And we're relaying that information accurately to patients. Product Director Scott Collado and CNO Amy McCarthy of Hippocratic AI from Episode 102, AI in Play, Smarter Care.

Check out this and more from our AI in Play series. Subscribe to see you now and share with your friends from wherever you get podcasts. I'm Rebecca McEnroy. Thanks for listening.