How SDSC Uses AI to Transform Surgical Training and Practice - Episode 241
26:08 Share

Hello, and welcome to the NVIDIA AI Podcast. I'm your host, Noah Kravitz. Our guest today is a machine learning and AI leader who's worked on projects ranging from sustainability and reforestation efforts to creating a virtual clothes swap platform for environmentally friendly fashionistas. But for the past year and a half or so, she's been serving as director of machine learning at the nonprofit Surgical Data Science Collective.

where she leads research focused on utilizing video data from surgeries to develop tools that can provide surgeons with immediate feedback and insights on their performance. She also recently gave a TEDx talk titled

Why you want AI to watch your surgery, which I encourage you all to go check out on YouTube after you listen to our conversation. Because she's here right now to talk with us about the potential for AI to help surgeons bring better healthcare to everyone. Margot Mason-Forsyth, welcome and thank you so much for joining the NVIDIA AI podcast. Hi, Noah. Thanks for having me.

Margot, maybe we can still talk a little bit about your background. I alluded just a little bit in the intro. You've worked on various projects after studying machine learning and leveraging your skills and experience for a lot of AI, for what we'd call AI for good projects, but really just, in my view, projects that are helping, you know, improve quality of life for everyone. So maybe you can detail that a little bit and then kind of bring us up to the present with how you started

You got involved with the Surgical Data Science Collective. Yeah, sure. So like you said, I've been working in the AI field for quite some time now. My first field of studies was computer science, so I've done enough software development. And then I realized that I wanted to do a bit more scientific projects, so went back to finish my master's and specialized in computer vision.

Computer vision is something that I really, really love because I'm a very visual person. And so analyzing images and videos is something that I'm pretty passionate about, I would say. And

From that, I've worked on many different projects with very big video and image files. So like you said, I've worked on several projects that go from analyzing lumber scanning, for example, to satellite imagery to detect deforestation or nasurgical videos. So it's been quite a ride for sure. And I

And I've learned a lot mostly on how to productize AI and how we can use AI to make an impact in the world and have a focus on is it actually going to be useful, you know, and make a difference at some point.

That's kind of how I see my career so far. Have you found that the projects you've been drawn to, has it been kind of being drawn to the next sort of technical or scientific challenge, kind of pursuing the craft and kind of pushing the boundaries of what computer vision and image and video analysis can do? Or have you been driven more by the mission?

of these different projects or have you just kind of worked out that you've sort of been able to follow both of your blessings, so to speak? I would say both actually, yes. Slightly luckily. Yeah, always. I mean, I've always been passionate about different sciences. So I actually have a hard time focusing on only one thing or one project. And

And when I learned about climate tech, for example, I really wanted to see how I could help and how AI could help process all of this gigantic satellite imagery. You know, remote sensing is really hard to process. And then I

I learned about surgical videos and I learned that there were thousands of terabytes of surgical videos that were not used. And it's a pretty big challenge because surgical videos are really heavy, really long. You can imagine eight hours procedure. No one really wants to watch those videos. So that's when I was thinking that

AI is indeed a perfect tool for this kind of data that is really big, really long, but also has a temporal element to it, which is

quite difficult. And when I thought about that, it was a really interesting, impactful project. But also in terms of technical challenges, I thought it was interesting. And that's actually what made me join SDSE in the first place. So how did you find out? What drew you to Troves of Surgical Videos?

I started to work at the Surgical Data Science Collective, SDSC, when I met the founder, who is a pediatric neurosurgeon, Dr. Denneho. And he introduced me to this issue. You know, he was telling me I have all these videos saved in drives and I don't do anything with them. And I know many of my coworkers and friends and other surgeons have drives with hours and hours of surgical videos and they are just sitting on their desk, not really doing anything with them.

Right. Just to context set for the audience, for me too. Is it standard procedure that surgeries are just all video too? Are these cameras that are inside of people sort of, you know, guiding the surgeons or are they sort of operating room overhead cameras or how does this, how does it work? Surgical video. I,

That's a great question. It's pretty diverse. We have a lot of endoscopic videos and microscopic videos. So endoscopic will go, for example, through the nose or any other part of the body where you need to see inside. And actually, the endoscopic videos are a really good data point for us because we see the computer vision algorithm sees what the surgeon sees. And that is golden because there is a lot of information in these videos

For the microscopic videos, it is also used by the surgeons sometimes when they do the surgery to really magnify what they're looking at. For example, if they're operating on very small arteries, they need to have this big intense zoom. That's what they're going to be using. Often it's in 3D for them, which we have some on the 3D videos as well. So yeah, it's a mix of microscopic surgical videos, endoscopic videos.

And we don't have yet the video, you know, kind of like camera security from the OR, but some of our collaborators use this kind of videos to get a sense of what is happening in the operating room. Right. And so you,

You met the founder. So the Surgical Data Science Collective was already in existence and you met the founder and got involved? Yes. It was pretty early on. So the Surgical Data Science Collective is a nonprofit organization that was started by Dr. Donohoe. And the main mission and the main idea was to create and analyze data.

a repository of surgical videos in order to improve surgical techniques and patient outcomes. Because till today, there are 5 billion people who lack access to safe surgery, and there are at least 4.2 million people around the world who die within 30 days of surgery.

So if we consider surgery as a disease, it would be the leading cause of death. That's why, you know, the goal of SDSC is to utilize this annual surgical videos to identify best practices, support medical education, or even predict potential outcomes and complications in advance of surgery. Right. So we've had other...

healthcare practitioners and people from the health industry on the podcast talking about the thing that comes to mind is talking about analyzing still images, you know, x-rays and scans and using AI to discover things often kind of, as you said, at that super zoomed in level that, you know, cancer prediction and that kind of thing.

Tell us about some of the opportunities and challenges involved with analyzing all of the surgical video. I would assume, you know, the first challenge is just gathering the data and then processing all of it, but kind of take us through it. What is it that, you know, you said improving best practices and real-time feedback, so maybe you can speak to that as well.

Yes. So the first challenge, like you said, is actually to gather all of this data. And like I said earlier, a lot of these surgical videos are stored on drives. And it's really difficult to get access to these drives. Sometimes, you know, you kind of have to go and fly somewhere and meet with the surgeons to be able to get the videos. Most of the time, the videos are not even recorded.

because people don't know what they can be used for. So why would they record them? So actually one of the biggest challenge is asking people to press the record button. - Right, I'm laughing, but I'm imagining, you know, if I was a surgeon, that'd probably be the last thing on my mind, right? So yeah. - Exactly, yeah. I mean, that is definitely not the priority. And then if they think about recording, so pressing this button, they have to export the video from the device

Walk around with a USB key, upload the videos on a laptop, upload to the cloud. So there are so many steps here for these people who are extremely busy, who have so many other important things to do of their day. It's definitely not a priority. So that is our first challenge. And that has been one of the biggest challenges that we've had.

But we've been pretty successful in gathering at least a good first base of surgical video library. By now, we are about 40 terabytes of surgical videos. Okay. And we expect to get more, you know, and the other challenge here is to get diverse surgical videos.

We don't want, obviously, for an AI model, we don't want videos from one surgeon in one hospital doing the same procedure. Hundreds of hours of tonsillectomies is only going to get you so far, I'd imagine. Yes, exactly. So that is the other challenge is how do we get these videos from doctors

diverse sources and diverse fields, which is also a lot of networking because you have to go and talk to the people and ask them to record and then do

Do they want to work with us so that we can start gathering these videos? So this is the other second part of this challenge of data collection. But in terms of the other challenges, obviously surgical videos are quite long, but they are also temporal. So videos, right? So it is a different type of models that you would use for still images. We have kind of the same architecture that you would use for other models, but

But we always have to think about the temporality of what is happening in the video. And that is actually how we implement most of our models. Let's say if you're trained to track surgical tools, you know, you have to think about all of the challenges that comes with that in surgical videos, which are going to be obstructions and can sometimes you have, you know, explosion of blood or something like that. And you want to be able to start trying the tools without losing them or...

who's dealing with these problems, which are pretty similar to other computer vision problems, but it is slightly more challenging because of how messy these environments are. Sure, I can only imagine. And so from a technical perspective, you know, obviously there are

We're hearing all the time these days about video models in the news kind of being opened up for consumer use, that kind of thing. You've been working with the Data Science Collective for going on two years now, is that right? Yes. So are you using, are you building tools yourself? Are you using off-the-shelf tools and kind of modifying them to suit? Do you have partnerships with other AI labs? How are you kind of fine-tuning the tools to get what you need out of them? I

A mix of all of what you just said. We usually will, so, you know, we're a pretty small team and a nonprofit organization, so we will try to use the most efficient methods for us. A lot of the times that we'll be reusing some architectures that are existing and then fine-tuning them to our needs. Combining some architectures is something that we've done a lot, especially with the temporal model, so having...

you know, a mix of a CNN and a temporal architecture or we've been playing a lot with vision transformers and more recently with vision text transformers, which are the big models you're talking about here. And I

And a lot of the time, we will always be careful about new technologies. So we want to try them and we want to make sure that we stay on top of the innovation that is happening to see if we can apply it to the surgical data science field. Something that is quite

interesting and challenging with this kind of data is that it requires a lot of expertise that as computer scientists, engineers, we don't have. So we need to work very closely with clinicians and surgical experts. And that's where the most important part of the work is happening, actually. Not even the model architecture or the new cool AI tools. For us, it's really understanding what the

the expertise is and then what models should apply to bring that information that will be useful to the surgeons. Can you maybe walk us through an example of, and correct me if I'm wrong here, but I'm imagining you have partnerships with surgeons and other medical professionals and institutions, and are you sending them images?

images or videos to just kind of analyze and let you know what they see or kind of what's the I guess I'm wondering what the process is or what it's like getting from footage and people revealing the footage to then an outcome that other practitioners can benefit from whether it's you know a new technique or refining a best practice or something like that so for most of our collaborations we

We will work with clinicians and surgeons who have videos, but they don't have the computer science knowledge. So they will come to us to do all of the computer vision and analysis. So when they start working with us on these projects, maybe I can go through a concrete example of

We've been working with several NGOs who are focused on surgical education. One of them is called AllSafe, and they focus on teaching surgical procedures to several students all across low-income countries. And they do it through a digital platform. So it's online courses, and then the review is done through videos.

And so what we're trying to see here is can we analyze these videos and give feedback to the students with computer vision? And

that is useful. So that is the important point is that is useful. So then we will work on in developing the computer vision models to extract the features that we need to be extracted to do the analysis and then collaborate with the clinicians and the surgeons on what exactly do they need to have in the feedback or what do they believe is something we should focus on? Because most of the time, you know, we're going to look at something and I'm

I'm going to think with my engineer mind, oh, I'm going to look at this feature and I'm going to make this graph and it's going to be amazing. And then I say, and they're like, what?

So that's why it's called the Sociocode Data Science Collective, because the first step is creating a community with the clinicians and the computer science experts. And we also have some collaboration with computer scientist groups.

where we will work with them to analyze some of the videos we have. So that is almost a connection between we have surgeons who want to do something very specific and we have computer scientists collaborators who can help us do that specific task that maybe we don't have bandwidth for. So we are trying to expand that part of our community as well.

to really have a real impact and scale that because it's not only gonna be us, it's gonna be a whole community effort. - I'm speaking with Marder Mason Forsyth. Marder is the Director of Machine Learning at the Surgical Data Science Collective, a nonprofit that is using AI machine learning tools to analyze video data from surgeries to develop

tools and feedback loops and other mechanisms that can help surgeons with insights and feedback on their procedures and techniques and really just bring better healthcare to more people across the globe, as Margot was just talking about. You mentioned, you know, being a nonprofit doing AI research is a little bit unusual right now. What is that like? Are there big things that are either, you know, well, I mean, obviously I would imagine funding the

Resources is an issue, as it is for almost all nonprofits. But are there things specific to being a nonprofit AI kind of research group that stick out to you? So there are many interesting aspects that come from being a nonprofit. Like you said, resources are indeed limited. So we have to be creative in the way we train computer vision models.

We will always start simple, which is actually something I've always done and advocated for is if you want to start a computer vision project, maybe you don't need to start with the biggest model that exists. You know, start simple with a small data set, do a proof of concept and then iterate. So that is what we have as a development pipeline and research pipeline. We will always start simple and small and then scale. And that is great.

because of our resources, obviously. But to me, it's something that is actually good. I would do the same if I had, you know, 10x budget. I would probably do the same, but it helps in that way. And then...

It brings a lot of different projects. Being a nonprofit, we're able to work on projects that maybe we wouldn't be able to work on if we were a for-profit. For sure, it would actually be completely different. And that's why I really wanted to give SDSE a shot when I first met Dr. De Noho because I was curious about it. I was like, how are we going to do that? You know, I've never done AI. I've never seen AI done in a nonprofit. There are some others, but...

It's really research-focused and community-focused, which you wouldn't really be able to do as well, you know, for profit, I believe. We're going to ask this and answer, please, based on what's happened so far and or, you know, what you see coming in the near future. What are some of the big benefits for clinicians, for patients that you've seen or expect to see from, you know, not just the work that you're doing at The Collective, but more broadly leveraging AI to help

with the surgical process? The AI field will bring a lot of new and good things to the medical field, I believe. In the surgical space, which is what I've been exposed to mostly, it will bring a lot of standardization. Something I've discovered working in that field is that every surgeon in every hospital will perform surgeries and procedures in different ways, and no one really knows...

A, B, C, D of how you're supposed to do a specific procedure. So by having a tool here, AI, to first encourage people to collect the data. So first we're going to get the surgical videos. We're going to finally start looking at these videos that are not being looked at and then share them between surgeons all across the globe. That will bring a lot of stability

standardization, or at least they will start to talk to each other, which I think is kind of beautiful because right now you don't really have a good way to talk to each other. And through the surgical videos, the hope is that they will start to talk to each other. And when you can imagine so many applications, for example, I mean, the one that always comes back is education. Instead of using a medical textbook with drawings, the students can watch

a tutorial on how to do the specific procedure. So that's a big difference that I think will change a lot of things. And then being able to find best practices through this analysis of surgical videos is going to be pretty interesting because who knows what

is in these videos. And there's so much that has to be discovered. And there is a big need to be creative when we think about this data because no one has ever looked at this data and no one has ever really thought about what can we do with all of that? And what is my question that I want to be answered?

Right. And that's one of our challenges actually is sometimes we ask surgeons, oh, what do you want to answer through all of these videos that you have? And they don't really know because they haven't had this option before.

Right. That's interesting. It makes me think of, I mentioned there were some of the examples of MRI and scan analysis and cardiac care and that kind of thing. And I'm thinking about the AI tools being able to help practitioners find, you know, differences in cells on a very, very, you know, sort of nano basis, right? But even with that, I'm thinking, oh, well, they know what they're looking for.

or even if it's they're looking for an anomaly, it's still kind of we know what we're looking for. But yeah, with surgery, my very kind of naive, you know, not knowing much about the field coming in this conversation thinking, oh, well, you know, video footage is being used to train AI systems. So are we moving towards, you know, better education for humans or even training robotics?

surgical arms or that kind of thing. But it's fascinating to hear you say that. It makes sense to me as a non-surgeon that what would they be looking for? It's, you know, it's not the same as looking for, you know, an anomaly in a cell that might stick out. I think at the beginning, probably of the, when they first started to analyze MRI with AI, they also had to be creative because

Someone had to be asking for these questions. And for surgical videos, one of the first steps would be to look at anomalies, which actually we're starting to do now is what are the outliers? Who is using this tool and no one else is using it for the same procedure? So we are kind of starting with the low hanging fruits, I guess. But the deeper...

existential questions are not there yet. And I'm really excited to work with the clinicians to help them come up with these questions by showing them the data because no one else is going to come up with these questions. It has to be the people who are working every day in the OR. And actually, the videos are a really great source of data, but there is so much more going on. Obviously, there is, you know, the patient data, there's the patient outcomes, there is everything that is going on in the operating room. And

And all our engineers have actually been in the operating room so that they understand what is happening behind that camera. And I've been in the operating room too a couple of times now, and it's really helped me understand better what is happening. And sometimes when we have a new procedure time that we're exposed to, I...

go to the OR because I want to understand better like oh some random questions sometimes like where are you where is it in the body or how many people are operating because sometimes you have more than one surgeon it's just so many things that you don't capture in the video but there's still obviously a lot of information in the videos fantastic well

Let's go for listeners who would like to learn more, or hopefully, perhaps, there's even some surgeons, some clinicians listening who are thinking, oh, I have a surgical video that, you know, in a shelf on a drive somewhere. Maybe I can send it and help out the cause. Where can listeners go to find out more about the work that the Surgical Data Science Collective is doing, the work that you're doing?

perhaps to get involved as a partner, who knows? Where can listeners go to learn more? - So we have our website is the sojucorvideo.io

And you can also find us on social media at Surgical Data Science Collective. And I would also encourage if anyone is a computer engineer, computer scientist who wants to work on a different project that they've been working on and are interested in surgical AI, to also reach out to us because we are working with quite a lot of different parts in this. So anyone who's interested should reach out to us. Fantastic.

Well, Margot, thank you so much for taking the time to stop by, join the podcast and talk about the work you're doing. It's, I don't know, stories like this with the technical aspects kind of match up with the societal impact, I think are just fantastic.

fantastic stories. There's sort of something for everybody, right? And it sounds like you're finding a really interesting path to fuse your technical interests with making an impact in your own work. So congratulations and all the best of luck to you and all of your partners and cohorts at the collective. Well, thanks, you know, thanks for having me on the podcast. It was really enjoyed the conversation. Me too. Our pleasure. Thank you.