Fraud Detection in the AI Era // Rafael Sandroni // #301
41:19 Share

Rafael Sandroni, I'm the founder of the Guardian AI. It's an AI-secured company. Usually I mix Brazilian coffee with Italian mocha, so that's the best combination, but also I like the espresso.

We'll be back for another MLOps Community Podcast. I'm your host, Demetrios, and today we're talking with Rafa all about the security vectors that you got to watch out for when it comes to building apps or just using LLMs. I particularly enjoyed the moment that we touched on traditional fraud detection that he's done at places like Nude Bank, NU and U.

Shout out to their engineering blog. And then we contrast that with these days, the kind of fraud that you can do when it comes to LLMs and if you have AI systems in place. Let's get into this conversation with my man, Raphael.

I know you were working on Siri and I know that you were doing stuff that was to help Siri understand other languages, but Siri's been around for a while. It's also been pretty wild how the evolution of Siri has been happening. And you posted some stuff that caught my eye on LinkedIn about like how to make assistance, AI assistance, better things that you need to like keep in mind when you're

creating AI assistants. What are some of these things that you've recognized as best practices that you can share from your time doing that? And I know Apple has some pretty strong NDAs, so you don't need to get into specifics, but just what are some key learnings? Okay, so what I understand is that with the LMs, the APIs and so on, it's much easier for everyone to create AI assistants.

And based on my experience, I don't know, Siri is one of the top five AI users in the world. It's a huge problem to make sure that everything is going like a testing, make sure that the response is like a kind of quality, but also the orchestration, like depending on what AI assistant is this. So you need to...

orchestrate, integrate with APIs. You need to make sure that everything is integrated. Test disintegrate because, I don't know, you need to make sure that you have contracts between the integration, the tests as well. So,

You can see recently using a mix of AI and human to test. Yeah, I will say like the test is one of the most important part. Like if you are creating some value for people, you need to make sure that the AI, this AI system is like working well. You don't have a kind of regression. So I work at Citi on some components to make sure guard rails, to make sure that you don't have a kind of ulcers

referencing like a kind of profanity and competitors and so on.

And if you think about, I don't know, a bank using this is a regular regulatory markets that have a kind of regulatory as well. You need to make sure that the answers of this AI is you have a kind of control, right? Yeah. So recapturing tests, the integration, make sure I have a kind of contract and also controlling like I have the gadget rails or controlling the output. Yeah.

I like that. A good start point. I feel like Siri is the ideal abstraction layer that I want an agent to play at or an assistant because it can do anything on the phone and it can go into the apps. I wish it could go deeper into the apps, right? But I guess that's just each app has to make things discoverable for Siri. And so there's this

question, do I want to have my own AI assistant inside of my app or do I want to have Siri be the AI assistant inside of my app and that type of thing.

potentially are what different apps have to grapple with all the time. You were doing the guardrails before guardrails were cool and before there was these huge blunders. So it's funny to think about if you didn't do your job well, Apple could have been the Air Canada. It could have had that brand name.

awareness that no brand really wants. But then I know you went and after you work at Apple, you went and worked for one of my favorite engineering blogs of all time, which was NewBank, right? And NewBank is awesome because of how transparent they are in their engineering blog. I think they're, I say this a lot, but I don't say it enough.

The NewBank engineering blog, I've learned so much from them and they are so transparent that I'm thankful for that. What were you doing at NewBank? Yeah, so it's a little bit complex because I spent almost three years over there and I had the opportunity to work together with the part of this team. But I guess...

The important part, the thing that I like at Nubank is like sharing like knowledge. So you have a lot of, let's say, workforces, meetups internally to make sure that everyone is in the same bar. And my first experience at Nubank was interesting because I was part of, let's say,

a new feature internally. You were building a personal finance assistant for Nubank users. And it was part of my startup that I founded years ago. And you had the opportunity to create the basis of this AI assistant. And the funny part, Nubank decided to buy Nubank

One of my competitors, like a personal files management app, and it was part of the due diligence, understanding internal tools and so on.

So it was an interesting part building this AI assistant. I guess now it is part of the new big app, helping people to have savings, controlling the savings, have a better planning, financial planning. But after that, it was part of like a fraud team. Fraud team was basically the same.

And the challenge was like, I guess maybe you saw this blog post about the real time machine learning for machine learning models for fraud. I guess everything tech and bank have this challenge, like understanding blocking fraudsters in transactions online or like onboarding for new customers.

And yeah, I had the opportunity to like create the tools, like doing retraining models automatically for to understand and update the models with like new patterns and also improving this, the performance side of this model. So I have, I have the data design interesting blog post about this.

Yeah, and that was all traditional ML, right? So you went from... Yeah. With Siri, you were working on traditional ML or you were doing the guardrails or you were doing software engineering. What do you feel like the evolution was there between those two? Yeah, it's kind of funny to say traditional ML because I know when you say NLP, I believe it's traditional ML, right?

But yeah, but I believe everything is connected, right? Probably in a level of maturity of AI, you need to implement traditional ML. You need to go out of APIs and implement a kind of some models internally to have more control. So not only the LLMs to use open source, but also, I don't know, if you need guardrails, probably...

The best way to go to traditional NLP, like a classifier using small language models and implement this. But the challenges is like around not only the AI, but the software engineering, like how to deploy. Yeah. So I believe that everything is connected. It's easier right now to use, starting using AI, right? But I guess the challenge is the same of four or five years ago. So if you need traditional ML, so the challenge is the same. All right. Now,

Talk to me about what you've been learning about recently. Yeah, so this is the interesting part of this talk. But I left Apple months ago to start something. And I started learning about AI security, how to make sure that every AI application, AI assistant is like it have, it's protected. Guys like crafters, hackers and so on.

So I started with the OWASP, top thing for LLMs. So later I joined as a member to understand more the challenges. And it's a huge problem, to be honest. Everyone that I'm talking about, like banks, larger fintechs, healthcare companies,

Legal, startups, local, like everyone is like implementing, but it's a kind of challenging topic because you need to make sure you need to some people like. So basically they want to get hacked and then they want to put a password manager in place. Yeah. So basically, so I think.

depends on the culture of the company. So I guess the banks and the financial markets are more aware of these risks. So they face the same challenges on the software side. So probably the AI, they are taking care of this. But that's an interesting part. I would say that

You have two components to make sure that you implement like a secured standards on AI. So the first is you need to implement the AI following a zero trust approach. So make sure that you have the permissions, the credentials and the integration is safe.

After that, you need to do like a kind of test, security test, so it's ready-teaming. The same way that you have like a pain test in the software side. So to discover the vulnerabilities. Understanding these vulnerabilities, like you have the opportunity to fix on your implementation or implement use guardrails. So the second part is a way to understand

Following the zero-trust approach, you don't believe that the LLM is 100% safe, and we can implement these guardrails not only for the security, but also, let's say, to prevent a kind of hallucinations, a kind of off-top, and so on. And the security part of the guardrails is very challenging because these guardrails show this model, show no security,

what patterns, what's like an impromptu injection method that have more effectiveness to block these attempts. Yeah, so an overview of the AI security, I would say, like a starting on the AI application. So,

have this confidence. So as you're looking at these different vectors that LLMs can be basically corrupted on, and you're thinking about guardrails, you're also thinking about zero trust. First of all, I would love to understand what you mean by zero trust for the LLMs. What does that actually look like in practice? It's a certain approach, like when you implement a software, so make sure that you don't have a

You have the access just for that function, for that resource.

So, AI is the same thing. So, if you have AI assistants that have access to APIs and other tools, like making sure that the scope is very defined, the access, the permission is very defined. Basically, it's like an approach that the developers, the dev team can follow and adjust in the implementation. Yep, that makes sense. It's more like a mindset. And trying to limit

the scope, which is a hard thing to do, or it's a very fine line because with the LLMs, they generally perform better the more context you give them. And so you want to give them as much context as possible, as long as it is the right

permissions and so trying to figure that out and trying to architect your system in a way that is going to know who has permission to what can get a little tricky have you seen good ways of doing that what do you mean about the permission here so i guess it would be like who has access to what data who has access to what documents that type of permission and then because you don't want

an LLM to be able to, maybe it has access to every database or every channel in Slack. But if I'm asking a question, I don't have access to every channel in Slack, right? And then all of a sudden the LLM does. And so I'm able to find out things that I am not privy to.

Yeah, so this is a part of making sure, like defining what's the most important data to share with the AI. So you don't have to share all of your database with the AI. And maybe you don't need to share all of your table, data table with AI. So it's complex, but I guess it's following this mindset and being more like restrictive about the data.

And then what... So we were talking about the different attack vectors that you've been seeing, and one of them was jailbreaking, one of them was red teaming. I think people have...

Seeing a lot of examples of those on the internet, what have you been thinking about when it comes to the security vulnerabilities and then making sure that guardrails work? Like how do you architect the guardrails so that they are more effective? I guess the most challenging vulnerability of AI is the product injection. So you can pass some contacts, do a kind of social engineering with AI, and then, I don't know, some X.

have access to some data, the system prompt will be like the first step to understand more dynamic of the AI or AI agents. I believe like implementing the right guardrails to prevent that you have, let's say, malicious inputs that can affect your LLM or your AI agents, like the chain of LLMs, you'll be

you'll be like the best practices. So that's what I'm seeing right now in the industry. So it's not the final, it's a kind of mitigation. It's like a one, it's not 100%, but you can eliminate a good part of fraudsters or malicious users that can try to expose your AI. So

And usually it is guard-real, so you implement this like a real-time. You are implementing the orchestration and it's working real-time blocking some patterns. And also, if you have applications that have access to the internet or documents or data that you are not the owner of,

you should make sure that you can, you should like a filter and make sure that you don't have an indirect injection. So there's an example, a very interesting example from the conference called Black Hat of one guy trying to the Microsoft copilot that was configured to summarize emails. And basically it opened a vulnerability that said,

An external person can send an email to this corporation and then use a RUGAP. So every email could be summarized and in the last step, let's say, replied with the AI agent, depending on the topic. And this external user sends an email with a prompt injection, hidden by the human eyes, but using coding and colors. But the AI code understands and the

it starts sending phishing emails for every email in the user base. So it's an example. So if you have access to an external road, you should make sure that you don't trust on the prompt instructions. Don't put your guardrails on the system prompt. Use like independent models, machine learning models, to filter and detect this kind of, let's say, patterns.

So it's very challenging, again, because it's not 100%. It's just a mitigation, so you can be like 100% safe. So let me see if I understood that black hat example, because that is a new one that I haven't heard before, where if someone is using Copilot and it's in their email account,

then you can get a phishing email that has white on white text. So white text on a white background that the human eye is not going to see, but the copilot

will see and it will think, oh, cool, I can summarize this. And when it summarizes it, then that goes into the LLM and it does an indirect prompt injection, which says basically send an email or summarize this email and send it back to XYZ. And so if you don't have two things in place, one, just the human touch that needs to OK the send.

or return reply email, or two, that you have some kind of an external independent critic that is judging what the co-pilot LLM call or agent is doing. And then it's saying, hey, look, here's all the information. Here's what I did.

is this correct? Here's the prompt. Here's what I've done. Here's what information I got. And then you have that judge that says yes or no, this is correct or this isn't. You might want to go back to step one. So you could get into a whole lot of trouble if that is not what you're doing and how you're architecting it. So I've seen that in a way, that type of design pattern where you have almost like this catch-all critic at the end and

and you can throw everything into it and you can say, looking at all of these steps, this is what I've done. What do you think? Is this the right way of doing it? Yeah, I guess the point is that looking for the, you have a judge, LLM is a judge, and the same vulnerability that you have in the LLM you have on these LLM as a judge. I guess the solution that I'm seeing, these are like using traditional machine learning classifiers

And that you don't have the same vulnerability of the LLM. It's not a LLM. And then we can mitigate. Don't expose another channel, another space. Okay, so you're saying don't even use another LLM because then you might just get this cascade of prompt injections.

Yeah, don't use another LLM to filter malicious inputs because this another LLM can suffer and can follow these malicious inputs as well. And what are some other ways that you've been seeing security vulnerabilities or some real life examples of things that are happening in the wild? I'm working with some fintechs, understanding the challenges and validating these business tasks or validating some ideas around AI security.

There are some companies, the fintechs, working, building AI assistants on the WhatsApp. And these AI assistants can do transfers, bank transfers, can pay bills, and so on. So we have basically like AI assistants that have access to your bank account. Or in other scenarios, you have your bank account managed by AI on the WhatsApp. Yeah, the tricky part is that

i saw an interesting example recently some hackers trying to override the value of the transfer for example or override the account balance that you have in your account so so this is in real life you actually don't have you you don't have a lot of problems because you this scenario that i'm explaining like

These startups have, let's say, infrastructure as a banking in the core. So we have another kind of validation. But that's why it's important to follow like the zero trust mindset because we can like defend and protect these malicious behaviors on the software part, not the AI as well. But it's important because as well, so...

You can have the AI trying to answer something that is following some bias and talking about competitors and generating a kind of reputational risk for the company. I'm trying to think how that looks and what the benefit would be of having

me be able to do my banking through WhatsApp. And so I guess the benefit for a end user like myself, I'm using WhatsApp and I say, oh, send 50 bucks to Rafa.

And then it knows which Rafa I'm going to send it to and it will send it or I tag Rafa as a person and then it has your phone number and my contacts book and it will then send it to you. But it feels like there's so many things that can go wrong there because it could just hallucinate a different Rafa or a different phone number or whatever. There's so many things.

vulnerabilities that I wonder if the small lift that we get from being able to interact with our bank account on WhatsApp is worth the gigantic risk that it brings.

Yeah, I guess that's the challenging of the developers, the ML engineers nowadays. So you should follow the tests, have the kind of AI quality to make sure that it doesn't happen. And I believe that you're learning how to create this kind of application, right? Like a mission creation application.

Yeah, that's a very challenging, but I guess very interesting problem to work on. Yeah, it does feel like we're still exploring where and how we are going to interact best with AI assistants, AI agents, what the ways that

the public prefers to use them like you said maybe for a banking app it's not the best option but potentially for a food app it is the best because we can just text the food app and say I'm hungry I want something in the next 30 minutes and it gives me five different suggestions and then I can just say yeah I go with suggestion two but if it's for a banking app and it's send

$20 and next thing you know I sent $2,000 because there was some kind of a mess up or a hallucination then it's a lot more risk than if I get pizza or pita so what I'm seeing is you have this I don't know you have AI bank assistant like you can talk with the app so depends I believe that there is a profile a specific customer profile for this for this application

And so it's a risk, it's risker, but I don't know. I believe that the market's going from this side, I don't know, looking for early stage fintechs building all this space. But later, in the next few years, I don't know, like banks could have a new channel as the same of the mobile. So we have these legacy systems, banking system, and then they

try to follow the fintechs and build the mobile application, make sure that the UX is nicer, everything is smoother, and maybe the next wave you'll be having an AI helping people to make transfer, understanding the expense and so on. It's a mission-critical application, more critical than asking for ordering food, but I believe that

There is a value of this application has value from some customer profiles. Yeah. And when you have worked on traditional fraud detection systems, and then you think about these new ways of fintechs having fraud vectors, how do you weigh these different areas and vulnerabilities? And how do you think about traditional fraud and almost...

I don't know what the new one would be called, but I'm going to coin it. I'm going to say new age fraud. What do you look at? Is it different? Is it the same in your mind? How do you view those two?

I believe that with AI, both of parts of a fraud, the fraudster and the, let's say the bank, it's like they have superpowers. So it's the fraudster have like the tools to create live and as face and fake documents and so on. For them, it's becoming easier.

to bypass a fraud system and also internally inside a bank, a fraud thing. I guess you have the tools, but it's a challenging moment looking for this new kind of AI. But also we have the fraud around the AI, like an iteration. So it's a kind of fraud as well.

So basically you have more vulnerabilities around. You have the frauds with superpowers, with AI superpowers, trying to figure out what is the vulnerability that they could follow and deep dive.

And I don't know, it's a challenging moment. If you look for the cybersecurity as well, so not only on the financial space, but the cybersecurity in general. So it's a challenging moment because they think...

You have seen an increase of incidents, security incidents. That's because the fraudsters, the hikers have more powers right now. They can automate, speed up the process that they spend one month with some minutes with AI. Yeah, and you're able to just test where the vulnerabilities are so much quicker. I believe that the fraud, the same dynamics of fraud that you know right now and all these,

you see this dynamic in the AI as well. I guess the challenge would be almost the same. So we have some fraudsters trying to injection, doing prompt injection, and other side you have the guardrails or other reserves to protect them, learning the new patterns and so on. I don't know, maybe human in the loop as you have in the fraud operation as well. Maybe you should have this human in the loop in the AI operation as well.

Yeah, you make a great point about how many different levels there are that fraudsters can play at because it can just be falsifying documents or now these days you can clone voices with just like five seconds of voice. And so you can do social fraud and get information by doing those like social hacks. And then on the other hand, you have the LLMs themselves and how they can be vulnerable and

and you have the new systems that we're putting into place, like we talked about how, okay, now we're talking to our bank through WhatsApp. There's a whole bunch of vulnerabilities that come along with that. Or you have this superpower fraudster who can use AI to their advantage and scan for security vulnerabilities or fraud vulnerabilities. Are there specific design patterns that you've been seeing

that are now becoming very common and something that is like table stakes for if you're trying to deal with AI fraud on any of these levels. So I believe that knowing by first hand the vulnerability that a fraudster can attack, it's important. So doing the red teaming, testing,

your assistant doing a pain test on software and red team in general. I guess it's a must-have. You should do it before. So if you have a mission or critical obligation and so on, that's the best way. So you can have the first hand knowing the vulnerabilities and then trying to protect this, implementing the protection. I believe it's an approach that is becoming more common

And also as well, like the guardrails, implementing the correct guardrails for their applications, like depending of the use case, depending of the preparation, the risks that you like to accept or not. All right. So what other kind of things have you been seeing out there in the wild? AI agents. Actually, there is a recent paper report from OWASP.

for about the vulnerabilities in the AI agents. And it's interesting because when you have a lot of many LLMs talking with each other, it's become like more complex to make sure to control. LLM is not predictable and we have a lot of LLMs working together. It's even unpredictable.

And the problem is that in the AI agent's obligation, you can, let's say, with a combination of tokens, like say, TAC, can broke the first one, but the context, the information, the kind of noise can follow through the next steps. And also depending of the steps, if we have some documents,

data to access externally, one simple, let's say, one prompt injection that was received in the first step can affect, I don't know, the last steps, you know? But it's more like a general, it's not an example, a specific example, but it's interesting because

You can have a kind of noisy going forward for each step. More LMs, more, yeah, yes, more LMs you have in your system, more protection you should have to make sure that you don't have this information, just noise information going forward. Yeah, it's almost like one prompt injection can get propagated throughout the whole system.

chain of agent calls and you have to make sure to architect your agents in a way that like we were talking about before potentially it's not even a another llm call critic at the end it is something that is a bit stronger and more rules-based type of guardrails that you have i'm also looking at the owasp.org page right now on their top 10 vulnerabilities

And I find it fascinating that the first one is broken access control, because I think a lot of folks are trying to figure out how to get agents to act on their behalf. And this just feels like

a WASP is calling it out early and saying, hey, look, you can try and have the agent act on your behalf, but there's a lot of potential that you're going to have some access control issues here. Yeah, this is the challenge because it's a balance between autonomy and the security, right? Yeah. This is the challenging part. I don't know. You have been seeing a lot of

value from agents that do ER steps by themselves. But I guess this is a part of the zero trust approach that you should follow. So to make sure that you have the right permissions, is restricted for that task, for that scope, and so on. Yeah. These are things you as a machine learning engineer...

Going into this world of security, it feels like you had some experience when it comes to the fraud detection side of things, but...

Now you're having to learn a lot about, I'm sure, the security side of things. And are there pieces that you wish you knew when you started going into this world? What are some things that potentially you can just share as a nugget of insider knowledge that took you a while to figure out? But now that you know, you're very happy that you do.

I would even classify that a little bit further and say, what are some things you've learned in the last three months that you did not realize as a machine learning engineer, but now you see as like you're getting into the security world? Yeah, the first thing is that fraudsters and hackers, they have a lot of creativity. So even as a machine learning engineer, you are building an AI, LLM and so on.

you don't know what's the vulnerabilities so you know there is like a oh someone can use a base 64 and encoding a kind of method a b or c but when you put in it in the road like a lot of new stuff so you didn't figure out like that people can do this and it's very wild

And one of the most important things that I did recently is joining cybersecurity engineers with an understanding topic with these guys and learning their perspective about AI security. So because it's different as an AI machine engineer or AI research is different. They have another opinion. So let's say when you have

when you do red teaming they are not taking care of just the llm but the whole system for example so it's important and also about the protection the guardrail they don't it's not looking just for the llm but the whole system as well so the access the permission so and yeah i guess the

The insight that I have recently is they have another perspective. It's more broader about the AI system in general. I'm researching about the guardrails, security guardrails, data. It's important. So as we are building guardrails,

You should have data to understand what kind of methods that hackers are using or should or could use to bypass IAI or T-Series barge rails. Data is important. And if you look just for the open source data that you have, it's not enough.

Oh, interesting. Wait, tell me more about that. So it's like you should A, try to red team your own system, pen test your own system and thinking about the system as a whole, trying to break into it in any which way possible, not just the LLM. But then if you're only doing that, you don't know what you don't know. And when you put something out live, it's going to get hit by the very creative hackers that they are. And so is it like...

you should get a third party to try and pen test also? Or is it that you should, like, how do you get that data if you don't have it and you don't know that you don't know? Yeah, you should choose, I don't know, if you have an open source model that was trained on this data or you have, I don't know, a tool or provider that works on this, that collects data, filters this data to train like the Guardrails models.

That's because it's very difficult to create the guardrails by internally, by itself. So it can use LLMs as well, but it's vulnerable as well. And also to create small language models, you should have data. So it's a challenge like a

cat and hat. But the challenge is this, like you should, what the best way here to collect data is to work together with the cybersecurity team, the right team to understand the patterns and then implement this solution. Yeah, it makes sense. And I believe, I don't know, you have more, you see more

open source, open science and open source model around AI security. I don't have many tools and models on this space. Let's say some models that you can, you could use as a guardrails and make sure that everything is okay. So it's part of a path that I'm looking for, like to create more a kind of open science around it. Yeah, almost like...

The trick there is that as soon as you put out a data set that can help you understand where there are attack vulnerabilities, that data set is no longer valuable because it needs to be behind closed doors. It needs to be something that is almost not shareable or not public. Yeah. Do you mean the data, right? Yeah. Yeah.

Yeah, but it's a challenge of collecting data, collecting new partners, learning, and then updating it constantly. So like almost like this, as we did in traditional ML and as you did a ton on retraining fraud models, you're in a way continuously collecting data and then retraining these LLMs, but you're not retraining them in the sense that you're not going in

You're not going to train or fine tune the LLM. You're just updating your, maybe it's your prompt or maybe it is the LLM calls, something like that. Yeah, I believe that is a good point. Iterating around learning and iterating and improving, right? Yeah, it's a must-have process. You can inspire on the traditional ML and use it in the AI application, LLM applications. ♪

♪ I'm so much more special ♪