TLDR
As we forge ahead in the age of digital transformation, AI/ML have become powerful tools in the fight against fraud. However, as with any technology, they come with their own set of challenges and risks. It’s encouraging to see reputable organizations like the National Institute of Standards and Technology (NIST) acknowledging these challenges and taking steps to address them, such as developing the AI Risk Management Framework (AI RMF) which has been recently released. I believe that understanding these risks and mitigating them in advance is the best and secure way forward.
Risks and Threats
The first step in our journey is to understand the potential risks and threats associated with using ML in fraud prevention:
Data Privacy: ML requires vast amounts of data for training, which often includes sensitive information. In some situations this can lead to privacy issues and project delays if not properly handled. Therefore, there is a need for solutions on how to overcome these issues without compromising on privacy, yet not giving up on our appetite for strong data signals (yes, have the cake and eat it too).
Bias: ML models are only as good as the data they’re trained on. If that data contains biases, the models will too, leading to unfair or inaccurate predictions.
Lack of Transparency (Black Box Problem): Often, it’s difficult to understand why an ML model made a specific decision, which can complicate the process of fraud investigation and mitigation.
Adversarial Attacks: A skilled and “dedicated” society of scam artists can manipulate the inputs to AI/ML systems and trick them into making incorrect predictions. Just a few ideas of what they can do:
- Model poisoning attacks – The goal of the attacker is to contaminate the model generated in the training phase. An attacker could exploit this by feeding the system with carefully crafted adversarial inputs, intended to gradually push the model parameters in a harmful direction. Even if the attacker doesn’t have access to the initial training data, they can still influence the model’s behavior by poisoning the online updates.
- Perturbation attacks – In perturbation style attacks, the attacker stealthily modifies the queries to get a desired response. In the context of financial fraud, an attacker might try to manipulate transaction data in a way that a fraudulent transaction is classified as legitimate by a model trained to detect fraud. For instance, if the model has learned that high-value transactions are more likely to be fraudulent, an attacker could distribute the fraudulent amount over multiple smaller transactions to evade detection.
- Model (and policies) inversion – The private features used in machine learning models can be recovered. In the context of financial fraud, consider a machine learning model that predicts a user’s credit score based on their transaction history, and other information. An attacker with access to the model’s predictions and some basic information about the user could use a model inversion attack to estimate sensitive financial details of the user that were used in training the model. And they can do it in scale.
More on different ML attack vectors is here: Failure Modes in ML
Solutions and Opportunities
Now that we understand the risks, let’s talk about solutions:
Regular Model Auditing: Regularly review models’ performance and predictions, as well as fairness and bias. If models start to behave differently (more positives or negatives), it could be a sign of poisoning.
Control Model Retraining Velocity: Too frequent model retraining can lead to model poisoning. Less frequent model retraining can degrade accuracy. Find the right balance.
Adversarial Attacks: Implement robust cybersecurity measures and conduct regular vulnerability and penetration assessments. Automate anomaly detection and detect any regression if products, models or policies change.
Adversarial Training: Explore adversarial training, where intentionally created (or generated) adversarial examples are included in the training data. This can help models learn to resist these types of attacks. More on this in this systematic review.
Lack of Transparency: Utilize explainable AI/ML techniques to understand the decision-making process of our models better.
Defensive Distillation: This technique involves training a second model (the distilled model) on the output probabilities of the first model (the teacher model), which can make the distilled model more robust against attacks.
Ensemble Learning: Use multiple models that each look at different aspects of the data. This way, even if an attacker manages to trick one model, the others might still catch the fraudulent behavior.
Final Thoughts
Just like Metallica’s song “Fight Fire with Fire”, the dedicated community of fraud specialists employ cutting-edge, top-tier tools and algorithms with one simple goal: to profit from fraud without detection. Counteract them using the very same advanced methods.
Michael Kalika's Blog 