Courts do not accept “the model said so” as evidence. If an AI-generated alert leads to a suspicious activity report that leads to a criminal prosecution, the defense will subpoena the model’s decision logic. If that logic cannot be explained in terms a non-technical jury can evaluate, the case is at risk.
Explainability is not a nice-to-have feature. For financial crime AI, it is a legal requirement that most current architectures are not built to satisfy.