Legacy compliance models break down when applied to high-speed AI systems, turning human-in-the-loop requirements into legal friction and "process theater."
Jimi Du, Senior Corporate Counsel at Lead, explains that oversight needs to shift toward system-level statistical analysis and continuous monitoring.
Du notes that proper AI governance requires a clean underlying data architecture and urges companies to experiment with AI to stress-test their internal controls.
Compliance policies were built to work at the speed of humans. But now that autonomous systems are in the mix, compliance teams are running into the limits of legacy governance models. Oversight processes tend to break when applied to billions of tasks, where keeping a human reviewer in the loop creates legal friction and process theater.
Jimi Du has observed the transition from multiple perspectives across the financial system. Currently Senior Corporate Counsel at the regulated fintech partner bank Lead, his resume reads like a tour of the modern regulatory machine: private practice at Linklaters, regulatory counsel at Crédit Agricole CIB and Revolut, and a stint as a Senior Attorney at the FDIC. That mix of Big Law, Wall Street, fintech, and federal supervision gives him a battle-tested view of how AI and data governance intersect.
"Human-in-the-loop compliance has to move beyond a checkbox safeguard into a deliberate decision about where human oversight adds value and where it only creates friction or theater," says Du. Legacy oversight models often begin to strain when AI takes on tasks humans never actually performed. A loan officer can review a single application, but no one expects them to oversee a fraud-detection engine. At that scale, it's more about putting human oversight into more productive, high-level positions.
Outcomes over optics: Pointing to the gap between where human checkpoints typically sit and where they would actually be useful, Du says that “The biggest shift is moving from checking whether a human reviewed it, to whether the system is truly explainable, traceable, and trustworthy at scale. Are we involving them down at the transactional level where they're looking at samples, or are we putting them at a higher level to review outcomes, look at trends, and spot drift in the system?”
That tendency toward process theater stems from a historical baseline. Regulators have historically favored prescriptive checkpoints as a preventive measure, even as they acknowledged the limits of human review and the hurdles posed by auditing black-box models. Yet supervisors are actively moving away from that baseline. In recent congressional testimony, the FDIC's head of risk management described an outcome-focused approach to AI use without prescribing rigid human-in-the-loop requirements. Globally, authorities are encouraging a diversification of approaches, resulting in a patchwork of rules that ranges from the EU’s prescriptive mandates to the more open sandbox fostered by the FCA in the UK.
Some institutions still lean on legacy compliance reflexes. Relying on a human reviewer as a defense limits real risk control, particularly when that employee shares the exact same institutional assumptions and blind spots as the model they are meant to oversee. It's just process theater. To build actual safeguards, oversight must move toward system-level statistical analysis and continuous monitoring.
The liability boomerang: On the question of who ultimately owns the outcome when AI makes the decision, Du is direct. “Even if you use AI, liability cannot be outsourced. You are still accountable for everything the system does." He adds that the human reviewer is not the failsafe that organizations assume it to be. “If the human is sharing the same biases as the model, their review is not going to mitigate the problem.”
Post-mortem policing: Reflecting on the structural tension regulators face between outcome-based standards and the pull toward checkbox compliance. Du says, “When you get to the bad outcome, it's already too late. So how do we prevent that? A lot of times, there's a tendency for regulators to go back to checking the box.”
As AI systems take on more intricate workflows, individual accountability at the level of a single reviewer fails to map onto reality. Instead, institutions now treat AI accountability as an enterprise-wide decision. Navigating that transition typically requires a blend of new technical infrastructure and cultural adjustments. On the technical side, governance needs to support a big picture perspective on how models behave in production. Culturally, leaders are rethinking how teams share ownership of systems that cut across risk, legal, and product functions. Focusing solely on a written human-policy document misses the mark; teams must also prioritize auditability and the ability to trace the data behind the models.
Silos sabotage scale: Du says that coordination across teams is often the first thing to break down when organizations try to build real AI governance. “When you have siloed teams, where each team owns their own piece and they just want to be accountable for that piece, they're not going to work well together to put together an AI policy that's going to govern everything."
Garbage in, governance out: Du explains that implementing AI often forces a diagnostic reckoning with the underlying data infrastructure. “You cannot have proper AI governance without proper data governance underneath. It all breaks if the foundation isn't clean,” Du says, adding that implementing AI often serves as a “revealer of deficiencies in your whole governance system."
Kinks and all: On why borrowed frameworks will only get organizations so far, Du says. “You can have principles that can help your organization that you learn from the market, but all the kinks are going to be your own. Just going out there, trying things, and experimenting will give you the push needed to address a lot of these issues.”
None of that work is expected to resolve cleanly in the near term. The organizations that come out ahead will be those willing to treat the next twelve months not as a waiting period, but as a working one. Du doesn't expect the underlying issues to disappear quickly, but he's clear on the direction of travel: "I don’t know if these issues will be resolved in the next year, but the work to fix the underlying gaps has to start now, before they compound even faster under AI."
The views and opinions expressed are those of Jimi Du and do not represent the official policy or position of any organization.
