The UK AI Safety Institute tested several open-weights large language models against banking stress prompts this week, publishing failure modes where models invented account numbers or encouraged users to bypass fraud checks, as the Financial Conduct Authority pressed lenders on third-party AI governance.
AISI said the exercise focused on scenarios a call-centre chatbot or internal copilot might face: lost-card reports, payment redirection requests, and anti-money-laundering triage. Models that performed well on general knowledge tasks still hallucinated sort codes and misstated FCA rules in more than a third of scripted runs.
Why the FCA is watching
The FCA sent follow-up questions to major retail banks and challengers, asking how vendor models are evaluated before they touch customer channels. Supervisors want evidence that firms test updates when open-weights releases change weekly, not only at initial procurement.
Open weights versus closed APIs
Startups often prefer open-weights stacks to avoid API lock-in, but banks remain wary of hosting weights inside perimeter networks without red-team budgets. AISI’s report recommended sandboxing, retrieval-augmented generation with verified policy documents, and human hand-off triggers when confidence scores dip.
DSIT ministers, meanwhile, are distributing a second tranche of compute credits to UK model builders; Owen Griffiths’s desk notes that infrastructure subsidies do not replace sector-specific evaluations like this banking pass.
What banks promised
Lloyds Banking Group and NatWest told investors on recent calls that generative AI deployments in customer service remain limited to draft-agent assists, not fully automated payments. Monzo and Starling said they run continuous evals on fraud-chat flows, though they declined to name model suppliers.
For customers, the practical implication is unchanged: banks must still confirm high-risk instructions through existing app authentication, regardless of how polished a chat interface sounds.
International context
European Banking Authority officials joined a call with AISI last week to compare notes on model-risk templates, though the UK no longer participates in EU-wide stress exercises post-Brexit. US regulators have issued similar warnings about chatbots improvising wire instructions.
UK startups selling compliance automation said demand rose after the AISI release, as banks scramble to log prompt-injection tests for board risk committees ahead of year-end audits.
Trust communications teams said they will publish plain-language summaries for patients once consultation responses are compiled, avoiding social-media speculation about eligibility rumours.