The UK AI Safety Institute (AISI) published evaluation summaries Friday for three frontier foundation models—red-team refusal rates, tool-use misuse attempts, and bias probes on UK-relevant civic scenarios—and issued procurement annex language telling central departments to weight those scores in enterprise AI framework renewals due before Christmas, according to the release pack and a Cabinet Office commercial note reviewed by InfoHandle. The evaluations are not pass-fail certificates; they are comparative tables departments can paste into business cases when ministers ask why one vendor beat another on safety grounds.
What AISI tested
Each model ran through AISI’s protocol: automated jailbreak suites, human expert attempts to elicit harmful instructions, and structured tasks mimicking civil-service drafting with citation requirements. Scores are normalised 0–100 per category with confidence bands where sample size was thin. One model led on refusal consistency but lagged on faithful summarisation of long policy PDFs; another reversed that split—exact rankings are in the published tables, not repeated here as a horserace headline.
AISI stressed evaluations snapshot weights frozen in August; vendors shipping continuous updates must resubmit for a new row. That matters for procurement officers who might otherwise treat a PDF score as a year-long warranty.
How procurement is supposed to use it
The Cabinet Office annex adds optional evaluation clauses to the AI marketplace and G-Cloud call-offs: bidders may attach AISI rows or third-party audits mapped to the same rubric. Departments must document trade-offs when they pick a lower safety score for price or integration—ministers dislike surprises in Public Accounts Committee hearings. NCSC’s parallel guidance on deployment architecture still governs where models may run; evaluations do not replace data classification rules.
What vendors cannot claim
AISI’s press team warned against marketing “AISI certified” stickers; the institute does not endorse products. Alan Turing Institute researchers contributed scenario design for welfare-benefit chat simulations; their commentary in the pack notes that lab scores may not predict call-centre behaviour once prompts are customised. Enterprise buyers should still run domain tests on their own casework samples.
Enterprise buyers outside Whitehall
Local authorities and NHS trusts buying through shared frameworks inherit the annex if they use the updated templates—many do for lack of in-house lawyers. Private firms are not bound, but insurers asking about model risk on cyber policies have started requesting AISI tables where UK revenue is material. That secondary market effect is why labs engaged voluntarily; regulatory stick is still light compared to EU AI Act paperwork.
How scores differ from red-team blogs
Vendor-published safety cards often cherry-pick benchmarks; AISI’s pack includes failure transcripts redacted for public release so researchers can see attack phrasing without reproducing harm instructions. Departments were told not to treat higher refusal scores as better user experience—over-refusal blocks legitimate civil-service drafting aids. Trade-off worksheets in the annex ask buyers to note when they accept more refusals for lower hallucination rates on long documents.
Devolved administrations in Scotland and Wales received the same annex through COSLA and WLGA procurement hubs; alignment reduces forum shopping where vendors pitch different safety stories by postcode.
Open questions
Parliament has not codified AISI scores in statute; a change of government could deprioritise the institute’s budget. Model weights and training data remain trade secrets—evaluations probe behaviour, not provenance.
Training procurement teams
Cabinet Office Commercial ran webinars for category managers on mapping AISI columns to G-Cloud lot descriptions; attendance was mandatory for departments with live AI call-offs. Slide decks show how to score bidders when only two of three models have AISI rows—tie-break rules still favour integration tests. Civil service learning modules added a forty-minute block on documenting ministerial submissions when safety and cost diverge.
For UK enterprise procurement leads this month, the concrete shift is dated: three frontier models with public AISI summaries and Cabinet Office text ready to drop into renewals—use it in tenders or explain why not.