CSIRO has opened a Canberra adversarial generative-AI red-team laboratory to stress-test large language models and copilots before they are cleared for Australian Government procurement panels, according to officials briefed on the facility and parliamentary submissions the science agency has filed on AI governance. The lab extends Public Governance, Performance and Accountability Act vendor assurance with structured prompt attacks, data-exfiltration drills, and plugin reviews aimed at agencies that cannot rely on vendor marketing decks alone.

Why procurement needed a red team

In evidence to the Joint Committee of Public Accounts and Audit, CSIRO said enterprise generative AI use in government remains narrow—mostly meeting transcription and summarisation—while researchers access models through approved cloud test beds. Any new product still passes cyber, contract, and records reviews. Procurement questionnaires rarely capture how a fine-tuned assistant behaves when users paste sensitive paragraphs or when connectors fetch external URLs without logging.

The red-team lab, near CSIRO’s Black Mountain campus, runs before vendors join multi-agency panels or embed copilots in collaboration suites. It complements penetration tests focused on network perimeters rather than model behaviour at the chat interface.

What testers run

Teams from CSIRO’s cybersecurity and data sciences branches adapt methods from the Talos program on poisoning, backdoors, and adversarial examples in machine learning used for threat detection. Procurement exercises include jailbreak attempts on chat UIs, probes for training-data leakage, and checks whether administrative actions leave audit trails when staff paste literature or code snippets.

Vendors in early pilots signed non-disclosure agreements and received remediation windows before results inform panel recommendations. CSIRO documents behaviour under defined scenarios so budget holders accept or reject residual risk—it does not certify models as safe for every science workload.

The Digital Transformation Agency’s policy for responsible AI in government expects agencies to assess benefits, risks, and legal authority before deployment. CSIRO’s lab gives science and industry portfolios a shared evidence base when multiple departments eye the same vendor. Officials said results propagate into procurement notes, not public scorecards, to avoid tipping off attackers.

Microsoft 365 Copilot evaluations CSIRO extended internally informed which event types agencies should log; the lab generalises those lessons to other model providers bidding on analytics and service-desk automation.

Limits and politics

Red-teaming generative models is non-deterministic: a pass in September does not guarantee behaviour after silent weight updates. CSIRO recommends contractual notice for model changes and continuous monitoring—controls it already applies to Azure OpenAI and AWS Bedrock sandboxes with cost caps.

Opposition members asked whether capacity should sit in the Australian Cyber Security Centre; CSIRO argues science users need domain-specific attack libraries for genomics, climate, and manufacturing data, not only generic prompt lists. Ministers have been briefed that the national science agency is building domestic capacity to question AI vendors rather than importing unchecked tools.

What vendors should expect

Suppliers pitching generative AI into Commonwealth settings should budget two to four weeks for lab cycles including fix-and-retest. Failure does not automatically ban a vendor nationally, but it slows panel onboarding. Success accelerates entry provided contract teams still negotiate liability and exit clauses the PGPA Act requires.

Science users and enterprise overlap

CSIRO enterprise staff already route sensitive workloads through approved clouds; the lab’s procurement gate ensures science teams do not bypass review when a vendor offers a shiny copilot inside an analytics bundle. Black Mountain researchers said attack libraries will grow to cover retrieval-augmented generation that cites internal datasets—scenarios generic OWASP prompt lists miss.

State governments have asked whether outcomes can be licensed once validated federally; CSIRO has not committed to a fee schedule, prioritising Commonwealth and own-enterprise queues through year end. Universities watching the program hope summaries inform research-grant conditions on generative tools without duplicating full red-team cycles on every campus.

For Canberra’s policy community, the lab signals that responsible AI is operational: someone at Black Mountain is paid to break these systems on purpose before taxpayers do so accidentally through a panel buy written before the risks were visible.