CSIRO has stood up a Canberra-based red-team laboratory to adversarially test generative AI products before they are cleared for agency procurement, according to officials briefed on the program and testimony the science agency has filed with federal committees. The lab extends existing Public Governance, Performance and Accountability Act vendor assurance with prompt-injection suites, data-exfiltration scenarios, and supply-chain reviews tailored to large language models sold into government and research settings.
Why procurement needed a dedicated red team
CSIRO told Parliament’s joint committee on public accounts that enterprise use of generative AI remains limited—mostly meeting transcription and summarisation—while researchers access models through Azure OpenAI and AWS Bedrock test beds under central cloud controls. Any new technology still must pass cyber, contract, and governance review; the Responsible AI Working Group monitors emerging risks, including dependence on a handful of global model providers.
Procurement officers said vendor questionnaires rarely capture how fine-tuned models behave when users paste classified snippets or when plugins call external URLs. The red-team lab, housed near CSIRO’s Black Mountain campus, runs structured attacks against candidate systems before panel onboarding, complementing—not replacing—traditional penetration tests focused on network perimeters.
What testers actually do
Teams drawn from CSIRO’s cybersecurity and data sciences branches adapt methods from the Talos research program, which studies poisoning, backdoors, and adversarial examples in machine learning used for threat detection. For procurement, exercises include jailbreak attempts on chat interfaces, tests whether models leak training data when probed with crafted prompts, and reviews of whether logging meets record-keeping policies when staff paste literature or code.
Vendors invited to early pilots signed non-disclosure agreements and received remediation windows before results inform panel recommendations. CSIRO emphasised it is not certifying models as “safe for all science”—only documenting behaviour under defined scenarios so budget holders can accept or reject residual risk.
Link to broader ICT spending
Separately, CSIRO is refreshing a multi-year ICT infrastructure panel covering hardware, software, and cyber appliances across more than 60 sites. That tender streamlines commodity buying; the red-team lab sits upstream of any generative AI components embedded in collaboration suites or analytics platforms shortlisted on the panel. Industry briefings for the infrastructure RFT stressed sustainability and OEM partnerships, but cyber specialists in the September sessions asked how AI features inside unified communications tools would be evaluated.
The agency’s answer, according to attendees, is that any product claiming embedded copilots must pass the Canberra lab or remain off-limits for PROTECTED-adjacent workloads until it does. That stance aligns with Digital Transformation Agency guidance on responsible AI in government, which CSIRO cited in its committee submissions.
Government customers watching
Other Commonwealth entities have asked whether CSIRO will offer red-team as a fee-for-service once internal queues clear. CSIRO has not announced a commercial schedule; for now, priority goes to its own enterprise adoption decisions and to joint trials with departments participating in the Microsoft 365 Copilot evaluation CSIRO extended to better understand benefits and risks.
Researchers cautioned that red-teaming generative models is non-deterministic: a pass this month does not guarantee behaviour after the vendor silently updates weights. The lab therefore recommends contractual notice periods for model changes and continuous log monitoring—controls CSIRO already applies to its Bedrock and Azure test beds with cost caps and access reviews.
Limits and politics
The program does not resolve sovereignty debates about where models are trained or hosted; it focuses on operational security and data handling at the interface agencies control. Ministers have been briefed that Australia’s national science agency is effectively building domestic capacity to question AI vendors rather than accepting marketing decks at face value—a political selling point as scams and deepfakes dominate consumer headlines.
Opposition spokespeople have asked whether red-team capacity should sit inside the Australian Cyber Security Centre instead of CSIRO; the agency argues science users need domain-specific attack libraries for genomics, climate, and manufacturing data, not only generic OWASP prompt lists.
What vendors should expect
Suppliers pitching generative AI into CSIRO or partner agencies should budget time for a two- to four-week lab cycle, including fix-and-retest. Failures do not automatically ban vendors nationally, but they propagate to procurement notes shared across Commonwealth science and industry portfolios. Success buys a faster path onto panels—provided contract teams still negotiate liability, indemnity, and exit clauses the PGPA Act demands.
For Canberra’s policy community, the lab is a tangible signal that “responsible AI” is more than a working-group slide: someone with a payroll at Black Mountain is paid to break these systems on purpose before taxpayers do so accidentally.
