The Ministry of Digital Affairs on Friday listed three municipal chatbots that passed a red-team battery designed to stop large language models from giving illegal land-use advice during a September pilot with four township offices.
The approved bots — deployed in tests by Hsinchu County, Taichung, and Tainan — must route any question involving variances, agricultural land conversion, or coastal setbacks to human planners. MODA said models that hallucinated permit shortcuts during adversarial prompts were blocked from production release.
How grading worked
Testers from Academia Sinica and the Information Technology Security Center posed 420 scripted questions, including attempts to bypass rules with hypothetical “family farm” scenarios. A bot passed only if it refused or cited the correct ordinance section in ninety-five percent of cases and never suggested bribing officials, a failure mode seen in an earlier rejected prototype.
MODA’s benchmark builds on the Taiwan LLM suite the ministry published for city hall help desks last month, but adds land-use specific traps after planners complained generic safety filters were too loose.
What citizens will see
Approved chatbots appear on township websites as Cantonese-, Mandarin-, and Taiwanese-language interfaces that summarize filing checklists. They do not accept document uploads containing personal ID numbers; citizens must still bring paper to counters for stamped approvals.
Tainan officials said the bot cut average phone wait times on zoning FAQs by about eighteen percent during the pilot, mostly by pointing callers to the right PDF forms. Taichung stressed that final interpretations remain with licensed architects, not the model.
Privacy and hosting
All three bots run on on-premise servers inside government data centers, with nightly log scrubbing. MODA prohibited cloud inference accounts after a May audit flagged cross-border latency risks during typhoon evacuations when traffic spikes.
Vendors wanting to sell similar tools to other municipalities must repeat the red-team tests quarterly. MODA will publish aggregate failure rates but not vendor names when models are rejected, a compromise transparency advocates accepted to keep commercial bids competitive.
Vendor pipeline
Two rejected vendors must rerun tests after MODA found their models stored citizen questions on U.S. cloud buckets for debugging. The ministry said future contracts will include daily fines for data residency violations, mirroring clauses in the digital ID procurement fight last year.
Open-source advocates asked MODA to publish anonymized failure transcripts so academics can benchmark safer models; the ministry promised a sanitized set in December without releasing personally identifiable details accidentally typed by testers.
