The Ministry of Digital Affairs published a Taiwan-focused large language model benchmark Friday, giving six city governments a shared test bed to compare chatbot vendors before help-desk pilots go live in December.
The suite includes 420 Traditional Chinese question-and-answer pairs drawn from published FAQs on household registration, parking fines, and recycling schedules, plus adversarial prompts designed to elicit hallucinated fees or fake hotline numbers. MODA will score responses on factual accuracy, citation of official URLs, latency under 3 seconds, and appropriate refusals when users ask for legal advice.
Who is in the pilot
Taipei, Taoyuan, Taichung, Tainan, Kaohsiung, and Hsinchu City signed memoranda to run the benchmark on their procurement shortlists, according to MODA’s press briefing. Each city may nominate up to three vendor stacks, mixing local startups with international cloud models hosted in Taiwan data centers to satisfy data residency rules announced earlier this year.
Vendors must log model version hashes and prompt templates so auditors can reproduce scores. MODA said it will not rank companies publicly; instead, cities receive confidential scorecards to attach to tender documents. That approach mirrors how the National Institute of Cyber Security evaluates encryption modules without publishing proprietary source code.
Why a local benchmark matters
Global English benchmarks poorly capture Taiwan’s mixed register of formal bureaucratic Chinese and colloquial Hokkien loanwords citizens use in 1999 hotline transcripts, MODA officials argued. The ministry hired linguists from Academia Sinica to tag dialect-heavy queries separately, ensuring models are not penalized for asking clarifying questions when users mix languages.
Privacy rules forbid uploading real call-center audio; every task is synthetic or redacted. Cities must still run penetration tests on webhook endpoints before connecting live databases, a step MODA’s checklist treats as pass-fail regardless of benchmark score.
Timeline and limits
Benchmark submissions close November 22, with MODA publishing aggregate error categories—such as wrong fee amounts or outdated office hours—without naming losers. Live pilots start December 8 on Taipei City’s 1999 portal and expand to other municipalities in January if uptime holds above 99.5 percent during business hours.
The ministry cautioned that chatbots are triage tools, not replacements for civil servants handling complex cases like land expropriation appeals. Each pilot interface must display a button connecting to human agents during office hours, a requirement MODA said it will enforce through spot checks rather than automated scoring alone, with fines for cities that hide the handoff path.
