Academia Sinica researchers secured a National Science and Technology Council grant to shrink Mandarin large language models so rural township offices can host citizen-service chatbots on single-GPU servers, keeping household registration and disaster alerts on prem rather than routing prompts through overseas hyperscale clouds.

Problem the project targets

Taiwan’s Digital Affairs Ministry has encouraged local governments to experiment with generative assistants for FAQs—trash pickup schedules, shelter locations, tax payment windows—but many townships lack fiber backhaul resilient enough to rely on always-on API calls. Privacy officers also worry about uploading dialect-heavy queries that include names and addresses.

The Sinica team proposes quantization and knowledge-distillation pipelines tuned for Taiwanese Mandarin and common Minnan loanwords, producing models under four billion parameters that still answer structured policy questions accurately. Pilot partners include offices in Miaoli and Pingtung counties.

Technical approach

Instead of training from scratch, engineers start from open-weights bases and prune layers using a curriculum of government-published Q&A pairs, court bulletins, and Central Weather Administration templates. Human reviewers in each township label failure modes—hallucinated office hours, wrong fee amounts—feeding a weekly retraining loop.

Hardware targets are modest: a single NVIDIA L40S or AMD MI210 class card in an air-conditioned closet. Offline inference also means chatbots keep working when typhoons knock out uplinks, a requirement disaster planners emphasized after 2024 outages.

Governance and MODA alignment

The Digital Affairs Ministry’s draft AI labeling rules expect synthetic replies to disclose machine generation; the Sinica stack bakes those disclaimers into template responses. MODA officials said compressed models could qualify for a “sovereign inference” procurement category if audit logs stay local.

Limits and risks

Compressed models will not draft legislation or interpret ambiguous legal disputes. Researchers warn that without continuous evaluation, small models may memorize outdated fee tables. The grant funds third-party red-team tests for prompt injection and data exfiltration attempts.

Timeline

Prototype chatbots should deploy in two townships by March 2027, with open-source tooling released for other counties that share the same data formats. If successful, the project offers a middle path between banning generative AI in government and outsourcing cognition to foreign APIs—a debate that has split city councils since ChatGPT-style tools went mainstream.

For citizens, the near-term benefit is mundane: faster answers on trash tags and shelter maps when the internet flickers. For Taiwan’s AI policy, it is a test of whether public research can keep language models small enough to fit democratic oversight and island-scale infrastructure.

Budget and staffing

The three-year grant covers four postdoctoral engineers and six township liaisons who will embed with clerks to document real queries rather than synthetic benchmarks. Sinica’s data governance board must approve each county’s training corpus, a step that slowed earlier open-data releases but now provides a paper trail auditors can follow if a model leaks personal information.

Commercial cloud vendors lobbied for hybrid architectures; researchers rejected always-online fallbacks, arguing they would recreate the dependency the project is meant to eliminate. Procurement law may need tweaks before townships can buy GPU boxes without federal tender thresholds—a side fight MODA lawyers are watching.

Benchmarks and open release

Researchers will publish a public leaderboard scoring models on township Q&A accuracy, latency on a single GPU, and refusal rates for illegal advice requests. Open weights will exclude raw citizen tickets to protect privacy, but distilled datasets with redacted names will ship for academics replicating the work.