Researchers at Academia Sinica released a distilled Mandarin large language model that fits on a four-gigabyte GPU, letting rural township offices run citizen-service chatbots without sending queries to commercial cloud APIs.
The model, called Taigi-Edge-7B after internal testing names, answers questions about trash pickup, land registration hours, and disaster shelter locations using documents stored on local servers. Project leads said they removed layers and quantized weights so the software can run on refurbished desktops already deployed in Nantou and Pingtung pilot sites.
Why offline matters
Township clerks handle sensitive ID numbers and property records. Sending those prompts to overseas-hosted chatbots raises data-sovereignty concerns under Taiwan’s Personal Data Protection Act. An offline stack also keeps services running when fiber cuts follow typhoons, a recurring issue in mountain districts.
Accuracy trade-offs
Benchmarks shared with reporters show the smaller model scores within five points of the full-size teacher model on government FAQ tasks, but it struggles with rare legal citations. Engineers added a retrieval layer that pulls paragraphs from PDF ordinances before the model drafts an answer, reducing hallucinated office hours.
The National Science and Technology Council funded the work under a digital-government grant. The Digital Affairs Ministry said it will review whether the stack can plug into the national government service portal once security auditors finish penetration tests.
Rollout plan
Academia Sinica plans to open-source the weights and training recipe next quarter, alongside Traditional Chinese prompt templates for Hakka and indigenous-language glossaries. Commercial vendors may bundle the model with maintenance contracts, but scientists stressed that towns should keep human staff in the loop for anything involving fees or fines.
Training data boundaries
Scientists excluded social-media posts from the training mix, relying on gazetteers, disaster manuals, and anonymized service-desk transcripts. They said that choice reduced toxic outputs but also limits the model’s ability to answer slang-heavy questions from younger residents.
A red-team exercise in August found the chatbot refused to generate fake land-title documents, a common prompt-injection attempt during pilot testing.
Pilot metrics
In Nantou’s Yuchi Township, clerks said average call hold times dropped eleven minutes after the chatbot began answering trash-schedule questions, freeing staff for in-person land-title appointments. Pingtung’s Chunri Township logged a 22 percent reduction in misrouted forms when the model suggested the correct PDF before citizens printed the wrong template. Neither site connected the bot to payment systems, a deliberate choice after auditors flagged card-data risks on a prototype in 2025.
Training consumed roughly 1,200 GPU-hours on Academia Sinica’s internal cluster, a fraction of what commercial labs use for frontier models. Researchers plan to publish power-use figures so towns can budget electricity for overnight inference jobs.