A hospital AI ethics panel convened by the Ministry of Health and Welfare said Friday that two Seoul pilot chatbots invented drug-interaction warnings in discharge summaries, a failure mode regulators are now cataloging before wider rollout of generative tools in inpatient wards. The panel’s red-team report covers tests run in September at a tertiary hospital in Mapo and a smaller acute-care center in Nowon, both using vendor-built assistants layered on top of electronic medical record exports.
Reviewers compared chatbot output against pharmacist-verified medication lists. In three of eleven flagged cases, the bot cited interaction pairs that included drugs absent from the patient’s active orders. Two additional warnings mixed correct drug names with incorrect severity levels, potentially alarming family members reading automated Korean-language summaries on patient portals.
What the pilots were supposed to do
The projects aimed to translate clinician notes into plain-language discharge instructions within ten minutes of ward checkout. Nurses triggered the bot through a tablet workflow; pharmacists were optional in the pilot design—a choice the ethics panel now says was a mistake. Vendors argued that large language models reduce readmission risk by improving comprehension; skeptics counter that hallucinated contraindications could send patients back to emergency rooms unnecessarily.
The ministry asked hospitals to keep generative tools on “draft” status until a licensed pharmacist clicks approve. It also recommended logging prompt versions and retrieval corpora so auditors can reconstruct why a bogus interaction appeared.
Industry response
Domestic EMR vendors told InfoHandle Network they will ship rule-based guardrails that cross-check bot claims against structured medication tables before text reaches patients. One startup proposed a dual-model setup: a smaller model extracts entities, and the generative layer may only paraphrase verified facts. Cloud providers hosting the pilots offered additional GPU hours for offline evaluation, but hospital IT chiefs said staffing—not compute—is the bottleneck.
Policy timing
The findings land as enterprise AI teams file compliance memos under Korea’s Basic AI Act grace year. Health is treated as a high-impact domain, meaning hospitals may face earlier disclosure duties than retailers experimenting with shopping assistants. For now, the panel’s prescription is conservative: keep humans in the pharmacy loop, and treat any bot-written warning as suspect until a name-signed pharmacist agrees.
Training data boundaries
Vendors told the panel they fine-tuned models on anonymized discharge letters dating back six years—a corpus rich in abbreviations that confuse entity extractors. The ethics group recommended banning free-text medication lists from training sets unless pharmacists validate token boundaries. Hospitals agreed to pilot “structured-first” workflows where bots may only read coded order tables, not clinician prose.
Patient advocates asked for a right to opt out of generative summaries entirely, receiving a promise of paper pamphlets for the remainder of 2026. Insurers watching readmission rates will compare hospitals that paused bots with those that accelerated adoption, a natural experiment regulators say they will monitor before setting reimbursement rules.