Skip to content 
Photo by Lukas: https://www.pexels.com/photo/pie-graph-illustration-669621/
You can listen to the audio version of this article in the above video.
It is widely known that large language models (LLMs), the technology behind popular chatbots like ChatGPT, can be surprisingly unreliable. Even the most advanced LLMs have a tendency to misrepresent facts, often with unsettling confidence.
This unreliability becomes particularly dangerous when dealing with medical information, as people’s health could be at stake.
Researchers at New York University have discovered a disturbing vulnerability: adding even a tiny amount of deliberately false information (a mere 0.001%) to an LLM’s training data can cause the entire system to spread inaccuracies.
Their research, published in Nature Medicine and reported by Ars Technica, also revealed that these corrupted LLMs perform just as well on standard tests designed for medical LLMs as those trained on accurate data. This alarming finding suggests that current testing methods may not be sufficient to detect these serious risks.
The researchers emphasize the urgent need for improved data tracking and greater transparency in LLM development, especially within the healthcare sector, where misinformation can have life-threatening consequences for patients.
In one experiment, the researchers introduced AI-generated medical misinformation into “The Pile,” a commonly used LLM training dataset that includes reputable medical sources like PubMed. They were able to create 150,000 fabricated medical articles in just 24 hours, demonstrating how easily and cheaply these systems can be compromised. The researchers point out that malicious actors can effectively “poison” an LLM simply by disseminating false information online.
This research highlights significant dangers associated with using AI tools, particularly in healthcare. This is not a hypothetical problem; last year, the New York Times reported that MyChart, an AI platform used by doctors to respond to patient inquiries, frequently generated inaccurate information about patients’ medical conditions.
The unreliability of LLMs, especially in the medical field, is a serious and pressing concern. The researchers strongly advise AI developers and healthcare providers to acknowledge this vulnerability when developing medical LLMs. They caution against using these models for diagnosis or treatment until stronger safeguards are implemented and more thorough security research is conducted to ensure their reliability in critical healthcare settings.
The study found that by replacing just one million out of 100 billion training units (0.001%) with vaccine misinformation, they observed a 4.8% increase in harmful content generated by the LLM. This was achieved by adding approximately 2,000 fake articles (around 1,500 pages), which cost a mere $5 to generate.
Crucially, unlike traditional hacking attempts that target data theft or direct control of the AI, this “data poisoning” method does not require direct access to the model’s internal workings, making it a particularly insidious threat.