Healthcare giant Optum has restricted access to an internal AI chatbot after a security researcher discovered it was publicly accessible online. The chatbot, which employees used to ask questions about claims and standard operating procedures (SOPs), was accessible without a password through a public IP address.
Internal Chatbot Exposed
The chatbot, dubbed "SOP Chatbot," was intended to help employees navigate patient health insurance claims and disputes. While the chatbot did not directly contain sensitive patient data, its exposure comes at a time when its parent company, UnitedHealthcare, faces criticism for using AI to deny patient claims.
Mossab Hussein, a security researcher from spiderSilk, alerted TechCrunch to the vulnerability. The tool, while hosted on an internal Optum domain, was accessible through its public IP address without requiring authentication.
Chatbot Usage and Data
According to a dashboard on the chatbot, Optum employees had used the tool hundreds of times since September. The chatbot stored conversation history including queries like, "What should be the determination of the claim," and, "How do I check policy renewal date." It also provided reasons for denial of coverage, citing internal documents related to dispute processes and eligibility screening.
It was also capable of answering queries outside of its training data, although failed at a "joke about cats." Additionally, there were several attempts by employees to "jailbreak" the bot.
“In the realm of healthcare’s grand domain Where policies and rules often constrain A claim arrives, seeking its due But alas, its fate is to bid adieu.”
When asked to write a poem about denying a claim, the bot generated the above seven paragraph response.
Optum's Response
Optum claims the chatbot "was a demo tool developed as a potential proof of concept" and was never deployed in production. The company confirmed the bot and its training data contained no protected health information. They stated that the bot only "enable[d] better access to existing SOPs" and did not make any decisions.
UnitedHealthcare Scrutiny
UnitedHealthcare faces growing criticism and legal action for its use of AI to deny patient claims. The company has been accused of using faulty AI models instead of medical professionals to deny critical healthcare to patients. They are the target of a federal lawsuit, and have come under further criticism since the death of its chief executive.
Despite these criticisms, UnitedHealth Group made $22 billion in profit on revenues of $371 billion in 2023.
