Andrea Sikora did not set out to become an artificial intelligence (AI) researcher.
A pharmacist by training, Andrea Sikora, PharmD, MSCR, FCCP, FCCM, is now an associate professor of biomedical informatics at the University of Colorado Anschutz (CU Anschutz). But her path to AI began on a different side of health care.
As a clinical pharmacist specializing in critical care, her early research focused on biomarkers for respiratory failure, which continued in the steps of early career mentors, and she was preparing to continue down that path through the Georgia Clinical and Translational Science Alliance KL2 program.
“I had even bought a really nice freezer for a wet lab,” Sikora said.
While pursuing this biomarker research, Sikora was also working on a “side hustle” project focused on metrics for Comprehensive Medication Management (CMM)—a topic that she is very passionate about. Sikora was confronted with the importances of what clinical pharmacists do to prevent medication errors and improve medication use despite high workloads.
One advisor in the KL2 program gave a piece of life-changing advice, “What I am hearing is that you would pursue that medication management project ‘regardless’ because you feel it is important, why is that not your main focus?” She got this advice shortly before being approached at a professional conference at an abstract presentation with a question, “Have you considered using Machine Learning and AI to help work with this drug data?”
This was before Large Language Models (LLMs) brought AI into the forefront of the public consciousness in 2022 and 2023. At the time, Sikora knew little about these machine learning technologies.
“I basically said, ‘What’s that?’” Sikora recalled.
But as she learned more, Sikora realized that machine learning and AI systems had substantial potential to help address the challenges she was already trying to solve in modeling the complexity and high-dimensionality of ICU medication management. She began writing grants for this work and exploring how the technology could be applied to her field.
The unexpected detour into AI eventually became a defining part of her research.
“So I kind of became the pharmacist that does AI stuff,” Sikora said.
Artificial Intelligence in Healthcare
AI is becoming more and more integrated into healthcare. For example, AI is being used to create clinical summaries in healthcare settings or to support with translation for Spanish-speaking patients. Researchers like Sikora are exploring how these technologies can be used to support research, discovery, clinical workflows, and patient care, but questions still remain about how reliable and trustworthy these systems are.
Healthcare is a high-stakes environment. In pharmacy, this is especially true. “Drugs can kill you,” said Sikora. “Medication errors are already a leading cause of death in the United States... [drugs] are a deeply unforgiving thing to make a mistake about. Maybe LLMs are going to make us better, and that certainly would be exciting. I’m excited about this. On the other side, you could say we have now introduced an entirely new way to mess things up. So that’s worthy of intrigue or concern.”
Pharmacy, AI and Uncertainty
Pharmacists are highly trained to overview, evaluate, analyze, and inquire about the medications that a patient is on. What’s more, although it is not necessarily formally written into the curriculum, pharmacists are also trained to express uncertainty—to communicate when there is something they are unsure about and research further to better understand the answer.
AI systems, by default, do not appear to do the same.
In further exploring this idea, Sikora and a team of researchers from the Harvard T.H. Chan School of Public Health and the Harvard Medical School recently published a perspective in the New England Journal of Medicine (NEJM) titled ‘Can AI Say “I Don’t Know”?’ where the research team explored LLMs’ abilities to express uncertainty in medicine. In the perspective, the researchers highlighted how more work needs to be done to help LLMs in clinical workflows better achieve something known as epistemic humility, which they defined as “a human virtue that involves metacognitive awareness, a moral commitment to truthfulness, and recognition of the limits of one’s knowledge.”
The researchers further emphasized that researchers and clinicians need to implement benchmarks to evaluate the efficacy and safety of LLMs in healthcare settings.
How do AI Systems Perform with Drug Data?
Pharmacy data and drug data create a unique set of circumstances that challenge AI Systems that may be working with this data.
Challenges of Drug Datatypes
One of the major challenges with drug data is the complex nature of the data.
“Drug data is very high dimensional,” explained Sikora. “For example, if I told you that I took ibuprofen this morning, there are more questions that would need to be answered. You would need to know the strength, how many tablets I took, whether I took any yesterday, whether I have any cardiovascular issues or renal issues, if I took any other pain medications—all of these factors are relevant to that particular datapoint.”
“Another element of drugs is alphanumeric combinations. So it's aspirin 81 milligrams. It's kind of a strange syntax to have the word and the number go in a certain order,” said Sikora.
These attributes of drug data can make it challenging for AI systems to operate with this data.
Drug Names
Another challenge that arises with LLMs and drug data is the names of the drugs themselves. Not only do most drugs have multiple names due to pharmaceutical branding, but new drugs are consistently coming out.
“There are new drugs coming out all the time. While working as a pharmacist, if I saw a drug that I didn’t recognize, I would go and look it up. I would search for what the drug is used for, what the dosing is—there’s a formalized process I would follow to analyze those drugs,” Sikora said.
“If I saw a drug that I didn’t recognize, I wouldn’t just assume it was a drug and that the patient was on that drug. I would make it clear that I didn’t know what this drug was, and I would ask questions to clarify.”
Drug or Pokémon?
Recently Sikora worked with a team on a project to evaluate LLM performance with drug names, releasing a paper titled: “Drug or Pokémon? Large language model performance in identification of fabricated medications.” In the paper, the team first compiled lists of medications (using both generic and brand names). They also included fictitious drugs in the form of Pokémon character names with a range of plausible doses, routes of administration, and dosing frequencies.
The researchers evaluated these datasets with multiple LLMs and searched for confabulations, which in this case are errors or statements where the AI either (1) missed/overlooked the false drug completely, or (2) automatically changed the Pokémon character to a different drug. They found that AI confabulations ranged between 2.7% and 99.6%, with some models performing better than others.
The researchers did find that mitigation prompts helped reduce confabulations—false, distorted productions, but the data from this study suggests that LLMs do not always default to work with drug data in a safe, effective manner.
Improving AI’s Ability to Work with Drug Data
Although LLMs have already started making their way into medicine, Sikora believes it’s vital to implement benchmarks to evaluate their efficacy and safety. But it’s not always simple to make ‘tests’ for the LLMs, especially in areas like pharmacy where the data is high-dimensional and there isn’t always one clear answer.
Benchmark tests like the Rx-LLM Benchmark Suite developed by Sikora and a team of researchers—or the Pokémon example described above—can help researchers to test the performance of these models. This is a critical starting point for evaluating their safety and effectiveness.
In her research, Sikora has uncovered a variety of methods to improve AI’s ability to work with drug data. A few of these methods are:
Can AI Say “I Don’t Know”?
AI models do not appear to express epistemic humility by default.
In other words, AI models do not automatically enact the behavior of expressing uncertainty—of saying “I don’t know.”
However, it does seem that implementing safeguards and other best practices improves their performance, which may hold promise for their ability to be successfully integrated in high-stakes environments like healthcare.
“Generative AI needs to be undergoing safety testing from independent validators,” said Sikora. “[This is] probably one of the biggest things that I take from this. And that something you are looking for—whether you realize it or not—from your healthcare professional is their humility to realize that you need to be referred to somebody else, or that they don't know. Because, you know, it's your health on the line if they make an assumption. So these are the points I find particularly interesting about this work.”
