(L to R) Co-authors Kiri Ness, PT, PhD, FAPTA, and Melissa Hudson, MD, with corresponding author I-Chan Huang, PhD, all of the St. Jude Department of Epidemiology & Cancer Control.
Artificial intelligence (AI) could help physicians determine if survivors of childhood cancer need extra support — and the more information included in AI prompting, the better its performance. This finding, published today in Communications Medicine by scientists from St. Jude Children’s Research Hospital, may guide future integration of AI into clinical workflows.
The scientists observed how well large language models, a type of AI, could analyze interviews with young survivors and their caregivers to detect multiple symptoms causing severe disruptions in their daily lives. By comparing different prompting approaches, the researchers found that more complex prompts, which provided additional information to the models, performed the best. The results suggest that future efforts to leverage AI to improve survivors’ care should consider these sophisticated prompting strategies over simpler ones.
“About 40%-60% of a clinical encounter is a patient talking to their physician about symptoms and related health experiences,” said corresponding author I-Chan Huang, PhD, St. Jude Department of Epidemiology & Cancer Control. “We have provided a proof of concept that large language models could help analyze that underutilized conversational data to detect symptom severity and its functional impact and assist physician decision-making to provide better care to survivors.”
Comparing prompting strategies for survivorship
Children who have cancer are treated during a critical time in their development, which can have a ripple effect later in life. Cancer- and treatment-related effects can occur long after the initial disease is cured. However, identifying which survivors have symptoms severe enough to need extra, targeted support is difficult for physicians. Much of the data that informs that identification exists in transcripts of conversations and answers to open-ended questions in surveys that cannot be reviewed quickly. Newer language-based AI offers physicians an opportunity to analyze, understand and use that information to help survivors.
The researchers interviewed 30 survivors between 8 and 17 years old and their caregivers. Two human experts analyzed the conversation transcripts for signs of excessive pain and fatigue, yielding more than 800 analyzable pieces of information. They categorized the symptoms by severity and their physical, cognitive or social impact. After performing that gold-standard analysis, the scientists gave the same transcripts to two large language models, ChatGPT and Llama, using four styles of prompts. Both models showed an ability to analyze the data in a similar way to the experts, though their performance depended on the prompt used.
Prompting is the art of instructing AI to perform a task. The researchers compared four common prompting strategies: two simple and two complex. The simple approaches were zero-shot and few-shot prompting, which provide no or minimal information beyond the basic instructions. These approaches produced unstable and inaccurate results.
“We found that simple prompts were not effective,” Huang said. “However, our more sophisticated prompting strategies performed significantly better and had a higher concurrence with our human reviewers.”
The two complex strategies were chain-of-thought and generated knowledge prompts. Chain-of-thought uses step-by-step logical instructions, while generated knowledge prompts a model to come up with background information before giving it the instructions. Both complex prompting methods distinguished the physical and cognitive impact of symptoms on survivors well, while having a moderate ability to detect social impacts.
Clinical use will require much more testing, but these early results suggest that chain-of-thought, generated knowledge or similar prompting methods should be used in the future. The findings provide one of the first concrete examples of how AI may be able to improve survivorship care.
“These AI-driven approaches provide us with a new way to unlock the complex symptom information hidden in the wealth of patient-physician conversations that currently go unused,” Huang said. “By making this information easier to capture and analyze, we can help physicians better identify survivors who need additional support in real time and improve care for this growing population.”
Authors and funding
The study’s first author is Jin-ah Sim, formerly of St. Jude. The study’s other authors are Madeline Horan, formerly of St. Jude, now Wake Forest University School of Medicine; Xiaolei Huang, University of Memphis; Minsu Kim, Hallym University; Kumar Srivastava, Kirsten Ness and Melissa Hudson, St. Jude; and Justin Baker, formerly of St. Jude, now Stanford University Medical School.
The study was supported by grants from the National Cancer Institute (U01CA195547, R21CA202210, R01CA238368 and R01CA258193), Cancer Center Support (CORE) grant (CA21765) and the American Lebanese Syrian Associated Charities (ALSAC), the fundraising and awareness organization of St. Jude.
St. Jude Children's Research Hospital
St. Jude Children’s Research Hospital is leading the way the world understands, treats, and cures childhood catastrophic diseases. As the only National Cancer Institute-designated Comprehensive Cancer Center devoted solely to children, St. Jude advances groundbreaking research and shares its discoveries worldwide to accelerate progress in pediatric medicine. Treatments developed at St. Jude have helped push the overall childhood cancer survival rate from 20% to more than 80% since the hospital opened more than 60 years ago. Through collaboration and innovation, St. Jude is working to ensure that children everywhere have access to the best possible care. To learn more, visit stjude.org, read St. Jude Progress, a digital magazine, and follow St. Jude on social media at @stjuderesearch.