Presented by: Vishnuvardhan Ganesan MD
University of Minnesota

Introduction

The number of patient portal messages sent to providers continues to rise with increased uptake of electronic medical records. Using large language models such as GPT-4 to triage or answer messages could help improve timely responses and patient satisfaction. In this study we evaluated the use of ChatGPT on its ability to answer patient questions sent to an outpatient endourology clinic. 

Materials

We collected all messages sent by patients through the EPIC MyChart portal to two endourologists at our institution between March 2023 to April 2023. After removing identifying information and rewording the question, ChatGPT was queried for responses. The quality of responses was then independently assessed by three urologists. Each response was carefully evaluated and graded on a scale ranging from "Very Poor" to "Very Good" to determine the quality of information provided. Additionally, the urologists determined whether the AI-generated response was helpful or not, categorizing it as either "yes" or "no."

Results

,

97 questions from patients were analyzed. When the quality of the information in the responses was assessed, they were good or very good in only 20% of responses. This was better for responses to patient updates (53%) and post-op questions (46%), Figure 1.  Responses were rated as “not helpful” for majority of questions (76%). Chat GPT included disclaimers in 25% (25/97) of the responses. It responded that it is not able to answer the question because it is “not authorized”, or it “cannot provide medical advice”, or it “doesn’t have access” to medical records or medical history. In 68% (66/97) of responses ChatGPT asks for the user to consult with a healthcare provider. In 4% (4/97) of responses ChatGPT suggested going to emergency room.

Conclusion

AI generated responses for patient questions were rated as "not helpful" by the assessors 76% of the time but appear to be better for subset of patient update and post-op questions. While not ready for clinical use in its current form, we feel with integration with clinical context or medical and surgical history there is significant potential for AI as a triage system.

Funding

None

Lead Authors

Hailey Fyre, BS
University of Minnesota

Co-Authors

Deepak Agarwal, MD
University of Minnesota

Michael Borofsky, MD
University of Minnesota