Presented by: Subiksha Subramonian MBBChir, MA (Cantab), MRCS
Royal London Hospital

Introduction

In this paper, we aim to study the use of Chat GPT4(Openai.com) as a source of information for patients with urinary stone disease. Chat GPT uses machine learning from a vast dataset on the Internet, to generate responses to questions, by predicting what comes next given a certain prompt. Patients with stone disease frequently use the Internet/AI resources for their diagnosis and treatment. We aim to study the accuracy and detailedness of Chat GPT by asking common questions posed by stone patients to their treating Urologists and assessing the reliability of answers by cross checking with a panel of Endourologists.

Materials

The following questions were asked using Chat GPT, and the answers rated from 1-5 (Least to Most Accurate) and 1-5 (Least to Most Detailed) by a panel of Endourologists. The same questions were repeatedly posed to Chat GPT to check the variability of answers.

Three simple questions:

    1. I have pain in my back and I am passing blood in my urine. What is the likely diagnosis?

    2. I think I may have kidney stones. What test should I do to diagnose them?

    3. The CT scan showed a 5mm stone in my ureter. What is the best treatment?

Five more nuanced questions:

    4. What are my chances of passing a 5mm ureteric stone?

    5. How long can I wait for the stone to pass safely?

    6. I have a 5mm stone in my ureter and fever and shivers. What is the best treatment ?

    7. Should I avoid calcium supplements to prevent kidney stones?

    8. Should I have lithotripsy or ureteroscopy for my 5mm lower pole stone ?

Results

,

Average panel ratings  for the answers from Chat GPT for the above 8 questions is displayed in Chart 1, for their accuracy and detailedness. The generic 3 questions were generally answered well by Chat GPT scoring on average 3.4 out of 5 for accuracy and 3.87 out of 5 for detailedness. The more nuanced 5 questions were answered less well by Chat GPT, scoring an average of 2.6 out of 5 for accuracy and 3.11 out of 5 for detailedness. Chat GPT scored better overall on the ‘detailedness’ parameter than the ‘accuracy’ parameter across all of the questions.

Repeated questioning to Chat GPT yielded more succinct answers with no change in the accuracy or content of the answers.

Conclusion

For general queries from patients, Chat GPT is a reliable source of information. However, for more specific questions about stone management, Chat GPT was not a reliable source.

Funding

Nil

Lead Authors

Kesavapillai Subramonian,
Queen Elizabeth Hospital

Co-Authors

Yuko Smith,
Queen Elizabeth Hospital

Ivo Dukic,
Queen Elizabeth Hospital

Joe Philip,
Bristol Urological Institute

Sri Sriprasad,
Darent Valley Hospital