The following is a summary of “Accuracy of Information given by ChatGPT for patients with Inflammatory Bowel Disease in relation to ECCO Guidelines,” published in the March 2024 issue of Gastroenterology by Sciberras et al.
Researchers conducted a prospective study to assess the reliability of ChatGPT3.5 as an information source for patients with Inflammatory Bowel Disease (IBD).
They prepared 38 questions, which were asked by patients with IBD and fed into ChatGPT3.5. The questions span various topics: Crohn’s Disease (CD), Ulcerative Colitis (UC), malignancy, maternal medicine, infection and vaccination, and complementary medicine. Expert gastroenterologists assessed ChatGPT’s responses for accuracy (on a scale of 1 to 5) and completeness (on a 3-point Likert scale from 1 to 3), comparing them with ECCO guidelines.
The results showed that most responses (84.2%) achieved a median score of ≥4 (IQR:2) and a mean score of 3.87 (SD: +/- 0.6) for accuracy. Regarding completeness, 34.2% of responses attained a median score of 3, while 55.3% fell between 2 and <3. Overall, the mean rating was 2.24 (SD: +/- 0.4, Median:2 IQR:1). Groups 3 and 4 showed higher means for both accuracy and completeness, no significant scoring variation was observed among the four question groups (Kruskal-Wallis test P:>0.05). However, statistical analysis revealed significant differences in both accuracy (P<0.001) and completeness (P<0.001) across individual questions. Smoking questions scored highest; malignancy and vaccination queries, especially considering immunosuppression and family planning, scored lowest.
Investigators concluded that the AI system showed promise as a patient tool for IBD information, but improvements in response certainty were recommended for specialist areas.
Source: academic.oup.com/ecco-jcc/advance-article-abstract/doi/10.1093/ecco-jcc/jjae040/7634107