BACKGROUND & AIMS: The treatment of advanced gastric cancer (GC) requires precise and comprehensive clinical decision-making. Artificial intelligence (AI) chatbots offer potential tools to enhance multidisciplinary team (MDT) discussions. This study aims to compare the performances of ChatGPT-4o and Gemini Advanced in generating treatment recommendations for advanced GC.

METHODS

The study involved three steps: (1) evaluating responses to ten critical clinical questions, (2) analyzing clinical cases from MDT meetings at our institution, and (3) reviewing rare GC cases from PubMed. It included 95 advanced GC patients discussed between November 2022 and July 2024, and 14 rare cases from PubMed. Prompts designed from advanced GC cases were submitted to ChatGPT-4o and Gemini Advanced using a standardized format. Outputs were evaluated for accuracy and completeness using a structured 4-point Likert scale. Interrater reliability was calculated to ensure consistency among evaluators.

RESULTS

For the ten clinical questions, ChatGPT-4o achieved better performances compared to Gemini Advanced. In MDT cases, ChatGPT-4o provided more valuable recommendations in surgical suggestion, chemotherapy recommendation, and chemotherapy regimens. Subgroup analysis confirmed these findings in both routine and complex cases with high interrater reliability. ChatGPT-4o also outperformed Gemini Advanced in the analysis of rare GC cases from PubMed, showing superior accuracy with high interrater reliability.

CONCLUSIONS

While our findings suggest that AI chatbots can generate clinically relevant and guideline-based treatment recommendations, their use in MDT decision-making should be viewed as supportive rather than autonomous. We emphasize that while AI chatbots have potential as decision-support tools, but they should be integrated only under expert supervision in a real-world clinical context.