BACKGROUND: Patients and healthcare professionals extensively rely on the internet for medical information. Low-quality videos can significantly impact the patient-doctor relationship, potentially affecting consultation efficiency and decision-making process. Chat Generative Pre-Trained Transformer (ChatGPT) is an artificial intelligence application with the potential to improve medical reports, provide medical information, and supplement orthopedic knowledge acquisition. This study aimed to assess the ability of ChatGPT-4 to detect deficiencies in these videos, assuming it would be successful in identifying such deficiencies. MATERIALS AND METHODS: YouTube was searched for "rotator cuff surgery" and "rotator cuff surgery clinic" videos. A total of 90 videos were evaluated, with 40 included in the study after exclusions. Using the Google Chrome extension ''YouTube Summary with ChatGPT & Claude,'' transcripts of these videos were accessed. Two senior orthopedic surgeons and ChatGPT-4 evaluated the videos using the rotator cuff surgery YouTube score (RCSS) system and DISCERN criteria. RESULTS: ChatGPT-4's RCSS evaluations were comparable to those of the observers in 25% of instances, and 40% for DISCERN. The interobserver agreement between human observers and ChatGPT-4 was fair (AC1: 0.575 for DISCERN and AC1: 0.516 for RCSS). Even after correcting ChatGPT-4's incorrect answers, the agreement did not change significantly. ChatGPT-4 tended to give higher scores than the observers, particularly in sections related to anatomy, surgical technique, and indications for surgery. CONCLUSION: The use of ChatGPT-4 as an observer in evaluating rotator cuff surgery-related videos and identifying deficiencies is not currently recommended. Future studies with trained ChatGPT models may address these deficiencies and enable ChatGPT to evaluate videos at a human observer level.