The Yale Pharyngeal Residue Severity Rating Scale (YPRSRS) provides an image-based assessment of pharyngeal residue in the fiberoptic endoscopic evaluation of swallowing (FEES). Its performance was investigated only in FEES frames. This study analyzed the reliability and construct validity of the YPRSRS in FEES videos and the influence of bolus consistency. Thirty pairs of FEES videos and frames, 8 thin liquids (<50 mPa·s), 11 pureed (2583.3 mPa·s at 50 s−1, 697.87 mPa·s at 300 s−1), and 11 solid food; were assessed by 29 clinicians using the YPRSRS; 14 raters re-assessed materials at least 15 days from the first evaluation. Construct validity and intra-rater reliability were assessed using weighted Cohen’s Kappa. Inter-rater reliability was assessed using weighted Fleiss Kappa. Construct validity and inter-rater reliability were almost perfect or excellent for frames (0.82 ≤ k ≤ 0.89) and substantial or intermediate to good for videos (0.67 ≤ k ≤ 0.79). Intra-rater reliability was almost perfect for both frames and videos (k ≥ 0.84). Concerning bolus consistency, thin liquids had significantly lower values of construct validity, intra-, and inter-rater reliability than pureed and solid food. Construct validity and inter-rater reliability were significantly lower for solid food than for pureed food. The YPRSRS showed satisfactory reliability and construct validity also in FEES videos. Reliability was significantly influenced by bolus consistency.