C9ORF72 hexanucleotide repeat expansion is the most common genetic cause of both amyotrophic lateral sclerosis (ALS) and frontotemporal dementia (FTD). One pathogenic mechanism is the accumulation of toxic dipeptide repeat (DPR) proteins like poly-GA, GP and GR, produced by the noncanonical translation of the expanded RNA repeats. However, how different DPRs are synthesized remains elusive. Here, we use single-molecule imaging techniques to directly measure the translation dynamics of different DPRs. Besides initiation, translation elongation rates vary drastically between different frames, with GP slower than GA and GR the slowest. We directly visualize frameshift events using a two-color single-molecule translation assay. The repeat expansion enhances frameshifting, but the overall frequency is low. There is a higher chance of GR-to-GA shift than in the reversed direction. Finally, the ribosome-associated protein quality control (RQC) factors ZNF598 and Pelota modulate the translation dynamics, and the repeat RNA sequence is important for invoking the RQC pathway. This study reveals that multiple translation steps modulate the final DPR production. Understanding repeat RNA translation is critically important to decipher the DPR-mediated pathogenesis and identify potential therapeutic targets in C9ORF72-ALS/FTD.