BACKGROUND: Digital health interventions have become increasingly popular in recent years, expanding the possibilities for treatment for various patient groups. In clinical research, while the design of the intervention receives close attention, challenges with research participant engagement and retention persist. This may be partially due to the use of digital health platforms, which may lack adequacy for participants. OBJECTIVE: This systematic literature review aims to investigate the relationship between digital health platforms and participant engagement and retention in clinical research. It will map and analyze key definitions of engagement and retention, as well as identify design characteristics that influence them. METHODS: We will carry out a mixed methods systematic literature review, analyzing qualitative and quantitative studies. The search strategy includes the electronic databases PubMed, IEEE Xplore, CINAHL, Scopus, Web of Science, APA PsycINFO, and the ACM Digital Library. The review will encompass studies published between January 2018 and June 2024. Criteria for inclusion will be the presence of digital health care interventions conducted through digital health platforms like websites, web and mobile apps used by patients, and informal caregivers as research participants. The main outcome will be a narrative analysis with key findings on the definitions of participant engagement and retention and critical factors that affect them. Quality assessment and appraisal will be done through the Mixed-Methods Assessment Tool. Data analysis and synthesis will follow the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 flow diagram. Quantitative data will be qualified and integrated into qualitative data, which will be analyzed using thematic analysis and synthesis. RESULTS: The study expects to map and summarize critical definitions of participant engagement and retention, and the characteristics of digital health platforms that influence them. The systematic review is expected to be completed in June 2025. CONCLUSIONS: This systematic review will contribute to the growing discussion on how the design of digital health intervention platforms can promote participant engagement and retention in clinical research. TRIAL REGISTRATION: PROSPERO CRD42024561650; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=561650. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): PRR1-10.2196/65099.