Digital self-help can improve access to mental health care, but poor engagement limits effectiveness. This single-blind 2 x 2 x 2 factorial randomized controlled trial examined whether an optimized graphical user interface (GUI), automated reminders (AR), and an adaptive treatment strategy (ATS) improved engagement and outcomes in a digital self-help insomnia intervention. Adults (N = 447) with moderate to severe insomnia were randomized to combinations of the factors. The GUI improved self-rated engagement, sleep log activity, login frequency, and usability. AR increased sleep log activity and logins, while ATS improved satisfaction. All three combined significantly improved insomnia symptoms (d = 0.50). No severe adverse effects were reported. Clinicians spent 13.74 min on average on the ATS. Statistical analyses included linear and multilevel regression. Factors were effect coded. Intervention design can enhance engagement and outcomes, requiring minimal clinician time. Pre-registered 2023-04-11 (ClinicalTrials.gov, NCT05826002). Funded by the Swedish Ministry of Health and Social Affairs, grant number: S2018/03855/FS.