BACKGROUND: Randomized controlled trials are considered the gold standard for evidence on therapeutic interventions; however, they are susceptible to bias. The objectives of this observational study were to describe the methodological quality of neonatal randomized controlled trials and quantify the bias related to specific methodological and study-level characteristics. METHODS: Twenty-five systematic reviews yielding 208 neonatal trials were included. Two independent reviewers assessed risk of bias (RoB) on seven domains consisting of nine items. For each domain, meta-analyses with at least one high/unclear and one low risk study were included in the analysis. For the primary outcome within each meta-analysis a ratio of odds ratios with a 95% confidence interval was generated. The ratio of odds ratios for each meta-analysis were combined using meta-analytic techniques with inverse-variance weighting and a random effects model to obtain a summary ratio of odds ratio. RESULTS: None of the studies had an overall low RoB. Most studies had a low RoB for the domain of incomplete outcome data (89%), while 63%, 55% and 46% of trials had low RoB for sequence generation, other sources of bias, and blinding of outcome assessors, respectively. For all other domains (allocation concealment, blinding of parents and investigators and selective outcome reporting), the majority of trials were assessed as unclear. Selective outcome reporting was rated as unclear RoB for 55% and high for 42% of studies. The only domain that showed a statistically significant association with the treatment effect was selective outcome reporting: trials at unclear/high risk of bias for this domain significantly overestimated the treatment effects compared with those assessed at low risk of bias (ROR = 1.87, 95% confidence interval: 1.26-2.78). CONCLUSIONS: This observational study of a sample of neonatal trials showed that most were at high risk of bias, indicating that there is room for improvement in the design, conduct and reporting of neonatal trials to ensure valid results for the most clinically important outcomes. We did not find an association between most risk of bias domains and effect estimates; however, we found that randomized controlled trials at high risk for selective outcome reporting were associated with overestimates of treatment benefits. These results need to be confirmed in larger samples.