Competency-based medical education (CBME) has been the subject of heated debate since its inception in medical education. Despite the many challenges and pitfalls of CBME that have been recognized by the medical education community, CBME is now seeing widespread implementation. However, the biggest problems with CBME still have not been solved. Two of these problems, reductionism and loss of authenticity, present major challenges when developing curricula and assessment tools.The authors address these problems by making a call for flexibility in competency definitions and for the use of mixed methods in CBME. First, they present the issue of reductionism and a similar concept from the field of data science, overfitting. Then they outline several solutions, both conceptual and concrete, to prevent undue reductionist tendencies in both competency definitions and in tools of assessment. Finally, they propose the reintroduction of qualitative methods to balance the historically quantitative emphasis of assessment in medical education.The authors maintain that mixed-methods assessment with multiple assessors in differing contexts can yield a more accurate representation of a medical trainee's skills and abilities, deter the loss of authenticity, and increase the willingness of medical educators to adopt a feasible form of CBME. Finally, they propose the deployment of dedicated faculty assessors and physician coaches (which will reduce training requirements for other faculty), as well as the use of formal qualitative tools of assessment alongside established quantitative tools, to encourage a truly mixed-methods approach to assessment.