The actual distribution of radiation dose accumulated in normal tissues over the complete course of radiation therapy is, in general, poorly quantified. Differences in the patient anatomy between planning and treatment can occur gradually (e.g., tumor regression, resolution of edema) or relatively rapidly (e.g., bladder filling, breathing motion) and these undermine the accuracy of the planned dose distribution. Current efforts to maximize the therapeutic ratio require models that relate the true accumulated dose to clinical outcome. The needed accuracy can only be achieved through the development of robust methods that track the accumulation of dose within the various tissues in the body. Specific needs include the development of segmentation methods, tissue-mapping algorithms, uncertainty estimation, optimal schedules for image-based monitoring, and the development of informatics tools to support subsequent analysis. These developments will not only improve radiation outcomes modeling but will address the technical demands of the adaptive radiotherapy paradigm. The next 5 years need to see academia and industry bring these tools into the hands of the clinician and the clinical scientist.