OBJECTIVES: Real-world evidence (RWE) plays an increasingly important role in health technology assessment (HTA), as well as regulatory and clinical decision making. RWE studies, however, are subject to multiple sources of bias, which are often not easy to identify, impeding credibility and inclusion of RWE in decision-making. A comprehensive, fit-for-purpose, and easy-to-use bias assessment tool would help streamline RWE evaluation, enabling efficient utilization of RWE for decision-making globally. METHODS: A working group of the International Society for Pharmacoepidemiology (ISPE) collaborated with HTA experts to develop a tool that could guide bias assessment in observational studies on the comparative safety and effectiveness of medications, building upon existing methodological tools, best practice guidelines, and checklists for the analysis of real-world data. The tool was further tested and refined in collaboration with HTA agencies. RESULTS: APPRAISE (APpraisal of Potential for bias in ReAl world evIdence StudiEs) covers key domains through which bias might be introduced into an RWE study: inappropriate study design and analysis, exposure and outcome misclassification, and confounding. Each domain contains a series of questions. Responses to questions auto-populate a summary of the potential for bias within each domain and of the actions to take to avoid, mitigate, or explore the impact of bias. CONCLUSIONS: APPRAISE is a tool to guide bias assessment in observational studies on medication comparative effectiveness or safety. While the tool was designed for HTA, it will be useful for many other users of RWE and will help guide optimized RWE generation.