Peter L Reichertz Institute for Medical Informatics, TU Braunschweig and Hannover Medical School, Hannover, Germany.
Department of Digital Medicine, Medical Faculty OWL, Bielefeld University, Bielefeld, Germany.
JMIR Mhealth Uhealth. 2022 Jan 11;10(1):e26563. doi: 10.2196/26563.
Health apps are often used without adequately taking aspects related to their quality under consideration. This may partially be due to inadequate awareness about necessary criteria and how to prioritize them when evaluating an app.
The aim of this study was to introduce a method for prioritizing quality attributes in the mobile health context. To this end, physicians were asked about their assessment of nine app quality principles relevant in health contexts and their responses were used as a basis for designing a method for app prioritization. Ultimately, the goal was to aid in making better use of limited resources (eg, time) by assisting with the decision as to the specific quality principles that deserve priority in everyday medical practice and those that can be given lower priority, even in cases where the overall principles are rated similarly.
A total of 9503 members of two German professional societies in the field of orthopedics were invited by email to participate in an anonymous online survey over a 1-month period. Participants were asked to rate a set of nine app quality principles using a Kano survey with functional and dysfunctional (ie, positively and negatively worded) questions. The evaluation was based on the work of Kano (baseline), supplemented by a self-designed approach.
Among the 9503 invited members, 382 completed relevant parts of the survey (return rate of 4.02%). These participants were equally and randomly assigned to two groups (test group and validation group, n=191 each). Demographic characteristics did not significantly differ between groups (all P>.05). Participants were predominantly male (328/382, 85.9%) and older than 40 years (290/382, 75.9%). Given similar ratings, common evaluation strategies for Kano surveys did not allow for conclusive prioritization of the principles, and the same was true when using the more elaborate approach of satisfaction and dissatisfaction indices following the work of Timko. Therefore, an extended, so-called "in-line-of-sight" method was developed and applied for this evaluation. Modified from the Timko method, this approach is based on a "point of view" (POV) metric, which generates a ranking coefficient. Although the principles were previously almost exclusively rated as must-be (with the exception of resource efficiency), which was not conducive to their prioritization, the new method applied from the must-be POV resulted in identical rankings for the test and validation groups: (1) legal conformity, (2) content validity, (3) risk adequacy, (4) practicality, (5) ethical soundness, (6) usability, (7) transparency, (8) technical adequacy, and (9) resource efficiency.
Established survey methodologies based on the work of Kano predominantly seek to categorize the attributes to be evaluated. The methodology presented here is an interesting option for prioritization, and enables focusing on the most important criteria, thus saving valuable time when reviewing apps for use in the medical field, even with otherwise largely similar categorization results. The extent to which this approach is applicable beyond the scenario presented herein requires further investigation.
健康类应用在使用时往往没有充分考虑到与其质量相关的方面。这可能部分是由于对评估应用程序时所需的必要标准以及如何优先考虑这些标准缺乏足够的认识。
本研究旨在引入一种在移动健康环境中对质量属性进行优先级排序的方法。为此，我们询问了医生对 9 个与健康相关的应用程序质量原则的评估，并将他们的回答用作设计应用程序优先级排序方法的基础。最终，我们的目标是通过帮助确定在日常医疗实践中应优先考虑的具体质量原则，以及可给予较低优先级的原则，从而更好地利用有限的资源（例如时间），即使在总体原则评分相似的情况下也是如此。
我们通过电子邮件邀请了两个德国骨科专业协会的 9503 名成员参加为期 1 个月的匿名在线调查。参与者被要求使用 Kano 调查（使用正向和负向问题）对一组 9 个应用程序质量原则进行评分。评估基于 Kano 的工作（基准），并辅以自行设计的方法。
在 9503 名受邀成员中，有 382 名完成了相关调查（回复率为 4.02%）。这些参与者被平均随机分配到两个组（实验组和验证组，每组 191 名）。两组的人口统计学特征无显著差异（均 P>.05）。参与者主要为男性（328/382，85.9%），年龄大于 40 岁（290/382，75.9%）。考虑到相似的评分，Kano 调查的常见评估策略不允许对原则进行明确的优先级排序，当使用 Timko 工作中的满意度和不满意度指数更详细的方法时也是如此。因此，我们开发并应用了一种扩展的、所谓的“视线内”方法来进行评估。该方法是对 Timko 方法的修改，基于“观点”（POV）度量标准，生成排名系数。尽管这些原则以前几乎都被评为必须有（除了资源效率），这不利于对它们进行优先级排序，但新方法从必须有 POV 应用，在测试组和验证组中产生了相同的排名：（1）法律合规性，（2）内容有效性，（3）风险充分性，（4）实用性，（5）道德合理性，（6）可用性，（7）透明度，（8）技术充分性，（9）资源效率。
基于 Kano 工作的既定调查方法主要旨在对要评估的属性进行分类。这里提出的方法是一种进行优先级排序的有趣选择，并能够专注于最重要的标准，从而在审查医疗领域使用的应用程序时节省宝贵的时间，即使分类结果大致相同。这种方法在本文所述场景之外的适用性程度需要进一步调查。
