University Medical Centre Groningen, Department of Epidemiology, University of Groningen, Hanzeplein 1, FA40, P.O. Box 30 001, 9700 RB, Groningen, The Netherlands.
University Medical Center Groningen, Department of Internal Medicine, University of Groningen, Groningen, The Netherlands.
BMC Psychol. 2021 Apr 28;9(1):65. doi: 10.1186/s40359-021-00566-x.
The Ruff Figural Fluency Test (RFFT) is a valid but time-consuming and labour-intensive cognitive paper-and-pencil test. A digital RFFT was developed that can be conducted independently using an iPad and Apple Pencil and RFFT scores are computed automatically. We investigated the validity and reliability of this digital RFFT.
We randomly allocated participants to the digital or paper-and-pencil RFFT. After the first test, the other test was performed immediately (cross-over). Participants were invited for a second digital RFFT 1 week later. For the digital RFFT, an (automatic) algorithm and two independent raters (criterion standard) assessed the number of unique designs (UD) and perseverative errors (PE). These raters also assessed the paper-and-pencil RFFT. We used Intraclass correlation coefficients (ICC), sensitivity, specificity, %-agreement, Kappa, and Bland-Altman plots.
We included 94 participants (mean (SD) age 39.9 (14.8), 73.4% follow-up). Mean (SD) UD and median (IQR) PE of the digital RFFT were 84.2 (26.0) and 4 (2-7.3), respectively. Agreement between manual and automatic scoring of the digital RFFT was high for UD (ICC = 0.99, 95% CI 0.98, 0.99, sensitivity = 0.98; specificity = 0.96) and PE (ICC = 0.99, 95% CI 0.98, 0.99; sensitivity = 0.90, specificity = 1.00), indicating excellent criterion validity. Small but significant differences in UD were found between the automatic and manual scoring (mean difference: - 1.12, 95% CI - 1.92, - 0.33). Digital and paper-and-pencil RFFT had moderate agreement for UD (ICC = 0.73, 95% CI 0.34, 0.87) and poor agreement for PE (ICC = 0.47, 95% CI 0.30, 0.62). Participants had fewer UD on the digital than paper-and-pencil RFFT (mean difference: - 7.09, 95% CI - 11.80, - 2.38). The number of UD on the digital RFFT was associated with higher education (Spearman's r = 0.43, p < 0.001), and younger age (Pearson's r = - 0.36, p < 0.001), showing its ability to discriminate between different age categories and levels of education. Test-retest reliability was moderate (ICC = 0.74, 95% CI 0.61, 0.83).
The automatic scoring of the digital RFFT has good criterion and convergent validity. There was low agreement between the digital RFFT and paper-and-pencil RFFT and moderate test-retest reliability, which can be explained by learning effects. The digital RFFT is a valid and reliable instrument to measure executive cognitive function among the general population and is a feasible alternative to the paper-and-pencil RFFT in large-scale studies. However, its scores cannot be used interchangeably with the paper-and-pencil RFFT scores.
Ruff 图形流畅性测试（RFFT）是一种有效的认知纸笔测试，但耗时且劳动强度大。我们开发了一种数字 RFFT，可以通过 iPad 和 Apple Pencil 独立进行，并且可以自动计算 RFFT 分数。我们研究了这种数字 RFFT 的有效性和可靠性。
我们随机将参与者分配到数字或纸笔 RFFT 组。第一次测试后，立即进行另一次测试（交叉）。参与者被邀请在一周后进行第二次数字 RFFT。对于数字 RFFT，使用（自动）算法和两名独立的评分者（标准）评估独特设计（UD）和持续错误（PE）的数量。这两名评分者还评估了纸笔 RFFT。我们使用了组内相关系数（ICC）、敏感性、特异性、%一致性、Kappa 和 Bland-Altman 图。
我们纳入了 94 名参与者（平均（标准差）年龄 39.9（14.8），73.4%随访）。数字 RFFT 的平均（标准差）UD 和中位数（IQR）PE 分别为 84.2（26.0）和 4（2-7.3）。数字 RFFT 的手动和自动评分之间的一致性很高，UD 的 ICC 为 0.99（95%CI 0.98，0.99），敏感性为 0.98，特异性为 0.96；PE 的 ICC 为 0.99（95%CI 0.98，0.99），敏感性为 0.90，特异性为 1.00，表明具有优异的标准效度。在 UD 的自动和手动评分之间发现了小但显著的差异（平均差异：-1.12，95%CI-1.92，-0.33）。UD 的数字和纸笔 RFFT 之间具有中等一致性（ICC 为 0.73，95%CI 0.34，0.87），PE 的一致性较差（ICC 为 0.47，95%CI 0.30，0.62）。参与者在数字 RFFT 上的 UD 比纸笔 RFFT 少（平均差异：-7.09，95%CI-11.80，-2.38）。数字 RFFT 的 UD 数量与较高的教育水平（Spearman's r=0.43，p<0.001）和较年轻的年龄（Pearson's r=-0.36，p<0.001）相关，表明其能够区分不同的年龄类别和教育水平。测试-再测试的可靠性为中等（ICC 为 0.74，95%CI 0.61，0.83）。
数字 RFFT 的自动评分具有良好的标准和收敛效度。数字 RFFT 与纸笔 RFFT 之间的一致性较低，测试-再测试的可靠性为中等，这可以用学习效应来解释。数字 RFFT 是一种有效且可靠的工具，可以测量一般人群的执行认知功能，并且可以作为大规模研究中纸笔 RFFT 的可行替代方案。但是，它的分数不能与纸笔 RFFT 的分数互换使用。
