Haunschild Robin, Bornmann Lutz
1Max Planck Institute for Solid State Research, Heisenbergstr. 1, 70569 Stuttgart, Germany.
2Division for Science and Innovation Studies, Administrative Headquarters of the Max Planck Society, Hofgartenstr. 8, 80539 Munich, Germany.
Scientometrics. 2018;116(2):997-1012. doi: 10.1007/s11192-018-2771-1. Epub 2018 May 19.
Thelwall (J Informetr 11(1):128-151, 2017a. 10.1016/j.joi.2016.12.002; Web indicators for research evaluation: a practical guide. Morgan and Claypool, London, 2017b) proposed a new family of field- and time-normalized indicators, which is intended for sparse data. These indicators are based on units of analysis (e.g., institutions) rather than on the paper level. They compare the proportion of mentioned papers (e.g., on Twitter) of a unit with the proportion of mentioned papers in the corresponding fields and publication years. We propose a new indicator (Mantel-Haenszel quotient, MHq) for the indicator family. The MHq is rooted in the Mantel-Haenszel (MH) analysis. This analysis is an established method, which can be used to pool the data from several 2 × 2 cross tables based on different subgroups. We investigate using citations and assessments by peers whether the indicator family can distinguish between quality levels defined by the assessments of peers. Thus, we test the convergent validity. We find that the MHq is able to distinguish between quality levels in most cases while other indicators of the family are not. Since our study approves the MHq as a convergent valid indicator, we apply the MHq to four different Twitter groups as defined by the company Altmetric. Our results show that there is a weak relationship between the Twitter counts of all four Twitter groups and scientific quality, much weaker than between citations and scientific quality. Therefore, our results discourage the use of Twitter counts in research evaluation.
塞尔沃尔（《信息计量学杂志》11(1):128 - 151, 2017a. 10.1016/j.joi.2016.12.002；《研究评估的网络指标：实用指南》。摩根和克莱普尔出版社，伦敦，2017b）提出了一个新的字段和时间归一化指标家族，该家族适用于稀疏数据。这些指标基于分析单位（如机构）而非论文层面。它们将一个单位被提及的论文比例（如在推特上）与相应领域和出版年份中被提及的论文比例进行比较。我们为该指标家族提出了一个新指标（曼特尔 - 亨泽尔商数，MHq）。MHq源于曼特尔 - 亨泽尔（MH）分析。这种分析是一种既定方法，可用于汇总基于不同子组的多个2×2交叉表中的数据。我们通过引用和同行评估来研究该指标家族是否能够区分由同行评估定义的质量水平。因此，我们测试收敛效度。我们发现，在大多数情况下，MHq能够区分质量水平，而该家族的其他指标则不能。由于我们的研究认可MHq作为一个收敛有效的指标，我们将MHq应用于由Altmetric公司定义的四个不同推特群组。我们的结果表明，所有四个推特群组的推特计数与科学质量之间存在微弱关系，远弱于引用与科学质量之间的关系。因此，我们的结果不鼓励在研究评估中使用推特计数。
