复制的可靠性：一项关于计算再现的研究。
The reliability of replications: a study in computational reproductions.
作者信息
Breznau Nate, Rinke Eike Mark, Wuttke Alexander, Adem Muna, Adriaans Jule, Akdeniz Esra, Alvarez-Benjumea Amalia, Andersen Henrik K, Auer Daniel, Azevedo Flavio, Bahnsen Oke, Bai Ling, Balzer Dave, Bauer Paul C, Bauer Gerrit, Baumann Markus, Baute Sharon, Benoit Verena, Bernauer Julian, Berning Carl, Berthold Anna, Bethke Felix S, Biegert Thomas, Blinzler Katharina, Blumenberg Johannes N, Bobzien Licia, Bohman Andrea, Bol Thijs, Bostic Amie, Brzozowska Zuzanna, Burgdorf Katharina, Burger Kaspar, Busch Kathrin, Castillo Juan-Carlos, Chan Nathan, Christmann Pablo, Connelly Roxanne, Czymara Christian S, Damian Elena, de Rooij Eline A, Ecker Alejandro, Edelmann Achim, Eder Christina, Eger Maureen A, Ellerbrock Simon, Forke Anna, Forster Andrea, Freire Danilo, Gaasendam Chris, Gavras Konstantin, Gayle Vernon, Gessler Theresa, Gnambs Timo, Godefroidt Amélie, Grömping Max, Groß Martin, Gruber Stefan, Gummer Tobias, Hadjar Andreas, Halbherr Verena, Heisig Jan Paul, Hellmeier Sebastian, Heyne Stefanie, Hirsch Magdalena, Hjerm Mikael, Hochman Oshrat, Höffler Jan H, Hövermann Andreas, Hunger Sophia, Hunkler Christian, Huth-Stöckle Nora, Ignácz Zsófia S, Israel Sabine, Jacobs Laura, Jacobsen Jannes, Jaeger Bastian, Jungkunz Sebastian, Jungmann Nils, Kanjana Jennifer, Kauff Mathias, Khan Salman, Khatua Sayak, Kleinert Manuel, Klinger Julia, Kolb Jan-Philipp, Kołczyńska Marta, Kuk John, Kunißen Katharina, Kurti Sinatra Dafina, Langenkamp Alexander, Lee Robin C, Lersch Philipp M, Liu David, Löbel Lea-Maria, Lutscher Philipp, Mader Matthias, Madia Joan E, Malancu Natalia, Maldonado Luis, Marahrens Helge, Martin Nicole, Martinez Paul, Mayerl Jochen, Mayorga Oscar J, McDonnell Robert, McManus Patricia, McWagner Kyle, Meeusen Cecil, Meierrieks Daniel, Mellon Jonathan, Merhout Friedolin, Merk Samuel, Meyer Daniel, Micheli Leticia, Mijs Jonathan, Moya Cristóbal, Neunhoeffer Marcel, Nüst Daniel, Nygård Olav, Ochsenfeld Fabian, Otte Gunnar, Pechenkina Anna, Pickup Mark, Prosser Christopher, Raes Louis, Ralston Kevin, Ramos Miguel, Reichert Frank, Roets Arne, Rogers Jonathan, Ropers Guido, Samuel Robin, Sand Gregor, Sanhueza Petrarca Constanza, Schachter Ariela, Schaeffer Merlin, Schieferdecker David, Schlueter Elmar, Schmidt Katja, Schmidt Regine, Schmidt-Catran Alexander, Schmiedeberg Claudia, Schneider Jürgen, Schoonvelde Martijn, Schulte-Cloos Julia, Schumann Sandy, Schunck Reinhard, Seuring Julian, Silber Henning, Sleegers Willem, Sonntag Nico, Staudt Alexander, Steiber Nadia, Steiner Nils D, Sternberg Sebastian, Stiers Dieter, Stojmenovska Dragana, Storz Nora, Striessnig Erich, Stroppe Anne-Kathrin, Suchow Jordan W, Teltemann Janna, Tibajev Andrey, Tung Brian, Vagni Giacomo, Van Assche Jasper, van der Linden Meta, van der Noll Jolanda, Van Hootegem Arno, Vogtenhuber Stefan, Voicu Bogdan, Wagemans Fieke, Wehl Nadja, Werner Hannah, Wiernik Brenton M, Winter Fabian, Wolf Christof, Wu Cary, Yamada Yuki, Zakula Björn, Zhang Nan, Ziller Conrad, Zins Stefan, Żółtak Tomasz, Nguyen Hung H V
机构信息
Organization and Program Planning, German Institute for Adult Education-Leibniz Center of Lifelong Learning, Bonn 53175, Germany.
School of Politics and International Studies, University of Leeds, Leeds LS2 9JT, UK.
出版信息
R Soc Open Sci. 2025 Mar 19;12(3):241038. doi: 10.1098/rsos.241038. eCollection 2025 Mar.
This study investigates researcher variability in computational reproduction, an activity for which it is least expected. Eighty-five independent teams attempted numerical replication of results from an original study of policy preferences and immigration. Reproduction teams were randomly grouped into a 'transparent group' receiving original study and code or 'opaque group' receiving only a method and results description and no code. The transparent group mostly verified original results (95.7% same sign and -value cutoff), while the opaque group had less success (89.3%). Second-decimal place exact numerical reproductions were less common (76.9 and 48.1%). Qualitative investigation of the workflows revealed many causes of error, including mistakes and procedural variations. When curating mistakes, we still find that only the transparent group was reliably successful. Our findings imply a need for transparency, but also more. Institutional checks and less subjective difficulty for researchers 'doing reproduction' would help, implying a need for better training. We also urge increased awareness of complexity in the research process and in 'push button' replications.
本研究调查了研究人员在计算再现方面的差异，而这是人们最不期望出现差异的一项活动。八十五个独立团队尝试对一项关于政策偏好与移民的原始研究结果进行数值复制。复制团队被随机分为“透明组”和“不透明组”，前者会收到原始研究及代码，后者仅收到方法和结果描述，没有代码。透明组大多验证了原始结果（95.7%符号相同且p值临界值相同），而不透明组的成功率较低（89.3%）。精确到小数点后第二位的数值再现不太常见（分别为76.9%和48.1%）。对工作流程的定性调查揭示了许多错误原因，包括失误和程序差异。在梳理失误时，我们仍然发现只有透明组取得了可靠的成功。我们的研究结果表明需要透明度，但还不止于此。机构检查以及降低研究人员“进行复制”时的主观难度会有所帮助，这意味着需要更好的培训。我们还敦促提高对研究过程和“一键式”复制中复杂性的认识。
Zotero 插件