Publication Date:
2021-09-03
Description:
Multiplicity arises when data analysis involves multiple simultaneous inferences, increasing the chance of spurious findings. It is a widespread problem frequently ignored by researchers. In this paper, we perform an exploratory analysis of the Web of Science database for COVID-19 observational studies. We examined 100 top-cited COVID-19 peer-reviewed articles based on p-values, including up to 7100 simultaneous tests, with 50% including 〉34 tests, and 20% 〉 100 tests. We found that the larger the number of tests performed, the larger the number of significant results (r = 0.87, p 〈 10−6). The number of p-values in the abstracts was not related to the number of p-values in the papers. However, the highly significant results (p 〈 0.001) in the abstracts were strongly correlated (r = 0.61, p 〈 10−6) with the number of p 〈 0.001 significances in the papers. Furthermore, the abstracts included a higher proportion of significant results (0.91 vs. 0.50), and 80% reported only significant results. Only one reviewed paper addressed multiplicity-induced type I error inflation, pointing to potentially spurious results bypassing the peer-review process. We conclude the need to pay special attention to the increased chance of false discoveries in observational studies, including non-replicated striking discoveries with a potentially large social impact. We propose some easy-to-implement measures to assess and limit the effects of multiplicity.
Print ISSN:
1661-7827
Electronic ISSN:
1660-4601
Topics:
Energy, Environment Protection, Nuclear Power Engineering
,
Medicine
Permalink