Scientific knowledge shapes many of our collective beliefs and decisions, but what of its integrity?
It’s a question that’s receiving increasing attention, most recently because of a strategically orchestrated sting operation by Science Magazine’s John Bohannon involving the submission of intentionally flawed papers to over 300 open-access journals 1. It revealed a less than pretty picture of the state of the peer review and quality control processes at scholarly open-access journals, a problem that is hard to believe does not extend to traditional subscription based journals, like Science 2. Certainly, Bohannon’s decision to exclude these journals is suspicious, as open-access journals are viewed as a potential threat to traditional subscription based journals 3. Nevertheless, Bohannon’s article highlights the pervasive issue of quality control in science 4,5,6.
That ‘quality control in science’ is an issue should not come as a revelation.
Science has become increasingly introspective, spurring debate and questions about how scientific knowledge is produced. Peer-review, the foundation of scientific quality control, is straining in its current form under the pressures of: modern technology, specifically the internet; social norms like the “publish or perish” mentality within the academic community; and as Bohannon points out, economic incentives. There is increasing concern and evidence that this is leading to flawed 7, biased 8,9,10 and fraudulent 11,12 research falling through the cracks.
The result is both frustration within the academic community with a process that is viewed as slow and inadequate, and the eroding of scientific integrity and trust.
There is no perfect solution to the quality control problem in science, one that absolves us of the responsibility of thinking critically and being aware of the social, political, and economic context in which scientific knowledge is produced; however, few will deny there are opportunities for significant improvement. One avenue for improving quality control is to ensure the authenticity and integrity of the data underling scientific research through rigorous and standardized production and curation processes 13,14,15 and open data standards 16. These standards ensure the accessibility 17 and provenance necessary to critically evaluate data quality 18. On this front, there has been particularly significant progress for hydrometric data in recent years 13,14,16, which environmental monitoring agencies are increasingly able to leverage through modern data production software systems 15. These systems and standards have an important role to play in improving quality control in science and ensuring the integrity of the scientific enterprise.
1. Bohannon, J. Who’s Afraid of Peer Review? Science 342, 60–65, 2013.
2. A Cloning Scandal Rocks a Pillar of Science Publishing. The New York Times, July 7, 2010.
3. Wray, R. Open access threat to Reed’s publishing empire. the Guardian, February 19, 2004.
4. Rice, C. Open access publishing hoax: what Science magazine got wrong. the Guardian, October 4, 2013.
5. How science goes wrong. The Economist, October 19, 2013.
6. Trouble at the lab. The Economist, October 19, 2013.
7. Ben Goldacre. The statistical error that just keeps on coming. the Guardian September 9, 2011.
8. Henrich, J., Heine, S. J. & Norenzayan, A. Most people are not WEIRD. Nature 466, 29–29, 2010.
9. Easterbrook, P. J., Berlin, J. A., Gopalan, R. & Matthews, D. R. Publication bias in clinical research. Lancet 337, 867–872, 1991.
10. Ioannidis, J. P. A. Why Most Published Research Findings Are False. PLoS Med 2, e124, 2005.
11. Fang, F. C., Steen, R. G. & Casadevall, A. Misconduct accounts for the majority of retracted scientific publications. Proceedings of the National Academy of Sciences 201212247, 2012.
12. Looks good on paper. The Economist, September 26, 2013.
13. Sauer, V. B. Standards for the Analysis and Processing of Surface-Water Data and Information Using Electronic Methods: U.S. Geological Survey Water-Resources Investigations Report 01–4044,. 91, 2002.
14. WMO. Guide to Hydrological Practices Volume II Management of Water Resources and Application of Hydrological Practices, 2009.
15. Hamilton, S. The best water data possible! 5 key requirements for modern systems, 2013.
16. Peter Taylor. OGC® WaterML 2.0: Part 1- Timeseries. Open Geospatial Consortium, 2012.
17. Heidorn, P. B. Shedding Light on the Dark Data in the Long Tail of Science. Library Trends 57, 280–299, 2008.
18. Hamilton, S. Unfit for Purpose: The Disservice of Disinformation. Hydrology Corner, July 29, 2013.