the site of Adam Stevens

Pre-prints and the new ‘way things are done’

Last week, I was excited to submit my first preprint of a paper. Before this I hadn’t really had to opportunity – people do submit planetary science preprints to arXiv, but I don’t really feel like they ‘fit’ there. But now, with an honest-to-go biology paper in my hands, I had a ready made preprint service I could submit to. The paper is here, if you’d like to see.

However, my experience after submission wasn’t exactly what I’d expected, and I’m pretty sure now it wasn’t a good one. Within seconds, my preprint had been immediately and automatically tweeted several times. Ok, I thought, that’s fine. I like the idea of preprints opening a conversation about the science outside of traditional peer review. Then it was tweeted a few more times. Seemingly automatically again, with no context, no discussion. Perhaps people or scripts set up to tweet anything from bioRxiv, or preprints with particular keywords. Anyway, nothing controversial, so I squared this away in my brain as probably just a bit of a weird social media thing.

Then I get an e-mail from a reporter asking about the research. Naively, as I’m always happy to talk to people about my research, I agreed to a call to discuss what was going on. At the beginning of the call, I made clear that I was a little uncomfortable about the press covering what was not peer-reviewed science, but they reassured me that there was a standard process in place for covering preprints and we got to talking about the research.

Long story short, we asked them not to publish a piece until the paper had been peer-reviewed. Let’s get this straight – anyone can submit a preprint to places like arXiv or bioRxiv, and while they get some vetting to make sure they’re not a joke submission (as far as I can tell), no-one checks whether the science is legit. So I’m very uncomfortable with journalists publishing pieces on preprints. People that don’t know much about peer-review might not realise that the content could substantially change, and having been through the process several times, we aren’t kidding when we say that. It’s entirely possible for a reviewer to spot a small (or a large) mistake that could even completely change the conclusions of the research. The majority of the time (I hope?) that doesn’t happen, but peer-review provides a really important check on scientific research. I would emphasise that peer-review doesn’t decide if research is true or not, that’s the domain of scientific reproduction, it merely checks for logical and systematic consistency, that the scientist/s haven’t missed something obvious or made a mistake in their working. Peer-review isn’t the arbiter of scientific truth, and nor should it be, but promoting research before it’s been peer-review can lead to misunderstandings, retractions (or in some cases it should), or potentially the ruination of careers.

So, safe to say, we weren’t comfortable with them putting our research out before we were ready, especially since astrobiology has already a pretty bad example of ‘science by press release’ that didn’t go well for anybody. However, we were told (not by the reporter themselves) that since the research was out there, they were going to cover it, whether we wanted to be quoted in the article or not.

Now, I realise that journalists need to make a living, and that if they hadn’t covered it, ‘someone else would’ (though I would point out no-one else has contacted us about the research, though the article has been ‘rehosted’ by other news sites). But this is really not ok. The article is here, if you want a look. I’m not trying to attack New Scientist – the article is reasonable, doesn’t make outrageous claims and covers our research pretty well. But say under review we have to substantially change the conclusions, or what if the paper gets resoundedly rejected from every journal we send it to because it’s rubbish? The article will live on, at the best being pretty embarrassing, at worst… something worse.

My point is that there’s a new paradigm emerging with the open science movement and the expansion of preprints in general. I welcome expanding the conversation around scientific research, that people other than the assigned peer-reviewers can give opinions on the paper, the general theme of making the process more open, but if it leads to peer-review, the cornerstone of scientific research for hundreds of years and an incredibly important of quality assurance in science, getting sidelined, then perhaps it’s not such a good thing?

Leave a Reply

Your email address will not be published. Required fields are marked *