- Loss of confidence in scientific integrity
- Publication biases
Publication bias is a concern for anyone who's ever written a review paper, conducted a meta-analysis, or interested in complex problems that require multiple experiments to tease apart interactive effects. In a nutshell, the reality is that journals aren't interested in publishing studies about non-significant results. If your study or experiment found no significant difference between treatments of interest, that's less likely to get attention (and thus improve the journal's impact factor), than statistically significant results. You can see how this can skew our perceptions of phenomena, if the only observations we are exposed to are the "interesting" ones.
Given how much we know about publication bias and the problems it causes, it has occured to Zoë and myself that what is really needed is a forum to publish all these non-significant results that no 'respectable' journal would sully their pages with. Given how easy it is to find even 'obscure' results via article databases, impact and significance may be more relevant at the scale of an individual article, rather than an entire journal (" the times, they are a-changin' ").
So, Zoë and I want to start 'the journal of non-significant results' to publish and provide access to all those results of experiments, studies, and methods that just didn't work out, or at least gave statistically non-significant, but perhaps scientifically interesting, results. Unless PLoS beats us to it (which would probably work better, actually).
Think about it for a moment:
How many grad students and research assistants have tried the same doomed method, without knowing that several people have already tried it before? How much better would results of meta-analyses become now that all the non-significant results could be included?
This is not intended to be entirely tongue-in-cheek (unlike the 'journal of irreproducible results' - that's totally different). Although it sounds cheeky, may not ever achieve the impact factor of Nature or even Ecology Letters, it would still provide a valuable and novel function in scientific publishing, with no shortage of suitable material. I think it's time to make science a little less sexy and a whole lot more practical and accurate.
In fact, this idea is not without precedent: there is already a Journal of Non-Significan Results in Education. Once again, education scientists are ahead of the curve (though apparently lagging in follow-through as the journal has yet to post it's first issue).
We also thought of the first spin-off sister journal: 'the journal of insignificant results' for results that are statistically significant, but biologically uninteresting.