- Loss of confidence in scientific integrity
- Publication biases
Publication bias is a concern for anyone who's ever written a review paper, conducted a meta-analysis, or interested in complex problems that require multiple experiments to tease apart interactive effects. In a nutshell, the reality is that journals aren't interested in publishing studies about non-significant results. If your study or experiment found no significant difference between treatments of interest, that's less likely to get attention (and thus improve the journal's impact factor), than statistically significant results. You can see how this can skew our perceptions of phenomena, if the only observations we are exposed to are the "interesting" ones.
Given how much we know about publication bias and the problems it causes, it has occured to Zoë and myself that what is really needed is a forum to publish all these non-significant results that no 'respectable' journal would sully their pages with. Given how easy it is to find even 'obscure' results via article databases, impact and significance may be more relevant at the scale of an individual article, rather than an entire journal (" the times, they are a-changin' ").
So, Zoë and I want to start 'the journal of non-significant results' to publish and provide access to all those results of experiments, studies, and methods that just didn't work out, or at least gave statistically non-significant, but perhaps scientifically interesting, results. Unless PLoS beats us to it (which would probably work better, actually).
Think about it for a moment:
How many grad students and research assistants have tried the same doomed method, without knowing that several people have already tried it before? How much better would results of meta-analyses become now that all the non-significant results could be included?
This is not intended to be entirely tongue-in-cheek (unlike the 'journal of irreproducible results' - that's totally different). Although it sounds cheeky, may not ever achieve the impact factor of Nature or even Ecology Letters, it would still provide a valuable and novel function in scientific publishing, with no shortage of suitable material. I think it's time to make science a little less sexy and a whole lot more practical and accurate.
In fact, this idea is not without precedent: there is already a Journal of Non-Significan Results in Education. Once again, education scientists are ahead of the curve (though apparently lagging in follow-through as the journal has yet to post it's first issue).
We also thought of the first spin-off sister journal: 'the journal of insignificant results' for results that are statistically significant, but biologically uninteresting.
Boy do I have a manuscript for you!
ReplyDeleteThe third chapter of my MSc was never published mainly because of the non-significant results it generated.
Back in the day (2002 - gee, I'm old) we were still trying to determine whether the relationship between biodiversity and ecosystem function was real, and what 'shape' it had.
I spent a whole year performing a mesocosm experiment which showed no relationship between soil microarthropod diversity and ecosystem level processes of N and P cycling.
Various reviewers suggested various ways of re-analysing the data, but there was simply nothing significant to be found. It died a slow and painful death.
I still think of that data, and wonder whether I should re-attempt another analyses, but I think it is probably well past the time to let it go.
Rest in peace pathetic MSc mesocosm experiment, rest in peace.
It seems we may all be in luck. A recent, similar blog here:
ReplyDeletehttp://scienceblogs.com/neurotopia/2009/09/whats_wrong_with_negative_data.php?utm_source=selectfeed&utm_medium=rss
brought up this journal (which Volkar Bahn has contributed to)
http://www.jnr-eeb.org/index.php/jnr
Zoë, you should submit your mesocosm results to the Journal of Negative Results!
ReplyDeleteThanks, Kiyoko!