Last week (14-15 Nov), I went to Melbourne for a workshop (“From Replication Crisis to Credibility Revolution”). The workshop was hosted by my collaborator and “credibility revolutionary” Fiona Fidler.
I suspect many workshops and mini-conferences of this nature are popping out all over the world as many researchers are very much aware of “reproducibility crisis”. But what was unique about this one is its interdisciplinary nature; we had philosophers, psychologists, computer scientists, lawyers, pharmacologists, oncologists, statisticians, ecologists and evolutionary biologists (like myself).
I really like the idea of calling “reproducibility crisis” “credibility revolution” (hence the title). A speaker at the workshop, Glenn Begley, wants to call it “innovation opportunity” (he wrote this seminal comment for Nature). What a cool idea! And these re-namings make things a lot more positive than a bit of doom-and-gloom feel of “replicability crisis”. Indeed, there are a lot of positive movements toward Open Science and Transparency, happening to remedy the current ‘questionable’ practice.
Although I live in Sydney, I was also in Melbourne early last month (4-5 Oct) for a small conference. This is because Tom Stanly invited me over, as an evolutionary biologist, to give a talk on meta-analysis to a bunch of economists who love meta-analyses. To my surprise, I had a lot of fun chatting with meta-enthusiastic economists.
Tom is not only an economist but also a credibility revolutionary, like Fiona. He has asked me to invite ecologists and evolutionary biologists to comment on his blog about a credibility revolution. It is an excellent read. And if you can make comments to join the conversation, Tom will appreciate it a lot, and get conversations going. Disciplines need to unite together to make this revolution successful or make the most of this innovation opportunity. So join the credibility revolution! (meanwhile, I am now off to Japan to talk more about meta-analysis, and sample nice food – will joint the revolution once I am back).
Thanks for running this website! Maybe you are interested in the following paper that was written by a zoologist, a psychologist and an epidemiologist / statistician. I’m not sure if the title sounds positive, but I would love it if the content was perceived as positive.
Amrhein, Trafimow & Greenland: “Inferential statistics as descriptive statistics: there is no replication crisis if we don’t expect replication.” https://peerj.com/preprints/26857
Thanks. Your pre-print presents a thoughtful and thorough assessment of current null hypothesis testing practices and potential alternatives. Its a really useful paper.
You (and anyone else reading this) may be interested that another major psychology replication effort was just made public today (psyarxiv.com/9654g). One of the features was massive samples that dramatically reduced the rates of false negatives. Interestingly, however, a sizeable portions of the ~half the results that ‘replicated’ (were ‘statistically significant’ in the replication thanks to very large n) apparently had such dramatically smaller effects that at least some psychologists consider them trivial (I can’t find the tweet I saw earlier that made this point, sorry). Just another example of ‘statistical significance’ failing to equal practical significance.
The Credibility Revolution in Empirical Economics: How Better Research Design is Taking the Con out of Econometrics
Thanks for this hint! Another useful resource is the current discussion on the following blog, “Language for communicating frequentist results about treatment effects”:
https://discourse.datamethods.org/t/language-for-communicating-frequentist-results-about-treatment-effects/934
Among other things, Sander Greenland says: “The resistance I have encountered in medical-journal venues to such change is staggering, with editors attempting to back-rationalize every bad description and practice they have published.”
So how could we help changing bad reporting practices? Frank Harrell replies: “Besides publicizing errors the usual way, a little twitter shame can help, plus journal club blogs. I think the lowhanging fruit is the use of cutoffs for anything, and the ‘absence of evidence is not evidence of absence’ error.”
I completely agree to the ‘low hanging fruit’: Most journals and most talks are usually full of proofs of the null like “there was no difference (p>0.05),” and I guess this happens or happened to most of us sometimes. I made an (unpublished) survey of 100 papers from five journals and found that about 40% of results with p>0.05 were reported as “there was no difference” (or using some similar wording).
I thus replied on the blog: “I am currently considering writing comments and short notes to journals in my field (ecology and animal behavior) – it is so easy to spot the low hanging fruit, most often as proofs of the null hypothesis. If everyone would do this in their field, in a kind of concerted action, perhaps we would have a real chance to change something?”
What do you think about this idea: Sending comments or forum contributions, or whatever form of short publication is applicable, to many journals in our field, pointing out ‘proofs of the null’ in random selections of papers from the respective journals?
In case you have not seen it: We discuss proofs of the null and other problems with significance testing in the following review, of course with references to Fiona, Shinichi, and Tim:
Amrhein V, Korner-Nievergelt F, Roth T. 2017. The earth is flat (p > 0.05): significance thresholds and the crisis of unreplicable research. PeerJ 5:e3544.
https://peerj.com/articles/3544
Hi Valentin,
I like your idea of documenting the extent of ‘proof of the null’ in the literature. I wonder if there is a text-mining algorithm that could identify this and thus allow you to expand your sample, and maybe to evaluate how this varies across different subfields. I think it would also be interesting to compare these claims to statistical power.
By the way, I’ve seen your Earth is Flat paper, and I really like it. You briefly mention equivalence testing in this paper. You may be interested to know that some avian behavioral ecologists have just published a paper advocating for this method:
https://theconversation.com/the-equivalence-test-a-new-way-for-scientists-to-tackle-so-called-negative-results-106041
https://doi.org/10.1016/j.anbehav.2018.09.004
Great, thanks, I haven’t seen that paper yet! (So bad Animal Behaviour is still behind a paywall…).