The Times Higher Education reports on two new initiatives in the Netherlands to bolster scientific standards. Here, I want to talk about one of these initiatives I was involved in: A fund for replication research. The board of the Dutch science funder NWO still has to officially approve the final call for proposals, but the draft text is basically done.

See the follow up post where I introduce my R package and spreadsheet TOSTER to perform TOST equivalence tests, and link to a practical primer on this topic. When you find p > 0.05, you did not observe surprising data, assuming there is no true effect.

After my previous post about one-sided tests, some people wondered about two-sided F -tests.

TL;DR: Don’t like one-sided tests? Distribute your alpha level unequally (i.e., 0.04 vs 0.01) across two tails to still benefit from an increase in power. My two unequal tails in a 0.04/0.01 ratio (picture by my wife). This is a follow-up to my previous post, where I explained how you can easily become 20% more efficient when you aim for 80% power, by using a one-sided test.

I've created an easy to use R script that will import your data, and performs and writes up a state-of-the-art dependent or independent t-test.

Because the true size of effects is uncertain, determining the sample size for a study is a challenge. A-priori power analysis is often recommended, but practically impossible when effect sizes are very uncertain. One situation in which effect sizes are by definition uncertain is a replication study where the goal is to establish whether a previously observed effect can be reproduced.

A meta-analysis of 90 studies on precognition by Bem, Tressoldi, Rabeyron, & Duggan has been circulating recently. I have looked at this meta-analysis of precognition experiments for an earlier blog post. I had a very collaborative exchange with the authors, which was cordial and professional, and led the authors to correct the mistakes I pointed out and answer some questions I had.

If you teach methods and statistics courses, organize talks or symposia on good research practices, or simply want to read some good papers, check out this OSF page with a list of papers on research methods.

This blog post is presented in collaboration with a new interactive visualization of the distribution of p-values created by Kristoffer Magnusson (@RPsychologist) based on code by JP de Ruiter (@JPdeRuiter). Question 1 : Would you be inclined to interpret a p -value between 0.16- 0.17 as support for the presence of an effect, assuming the power of the study was 50%? Write down your answer – we will come back to this question

Throughout the history of psychological science, there has been a continuing debate about which statistics are used and how these statistics are reported. I distinguish between reporting statistics, and interpreting statistics. This is important, because a lot of the criticism on the statistics researchers use comes from how statistics are interpreted, not how they are reported.

An extended version of this blog post is now in press at PeerJ. TL;DR version: De Winter and Dodou (2015) analyzed the distribution (and its change over time) of a large number of p -values automatically extracted from abstracts in the scientific literature.