A blog on statistics, methods, philosophy of science, and open science. Understanding 20% of statistics will improve 80% of your inferences.

[Now with update for STATA by my colleague +Chris Snijders] [Now with update about using the MBESS package for within-subject designs] [Now with an update on using ESCI] Confidence intervals are confusing intervals. I have nightmares where my students will ask me what they are, and then I try to

 define

them, and I mumble something nonsensical, and they all point at me and laugh.

Calculating confidence intervals for Cohen’s d and eta-squared using SPSS, R, and Stata

code{white-space: pre-wrap;} span.smallcaps{font-variant: small-caps;} div.columns{display: flex; gap: min(4vw, 1.5em);} div.column{flex: auto; overflow-x: auto;} div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;} ul.task-list{list-style: none;} ul.task-list li input[type="checkbox"] { width: 0.8em; margin: 0 0.8em 0.2em -1em;  vertical-align: middle; } pre &gt; code.sourceCode { white-space: pre; position: relative; } pre &gt;

Retrieving Planned Sample Sizes from AsPredicted Preregistrations

Researchers increasingly use the Open Science Framework (OSF) to share files, such as data and code underlying scientific publications, or presentations and materials for scientific workshops. The OSF is an amazing service that has contributed immensely to a changed research culture where psychologists share data, code, and materials. We are very grateful it exists.   But it is not always the most user-friendly.

Easily download files from the Open Science Framework with Papercheck

Are meta-scientists ignoring philosophy of science (PoS)? Are they re-inventing the wheel? A recent panel at the Metascience conference engaged with this question, and the first sentence of the abstract states “Critics argue that metascience merely reinvents the wheel of other academic fields.” It’s a topic I have been thinking about for a while, so I will share my thoughts on this question.

Are meta-scientists ignoring philosophy of science?

The Journal of Personality and Social Psychology is one of the main outlets for social and personality psychologists. It publishes around 110 empirical articles a year (and a small number of other types of articles) and is considered a prestigious outlet. It is also often criticized. That’s to be expected.

The Journal of Personality and Social Psychology: The Good, The Bad, and The Ugly

What is the goal of data collection? This is a simple question, and as researchers we collect data all the time. But the answer to this question is not straightforward. It depends on the question that you are asking of your data. There are different questions you can ask from your data, and therefore, you can have different goals when collecting data. Here, I want to focus on collecting data to test scientific theories.

Prediction and Validity of Theories

This blog post is based on a pre-print by Coles, Tiokhin, Scheel, Isager, and Lakens “The Costs and Benefits of Replications”, submitted to Behavioral Brain Sciences as a commentary on “Making Replication Mainstream”.

In a summary of recent discussions about the role of direct replications in psychological science, Zwaan, Etz, Lucas, and Donnellan (2017) argue that replications should be more mainstream.

The Costs and Benefits of Replications

A

 p

-value is the probability of the observed, or more extreme, data, under the assumption that the null-hypothesis is true. The goal of this blog post is to understand what this means, and perhaps more importantly, what this doesn’t mean. People often misunderstand

 p

-values, but with a little help and some dedicated effort, we should be able explain these misconceptions.

Understanding common misconceptions about p-values

If I ever make a follow up to my current MOOC, I will call it ‘Improving Your Statistical Questions’. The more I learn about how people use statistics, the more I believe the main problem is not how people interpret the numbers they get from statistical tests. The real issue is which statistical questions researchers ask from their data. Our statistics education turns a blind eye to training people how to ask a good question.

The Statisticians' Fallacy

Science-Wise False Discovery Rate Does Not Explain the Prevalence of Bad Science h1 {   font-size: 34px; } h1.title {   font-size: 38px; } h2 {   font-size: 30px; } h3 {   font-size: 24px; } h4 {   font-size: 18px; } h5 {   font-size: 16px; } h6 {   font-size: 12px; } .table th:not([align]) {   text-align: left; } .main-container {   max-width: 940px; margin-left: auto; margin-right: auto; } code {   color: inherit;

Science-Wise False Discovery Rate Does Not Explain the Prevalence of Bad Science

I was listening to a recent Radiolab episode on blame and guilt, where the guest Robert Sapolsky mentioned a famous study on judges handing out harsher sentences before lunch than after lunch. The idea is that their mental resources deplete over time, and they stop thinking carefully about their decision – until having a bite replenishes their resources.

Impossibly hungry judges

The Dutch toilet cleaner ‘WC-EEND’ (literally: 'Toilet Duck') aired a famous commercial in 1989 that had the slogan ‘We from WC-EEND advise… WC-EEND’. It is now a common saying in The Netherlands whenever someone gives an opinion that is clearly aligned with their self-interest. In this blog, I will examine the hypothesis that blogs are, on average, of higher quality than journal articles.

Five reasons blog posts are of higher scientific quality than journal articles

One of the challenges of trying to get people to improve their statistical inferences is access to good software. After 32 years, SPSS still does not give a Cohen’s d effect size when researchers perform a t-test.

Equivalence testing in jamovi

When designing a study, you need to justify the sample size you aim to collect. If one of your goals is to observe a

 p

-values lower than the alpha level you decided upon (e.g., 0.05), one justification for the sample size can be a power analysis. A power analysis tells you the probability of observing a statistically significant effect, based on a specific sample size, alpha level, and true effect size.

How a power analysis implicitly reveals the smallest effect size you care about

In the latest exuberant celebration of how Bayes Factors will save science, Ravenzwaaij and Ioannidis write: “our study offers through simulations yet another demonstration of the unfortunate effect of p-values on statistical inferences.” Uh oh – what have these evil

 p

-values been up to this time?

No, the p-values are not to blame: Part 53

In a previous post, I compared equivalence tests to Bayes factors, and pointed out several benefits of equivalence tests. But a much more logical comparison, and one I did not give enough attention to so far, is the ROPE procedure using Bayesian estimation. I’d like to thank John Kruschke for feedback on a draft of this blog post.

ROPE and Equivalence Testing: Practically Equivalent?

In this blog, I’ll compare two ways of interpreting non-significant effects: Bayes factors and TOST equivalence tests. I’ll explain why reporting more than only Bayes factors makes sense, and highlight some benefits of equivalence testing over Bayes factors.

Examining Non-Significant Results with Bayes Factors and Equivalence Tests

After performing a study, you can correctly conclude there is an effect or not, but you can also incorrectly conclude there is an effect (a false positive, alpha, or Type 1 error) or incorrectly conclude there is no effect (a false negative, beta, or Type 2 error). The goal of collecting data is to provide

 evidence

for or against a hypothesis.

Why Type 1 errors are more important than Type 2 errors (if you care about evidence)

I’m happy to announce my first R package ‘TOSTER’ for equivalence tests (but don’t worry, there is an old-fashioned spreadsheet as well).     In an earlier blog post I talked about equivalence tests. Sometimes you perform a study where you might expect the effect is zero or very small.

TOST equivalence testing R package (TOSTER) and spreadsheet

One widely recommended approach to increase power is using a within subject design. Indeed, you need fewer participants to detect a mean difference between two conditions in a within-subjects design (in a dependent

 t

-test) than in a between-subjects design (in an independent

 t

-test). The reason is straightforward, but not always explained, and even less often expressed in the easy equation below.

Why Within-Subject Designs Require Fewer Participants than Between-Subject Designs

I’m really excited to be able to announce my “Improving Your Statistical Inferences” Coursera course. It’s a free massive open online course (MOOC) consisting of 22 videos, 10 assignments, 7 weekly exams, and a final exam. All course materials are freely available, and you can start whenever you want. In this course, I try to teach all the stuff I wish I had learned when I was a student.

Improving Your Statistical Inferences Coursera course

I think it was somewhere in the end of 2012 when my co-authors and I received an e-mail from Greg Francis pointing out that a study we published on the relationship between physical weight and importance was ‘too good to be true’. This was a stressful event. We were extremely uncertain about what this meant, but we realized it couldn’t be good. For me, it was the first article I had ever published. What did we do wrong?

Why scientific criticism sometimes needs to hurt

You might have seen the ‘Dance of the

 p

-values’ video by Geoff Cumming (if not, watch it here). Because

 p

-values and the default Bayes factors (Rouder, Speckman, Sun, Morey, &amp; Iverson, 2009) are both calculated directly from

 t

-values and sample sizes, we might expect there is also a Dance of the Bayes factors. And indeed, there is. Bayes factors can vary widely over identical studies, just due to random variation.

Dance of the Bayes factors

See the follow up post where I introduce my R package and spreadsheet TOSTER to perform TOST equivalence tests, and link to a practical primer on this topic.

When you find

 p

&gt; 0.05, you did not observe surprising data, assuming there is no true effect.

Absence of evidence is not evidence of absence: Testing for equivalence

After my previous post about one-sided tests, some people wondered about two-sided

 F

-tests.

One-sided F-tests and halving p-values

A meta-analysis of 90 studies on precognition by Bem, Tressoldi, Rabeyron, &amp; Duggan has been circulating recently. I have looked at this meta-analysis of precognition experiments for an earlier blog post. I had a very collaborative exchange with the authors, which was cordial and professional, and led the authors to correct the mistakes I pointed out and answer some questions I had.

Why a meta-analysis of 90 precognition studies does not provide convincing evidence of a true effect

Because the true size of effects is uncertain, determining the sample size for a study is a challenge. A-priori power analysis is often recommended, but practically impossible when effect sizes are very uncertain. One situation in which effect sizes are by definition uncertain is a replication study where the goal is to establish whether a previously observed effect can be reproduced.

How many participants should you collect? An alternative to the N * 2.5 rule

If you teach methods and statistics courses, organize talks or symposia on good research practices, or simply want to read some good papers, check out this OSF page with a list of papers on research methods.

Diversifying Research Methods Syllabi

Throughout the history of psychological science, there has been a continuing debate about which statistics are used and how these statistics are reported. I distinguish between

 reporting

statistics, and

 interpreting

statistics. This is important, because a lot of the criticism on the statistics researchers use comes from how statistics are interpreted, not how they are reported.

Which statistics should you report?

Psychology journals

 should

require, as a condition for publication, that data supporting the results in the paper are accessible in an appropriate public archive. I hope that in the near future, the ‘should’ in the previous sentence will disappear, and that data sharing has become a requirement. Many journals already have requirements to share data, but often not in a public database.

Psychology Journals Should Make Data Sharing A Requirement For Publication

See our paper on this topic: Delacre, M., Lakens, D., &amp; Leys, C. (2017). Why Psychologists Should by Default Use Welch’s

 t

-test Instead of Student’s

 t

-test.

Always use Welch's t-test instead of Student's t-test

Excellent news this morning: The negotiations between Dutch universities and Elsevier about new contracts for access to the scientific literature have broken down (report in Dutch by VSNU and De Volkskrant). This means universities are finally taking a stand: We want scientific knowledge to be freely available as open access to anyone who wants to read it, and we believe publishers are making more than enough money as it is. Moving towards fully

Negotiations between Elsevier and Dutch Universities break down: Time for change

This blog post is presented in collaboration with a new interactive visualization of the distribution of p-values created by Kristoffer Magnusson (@RPsychologist) based on code by JP de Ruiter (@JPdeRuiter).

 Question 1

: Would you be inclined to interpret a

 p

-value between 0.16- 0.17 as support for the presence of an effect, assuming the power of the study was 50%? Write down your answer – we will come back to this question

How a p-value between 0.04-0.05 equals a p-value between 0.16-017

An extended version of this blog post is now in press at PeerJ.




 TL;DR version:

De Winter and Dodou (2015)

 analyzed the distribution (and its change over time) of a large number of

p

 -values automatically extracted from abstracts in the scientific literature.

A peculiar surge of incorrect conclusions about the prevalence of p-values just below .05

[This is a re-post from my old blog, where this appeared March 8, 2014]  Several people have been reminding us that we need to perform well powered studies. It’s true this is a problem, because low power reduces the informational value of studies (a paper Ellen Evers and I wrote about this, has now appeared in Perspectives on Psychological Science, and is available here). If you happen to have a very large sample, good for you.

More Data Is Always Better, But Enough Is Enough

This blog post is now included in the paper "Sample size justification" available at PsyArXiv.

Observed power (or post-hoc power) is the statistical power of the test you have performed, based on the effect size estimate from your data. Statistical power is the probability of finding a statistical difference from 0 in your test (aka a ‘significant effect’), if there is a true difference to be found.

Observed power, and what to do if your editor asks for post-hoc power analyses

I think it is important people report confidence intervals to provide an indication of the uncertainty in the point estimates they report. However, I am not too enthusiastic about the current practice to report 95% confidence intervals. I think there are good reasons to consider alternatives, such as reporting 99.9% confidence intervals instead.

98% Capture Percentages and 99.9% Confidence Intervals

Chris Fraley and Simine Vizare (F&amp;V) published a very interesting paper in PlosOne where they propose to evaluate journals based on the sample size and statistical power of the studies.

Why Do We Cite Small N Studies?

[a slightly adapted version of this blog post is now in press at QJEP: see https://osf.io/ycag9/ for the manuscript and R scripts]



 In this blog, I'll explain how p-hacking will not lead to a peculiar prevalence of p-values just below .05 (e.g., in the 0.045-0.05 range) in the literature at large, but will instead lead to a difficult to identify increase in the Type 1 error rate across the 0.00-0.05 range.

What p-hacking really looks like: A comment on Masicampo &amp; LaLande (2012)

I'd like to gratefully acknowledge the extremely helpful comments and suggestions by Anton Kühberger and his co-authors on an earlier draft of this blog post, who patiently answered my questions and shared additional analyses. I'd also like to thank Marcel van Assen, Christina Bergmann, JP de Ruiter, and Uri Simonsohn for comments and suggestions.

Publication Bias in Psychology: Putting Things in Perspective

I have no idea how many people take the effort to reproduce a meta-analysis in their spare time. What I do know, based on my personal experiences of the last week, is that A) it’s too much work to reproduce a meta-analysis, primarily due to low reporting standards, and B) we need to raise the bar when doing meta-analyses.

On the Reproducibility of Meta-Analyses

Recently some statisticians have argued we have to lower the widely used

 p

&lt; .05 threshold. David Colquhoun got me thinking about this by posting a manuscript here, but Valen Johnson’s paper in PNAS is probably better known. They both suggest a

 p

&lt; .001 threshold would lower the false discovery rate.

Why Psychologists Should Ignore Recommendations to Use α &lt; .001

You might have looked at your data while the data collection was still in progress, and have been tempted to stop the study because the result was already significant. Alternatively, you might have analyzed your data, only to find the result was not yet significant, and decided to collect additional data. There are good ethical arguments to do this.

Data peeking without p-hacking

An often heard criticism of null-hypothesis significance testing is that the null is always false. The idea is that average differences between two samples will never be exactly zero (there will practically always be a tiny difference, even if it is only 0.001). Furthermore, if the sample size is large enough, tiny differences can be statistically significant. Both these statements are correct, but they do not mean the null is never true.

The Null Is Always False (Except When It Is True)

I used to be really happy with any

 p

-value smaller than .05, and very disappointed when

 p

-values turned out to be higher than .05. Looking back, I realize I was suffering from a bi-polar

 p

-value disorder. Nowadays, I interpret

 p

-values more evenly. Instead of a polar division between

 p

-values above and below the .05 significance level, I use a gradual interpretation of

 p

-values.

The probability of p-values as a function of the statistical power of a test

Recently, people have wondered why researchers seem to have a special interest in replicating studies that demonstrated unexpected or surprising results. In this blog post, I will explain why, statistically speaking, this makes sense. When we evaluate the likelihood that findings reflect real effects, we need to take the prior likelihood that the null-hypothesis is true into account.

Prior probabilities and replicating 'surprising and unexpected' effects

It’s important to communicate any remaining uncertainty in the results of an experiment.

Communicating uncertainty with p-values

‘The last couple of years it has been repeatedly pointed out that it is essential to perform replication studies.’ This is the first line of a Dutch article by Annie van Bergen, written in 1963 for the Nederlands Tijdschrift voor de Psychologie en Haar Grensgebieden (Dutch Journal for Psychology and her Border Areas). I first heard about it one month ago, when a group of researchers were giving short presentations about the value of replication

Replications (Van Bergen, 1963)

When the scientific discipline started to emerge, Bacon (1620) remarked: “And it is nothing strange if a thing not held in honour does not prosper”. His idea was that we need to give honour to scientists who do good work, otherwise no one would want to become a scientist.

Prestige and Punishment: Preventing Norm Violations in Science

The replication crisis is over. Sure, not everyone has gotten the memo (either about it having started, or about it having ended) but the majority of scientists agree that there were (slightly) too many findings from the past that cannot easily be replicated. The underlying reasons are clear: publication bias, flexibility in the data analysis, low power, and not enough rewards for replication studies.

Towards a more collaborative science with StudySwap

Does science offer a way to learn what is true about our world? According to the perspective in philosophy of science known as

 scientific realism

, the answer is ‘yes’. Scientific realism is the idea that successful scientific theories that have made novel predictions give us a good reason to believe these theories make statements about the world that are at least partially true.

Verisimilitude, Belief, and Progress in Psychological Science

Greenland and colleagues (Greenland et al., 2016) published a list with 25 common misinterpretations of statistical concepts such as power, confidence intervals, and, in points 1-10, p-values. Here I’ll explain how 50% of these problems are resolved by using equivalence tests in addition to null-hypothesis significance tests.

How p-values solve 50% of the problems with p-values

The Times Higher Education reports on two new initiatives in the Netherlands to bolster scientific standards. Here, I want to talk about one of these initiatives I was involved in: A fund for replication research. The board of the Dutch science funder NWO still has to officially approve the final call for proposals, but the draft text is basically done.

NWO pilot project will fund €3.000.000 worth of replication research

TL;DR: Don’t like one-sided tests? Distribute your alpha level unequally (i.e., 0.04 vs 0.01) across two tails to still benefit from an increase in power.

 My two unequal tails in a 0.04/0.01 ratio (picture by my wife).

This is a follow-up to my previous post, where I explained how you can easily become 20% more efficient when you aim for 80% power, by using a one-sided test.

Who do you love most? Your left-tail or your right tail?

I've created an easy to use R script that will import your data, and performs and writes up a state-of-the-art dependent or independent t-test.

The perfect t-test

I like p-curve analyses. They are a great tool to evaluate whether sets of studies have evidential value or not. In a recent paper, Simonsohn, Nelson, &amp; Simmons (2014) show how p-curve analyses can correct for publication bias using only significant results. That’s pretty cool. However, trim-and-fill is notoriously bad at unbiased effect size estimation.

P-curves are better at effect size estimation than trim-and-fill (and Michael Jordan is better at free throws than I am)

Clintin Davis-Stober recently pointed me to his blog about the

 v

-statistic. The

 v

-statistic (nothing Greek, just the letter v) is a measure of how accurately your data is estimating the corresponding parameter values in the population.

Evaluating Estimation Accuracy With the Award Winning V-Statistic

Yesterday Mike McCullough posted an interesting question on Twitter. He had collected some data, observed a p = 0.026 for his hypothesis, but he wasn't happy. Being aware that higher

 p

-values do not always provide strong support for H1, he wanted to set a new goal and collect more data.

Sample Size Planning: P-values or Precision?

I'll be teaching my first statistics lectures this September together with +Chris Snijders. While  preparing some assignments, I gained a new appreciation for clearly explaining  the basics of statistics.

What is a p-value?

This Thursday I’ll be giving a workshop on good research practices in Leuven, Belgium. The other guest speaker at the workshop is Eric-Jan Wagenmakers, so I thought I’d finally dive in to the relationship between Bayes Factors and

 p

-values to be prepared to talk in the same workshop as such an expert on Bayesian statistics and methodology.

Bayes Factors and p-values for independent t-tests

Most of this post is inspired by a lecture on probabilities by Ellen Evers during a PhD workshop we taught (together with Job van Wolferen and Anna van ‘t Veer) called ‘How do we know what’s likely to be true’. I’d heard this lecture before (we taught the same workshop at Eindhoven a year ago) but now she extended her talk to the probability of observing a mix of significant an non-significant findings.

Too True to be Bad: When Sets of Studies with Significant and Non-Significant Findings Are Probably True

I recently read a meta-analysis on precognition studies by Bem, Tressoldi, Rabeyron, and Duggan (available on SSRN). The authors conclude in the abstract: 'We can now report a metaanalysis of 90 experiments from 33 laboratories in 14 different countries which yielded an overall positive effect in excess of 6 sigma with an effect size (Hedges’ g) of 0.09, combined z = 6.33, p = 1.2 ×10^-10). A Bayesian analysis yielded a Bayes Factor of 1.24 ×

The 20% Statistician

Communicating uncertainty with p-values

Replications (Van Bergen, 1963)