Mark Rubin

Researchers often distinguish between:  (1) Exploratory hypothesis tests — unplanned tests of post hoc hypotheses that may be based on the current results, and  (2) Confirmatory hypothesis tests — planned tests of a priori hypotheses that are independent from the current results  This distinction is supposed to be useful because exploratory results are assumed to be more “tentative” and “open to bias” than confirmatory results.

Exploratory hypothesis tests can be more compelling than confirmatory hypothesis tests

In this new article, I consider questionable research practices in the field of metascience. A

 questionable metascience practice

(QMP) is a research practice, assumption, or perspective that’s been questioned by several commentators as being potentially problematic for metascience and/or the science reform movement. I discuss 10 QMPs that relate to criticism, replication, bias, generalization, and the characterization of science.

Questionable Metascience Practices

In this paper (Rubin, 2020), I consider Fisher’s criticism that the Neyman-Pearson approach to hypothesis testing relies on the assumption of “repeated sampling from the same population.” This criticism is problematic for the Neyman-Pearson approach because it implies that test users need to know, for sure, what counts as the same or equivalent population as their current population.

Repeated Sampling from the Same Population?

In this paper (Rubin, 2022), I make two related points: (1) researchers should halve two-sided

 p

values if they wish to use them to make directional claims, and (2) researchers should

 not

halve their alpha level if they’re using two one-sided tests to test two directional null hypotheses.

One-Sided Significance Tests

In this paper (Rubin, 2021), I consider two types of Type I error probability. The

 Neyman-Pearson Type I error rate

refers to the maximum frequency of incorrectly rejecting a null hypothesis if a test was to be repeatedly reconducted on a series of different random samples that are all drawn from the exact same null population. Hence, the Neyman-Pearson Type I error rate refers to a long run of

 exact

replications.

What Type of Type I Error?

In this new paper (Rubin, 2021), I consider when researchers should adjust their alpha level (significance threshold) during multiple testing and multiple comparisons. I consider three types of multiple testing (disjunction, conjunction, and individual), and I argue that an alpha adjustment is only required for one of these three types.

When to Adjust Alpha During Multiple Testing

In this paper (Rubin, 2017), I consider Gelman and Loken’s (2013, 2014) garden of forking paths problem. Forking paths occur when researchers decide which analyses to perform based on information from their sample.

An Evaluation of Four Solutions to the Garden of Forking Paths Problem

The Costs of HARKing: Does it Matter if Researchers Engage in Undisclosed Hypothesizing After the Results are Known? While no-one’s looking, a Texas sharpshooter fires his gun at a barn wall, walks up to his bullet holes, and paints targets around them. When his friends arrive, he points at the targets and claims he’s a good shot.

The Costs of HARKing: Does it Matter if Researchers Engage in Undisclosed Hypothesizing After the…

Preregistration entails researchers registering their planned research hypotheses, methods, and analyses in a time-stamped document before they undertake their data collection and analyses. This document is then made available with the published research report in order to allow readers to identify discrepancies between what the researchers originally planned to do and what they actually ended up doing.

Does Preregistration Improve the Interpretablity and Credibility of Research Findings?

Stories by Mark Rubin on Medium

Preregistration entails researchers registering their planned research
hypotheses, methods, and analyses in a time-stamped document before they
undertake their data collection and analyses. This document is then made
available with the published research report in order to allow readers
to identify discrepancies between what the researchers originally
planned to do and what they actually ended up doing. In a recent article
([Rubin, 2020](https://doi.org/10.20982/tqmp.16.4.p376)), I question
whether this *historical transparency* facilitates judgments of
credibility over and above what I call the *contemporary transparency*
that is provided by (a) clear rationales for current hypotheses and
analytical approaches, (b) public access to research data, materials,
and code, and (c) demonstrations of the robustness of research
conclusions to alternative interpretations and analytical approaches.

<figure>
<img
src="https://cdn-images-1.medium.com/max/402/1*YYny0udOTH_iiN4dieeCmA.png" />
</figure>

My article covers issues such as
[HARKing](https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/the-costs-of-harking),
multiple testing, *p*-hacking, [forking
paths](https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/the-garden-of-forking-paths),
[exploratory
analyses](https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/p-values-in-exploratory-analyses),
optional stopping, researchers' biases, selective reporting, test
severity, publication bias, and replication rates. I argue for a nuanced
approach to these issues. In particular, I argue that only *some* of
these issues are problematic, and only under *some* conditions. I also
argue that, when they are problematic, these issues can be identified
via contemporary transparency per se.

<figure>
<img
src="https://cdn-images-1.medium.com/max/712/1*W15_55y4m_aPZ1E90yp5AA.png" />
</figure>

I conclude that preregistration's historical transparency does not
facilitate judgments about the credibility of research findings when
researchers provide contemporary transparency. Of course, in many cases,
researchers do not provide a sufficient degree of contemporary
transparency (e.g., no open research data or materials), and in these
cases preregistration's historical transparency may provide some useful
information.

However, I argue that historical transparency is a relatively narrow,
researcher-centric form of transparency because it focuses attention on
the predictions made by specific researchers at a specific point in
time. In contrast, contemporary transparency allows research data to be
considered from multiple, unplanned, theoretical and analytical
perspectives while maintaining a high degree of research credibility.
Hence, I suggest that the open science movement should push more towards
contemporary transparency and less towards historical transparency.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*Dk63b8_EVkeBDgMbu9fjQg.png" />
</figure>

For further information, please see:

Rubin, M. (2020). Does preregistration improve the credibility of
research findings? *The Quantitative Methods in Psychology, 16*(4),
376--390. <https://doi.org/10.20982/tqmp.16.4.p376>

I was invited to present my paper at the opening cermony of the Erasmus
Research Institute of Management Research Transparency Campaign at
Erasmus University Rotterdam on 1st March 2022. A recording of my
presentation is below:

<https://medium.com/media/426ebd8a39df3d5e3d87e9c18b29bb7b/href>

For more of my work in this area, please see:
<https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4e872ce8a3c){width="1"
height="1"}