Mark Rubin

Researchers often distinguish between:  (1) Exploratory hypothesis tests — unplanned tests of post hoc hypotheses that may be based on the current results, and  (2) Confirmatory hypothesis tests — planned tests of a priori hypotheses that are independent from the current results  This distinction is supposed to be useful because exploratory results are assumed to be more “tentative” and “open to bias” than confirmatory results.

Exploratory hypothesis tests can be more compelling than confirmatory hypothesis tests

In this new article, I consider questionable research practices in the field of metascience. A

 questionable metascience practice

(QMP) is a research practice, assumption, or perspective that’s been questioned by several commentators as being potentially problematic for metascience and/or the science reform movement. I discuss 10 QMPs that relate to criticism, replication, bias, generalization, and the characterization of science.

Questionable Metascience Practices

In this paper (Rubin, 2020), I consider Fisher’s criticism that the Neyman-Pearson approach to hypothesis testing relies on the assumption of “repeated sampling from the same population.” This criticism is problematic for the Neyman-Pearson approach because it implies that test users need to know, for sure, what counts as the same or equivalent population as their current population.

Repeated Sampling from the Same Population?

In this paper (Rubin, 2022), I make two related points: (1) researchers should halve two-sided

 p

values if they wish to use them to make directional claims, and (2) researchers should

 not

halve their alpha level if they’re using two one-sided tests to test two directional null hypotheses.

One-Sided Significance Tests

In this paper (Rubin, 2021), I consider two types of Type I error probability. The

 Neyman-Pearson Type I error rate

refers to the maximum frequency of incorrectly rejecting a null hypothesis if a test was to be repeatedly reconducted on a series of different random samples that are all drawn from the exact same null population. Hence, the Neyman-Pearson Type I error rate refers to a long run of

 exact

replications.

What Type of Type I Error?

In this new paper (Rubin, 2021), I consider when researchers should adjust their alpha level (significance threshold) during multiple testing and multiple comparisons. I consider three types of multiple testing (disjunction, conjunction, and individual), and I argue that an alpha adjustment is only required for one of these three types.

When to Adjust Alpha During Multiple Testing

In this paper (Rubin, 2017), I consider Gelman and Loken’s (2013, 2014) garden of forking paths problem. Forking paths occur when researchers decide which analyses to perform based on information from their sample.

An Evaluation of Four Solutions to the Garden of Forking Paths Problem

The Costs of HARKing: Does it Matter if Researchers Engage in Undisclosed Hypothesizing After the Results are Known? While no-one’s looking, a Texas sharpshooter fires his gun at a barn wall, walks up to his bullet holes, and paints targets around them. When his friends arrive, he points at the targets and claims he’s a good shot.

The Costs of HARKing: Does it Matter if Researchers Engage in Undisclosed Hypothesizing After the…

Preregistration entails researchers registering their planned research hypotheses, methods, and analyses in a time-stamped document before they undertake their data collection and analyses. This document is then made available with the published research report in order to allow readers to identify discrepancies between what the researchers originally planned to do and what they actually ended up doing.

Does Preregistration Improve the Interpretablity and Credibility of Research Findings?

Stories by Mark Rubin on Medium

### The Costs of HARKing: Does it Matter if Researchers Engage in Undisclosed Hypothesizing After the Results are Known?

While no-one's looking, a Texas sharpshooter fires his gun at a barn
wall, walks up to his bullet holes, and paints targets around them. When
his friends arrive, he points at the targets and claims he's a
good shot.

<figure>
<img
src="https://cdn-images-1.medium.com/max/357/1*E3s7VDeUsqljaIkTk-v7eQ.png" />
<figcaption>Source: Dirk-Jan Hoek. <a
href="https://www.flickr.com/photos/23868780@N00/7374874302">https://www.flickr.com/photos/23868780@N00/7374874302</a></figcaption>
</figure>

In 1998, Norbert Kerr discussed an analogous situation in which
researchers engage in undisclosed [*hypothesizing after the results are
known*](https://www.google.com/url?q=https%3A%2F%2Fjournals.sagepub.com%2Fdoi%2Fpdf%2F10.1207%2Fs15327957pspr0203_4&sa=D&sntz=1&usg=AFQjCNErBvYhNqrCLFAUJfVcc95e49Iv7w)
or HARKing. In this case, researchers conduct statistical tests, observe
their research results (bullet holes), and then construct post hoc
hypotheses (paint targets) to fit these results. In their research
reports, they then pretend that their post hoc hypotheses are actually a
priori hypotheses. This *questionable research practice* is thought to
have contributed to the replication crisis in science, and it provides
part of the rationale for researchers to publicly preregister their
hypotheses before they conduct their research. In a recent [BJPS
article](https://www.google.com/url?q=https%3A%2F%2Fdoi.org%2F10.1093%2Fbjps%2Faxz050&sa=D&sntz=1&usg=AFQjCNGBz4GZuUnqKjeSLzxjEmXi0A5e9A),
I discuss the concept of HARKing from a philosophical standpoint and
argue against the view that it is problematic for scientific progress.

I begin my article by noting that scientists do not make absolute,
dichotomous judgements about theories and hypotheses being "true" or
"false." Instead, they make relative judgements about theories and
hypotheses being more or less true that other theories and hypotheses in
accounting for certain phenomena. These judgements can be described as
*estimates of relative verisimilitude* (Cevolani & Festa, 2018).

I then note that a HARKer is obliged to provide a theoretical rationale
for their secretly post hoc hypothesis in the Introduction section of
their research report. Despite being secretly post hoc, this theoretical
rationale provides a result-independent basis for an initial estimate of
the relative verisimilitude of the HARKed hypothesis. (The rationale is
"result-independent" because it doesn't formally refer to the current
result. If it did, then the rationale's post hoc status would no longer
be a secret!) The current result can then provide a second,
epistemically independent basis for adjusting this initial estimate of
verisimilitude upwards or downards (for a similar view, see Lewandowsky,
2019; Oberauer & Lewandowsky, 2019). Hence, readers can estimate the
relative verisimilitude of a HARKed hypothesis (a) without taking the
current result into account and (b) after taking the current result into
account, even if they have been misled about when the researcher deduced
the hypothesis. Consequently, readers can undertake a valid updating of
the estimated relative verisimilitude of a hypothesis even though,
unbeknowst to them, it has been HARKed. Importantly, there's no
"double-counting" (Mayo, 2008), "circular reasoning" (Nosek et al.,
2018), or violation of the *use novelty* principle here (Worrall, 1985,
2014), because the current result has not been used in the formal
theoretical rationale for the HARKed hypothesis. Consequently, it's
legitimate to use the current result to change (increase or decrease)
the initial estimate of the relative verisimilitude of that hypothesis.

To translate this reasoning to the Texas sharpshooter analogy, it's
necessary to distinguish HARKing from *p*-hacking. If our sharpshooter
painted a new target around his stray bullet hole but retained his
substantive claim that he's "a good shot," then he'd be similar to a
researcher who conducted multiple statistical tests and then selectively
reported only those results that supported their original a priori
substantive hypothesis. Frequentist researchers would call this
researcher a "*p*-hacker" rather than a HARKer ([Rubin,
2017b](https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/the-garden-of-forking-paths?authuser=0),
p. 325; Simmons et al., 2011). To be a HARKer, researchers must also
change their original a priori hypothesis or create a totally new one.
Hence, a more appropriate analogy is to consider a sharpshooter who
changes *both* their statistical hypothesis (i.e., paints a new target
around their stray bullet hole) *and* their broader substantive
hypothesis (their claim). Let's call her Jane!

Jane initially believes "I'm a good shot" (H1). However, after missing
the target that she was aiming for (T1), she secretly paints a new
target (T2) around her bullet hole and declares to her friends: "I'm a
good shot, but I can't adjust for windy conditions. I aimed at T1, but
there was a 30 mph easterly cross-wind. So, I knew I'd probably hit T2
instead." In this case, Jane has generated a new, post hoc hypothesis
(H2) and passed it off as an a priori hypothesis. Note that, unlike our
original Texas sharpshooter, Jane isn't being deceptive about her
*procedure* here (i.e., what she actually did): It's true that she aimed
her gun at T1. She's only being deceptive about the a priori status of
H2, which she secretly developed *after* she missed T1 (i.e., she's
HARKing). Importantly, however, Jane's deception doesn't prevent her
friends from making a valid initial estimate of the verisimilitude of
her HARKed hypothesis and then updating this estimate based on the
location of her bullet hole:

> "We know that Jane's always trained indoors. So, it makes sense that
> she hasn't learned to adjust for windy conditions. We also know that
> (a) Jane was aiming at T1, and (b) there was a 30 mph easterly
> cross-wind. Our calculations show that, if someone was a good shot,
> and they were aiming at T1, but they didn't adjust for an easterly 30
> mph cross-wind, then their bullet would hit T2's location. So, our
> initial estimated verismilitude for H2 is relatively high. The
> evidence shows that Jane's bullet did, in fact, hit T2. Consequently,
> we can tentatively increase our support for H2: Jane appears to be a
> good shot who can't adjust for windy conditions. Of course, we'd also
> want to test H2 again by asking Jane to hit targets on both windy and
> non-windy days!"

<figure>
<img
src="https://cdn-images-1.medium.com/max/948/1*-CEx-xrzbzo6cx81MnLWtg.jpeg" />
<figcaption>We can predict the location of the sharpshooter’s bullet
hole on the basis of her (secretly HARKed) hypothesis that she is a good
shot but cannot adjust for windy conditions. We can then use the
location of the bullet hole to increase or decrease our estimated
relative verisimilitude for this prediction. Source: <a
href="https://www.google.com/url?q=https%3A%2F%2Fpixabay.com%2Fphotos%2Fwoman-rifle-shoot-gun-weapon-2577104%2F&amp;sa=D&amp;sntz=1&amp;usg=AFQjCNFiBLEmq2w5mxhFZzyaBsAhVLGWVA">https://pixabay.com/photos/woman-rifle-shoot-gun-weapon-2577104/</a></figcaption>
</figure>

For further information, please see:

Rubin, M. (2022). The costs of HARKing. *The British Journal for the
Philosophy of Science, 73.*
[https://doi.org/10.1093/bjps/axz050](https://www.google.com/url?q=https%3A%2F%2Fdoi.org%2F10.1093%2Fbjps%2Faxz050&sa=D&sntz=1&usg=AFQjCNGBz4GZuUnqKjeSLzxjEmXi0A5e9A)
or \*[Publisher's free
access](https://www.google.com/url?q=https%3A%2F%2Facademic.oup.com%2Fbjps%2Fadvance-article%2Fdoi%2F10.1093%2Fbjps%2Faxz050%2F5651026%3FguestAccessKey%3Db361c0e4-77f2-4443-9d77-7f603e566f53&sa=D&sntz=1&usg=AFQjCNHDGcxumGe2286Gum7DnqOpJRq1UA)\*
or \*[Self-archived
version](https://drive.google.com/file/d/1bGIUjHSEAoJYJke6RWtBphXJjZLr1UeX/view?usp=sharing)\*

For more of my work in this area, please see:
<https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ab5710f9808){width="1"
height="1"}

Preregistration entails researchers registering their planned research
hypotheses, methods, and analyses in a time-stamped document before they
undertake their data collection and analyses. This document is then made
available with the published research report in order to allow readers
to identify discrepancies between what the researchers originally
planned to do and what they actually ended up doing. In a recent article
([Rubin, 2020](https://doi.org/10.20982/tqmp.16.4.p376)), I question
whether this *historical transparency* facilitates judgments of
credibility over and above what I call the *contemporary transparency*
that is provided by (a) clear rationales for current hypotheses and
analytical approaches, (b) public access to research data, materials,
and code, and (c) demonstrations of the robustness of research
conclusions to alternative interpretations and analytical approaches.

<figure>
<img
src="https://cdn-images-1.medium.com/max/402/1*YYny0udOTH_iiN4dieeCmA.png" />
</figure>

My article covers issues such as
[HARKing](https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/the-costs-of-harking),
multiple testing, *p*-hacking, [forking
paths](https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/the-garden-of-forking-paths),
[exploratory
analyses](https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/p-values-in-exploratory-analyses),
optional stopping, researchers' biases, selective reporting, test
severity, publication bias, and replication rates. I argue for a nuanced
approach to these issues. In particular, I argue that only *some* of
these issues are problematic, and only under *some* conditions. I also
argue that, when they are problematic, these issues can be identified
via contemporary transparency per se.

<figure>
<img
src="https://cdn-images-1.medium.com/max/712/1*W15_55y4m_aPZ1E90yp5AA.png" />
</figure>

I conclude that preregistration's historical transparency does not
facilitate judgments about the credibility of research findings when
researchers provide contemporary transparency. Of course, in many cases,
researchers do not provide a sufficient degree of contemporary
transparency (e.g., no open research data or materials), and in these
cases preregistration's historical transparency may provide some useful
information.

However, I argue that historical transparency is a relatively narrow,
researcher-centric form of transparency because it focuses attention on
the predictions made by specific researchers at a specific point in
time. In contrast, contemporary transparency allows research data to be
considered from multiple, unplanned, theoretical and analytical
perspectives while maintaining a high degree of research credibility.
Hence, I suggest that the open science movement should push more towards
contemporary transparency and less towards historical transparency.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*Dk63b8_EVkeBDgMbu9fjQg.png" />
</figure>

For further information, please see:

Rubin, M. (2020). Does preregistration improve the credibility of
research findings? *The Quantitative Methods in Psychology, 16*(4),
376--390. <https://doi.org/10.20982/tqmp.16.4.p376>

I was invited to present my paper at the opening cermony of the Erasmus
Research Institute of Management Research Transparency Campaign at
Erasmus University Rotterdam on 1st March 2022. A recording of my
presentation is below:

<https://medium.com/media/426ebd8a39df3d5e3d87e9c18b29bb7b/href>

For more of my work in this area, please see:
<https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4e872ce8a3c){width="1"
height="1"}

In this new paper ([Rubin,
2021](https://doi.org/10.1007/s11229-021-03276-4)), I consider when
researchers should adjust their alpha level (significance threshold)
during multiple testing and multiple comparisons. I consider three types
of multiple testing (disjunction, conjunction, and individual), and I
argue that an alpha adjustment is only required for one of these
three types.

### There's No Need to Adjust Alpha During Individual Testing

I argue that an alpha adjustment is *not* necessary when researchers
undertake a single test of an individual null hypothesis, even when many
such tests are conducted within the same study.

For example, in the jelly beans study below, it's perfectly acceptable
to claim that there's "a link between green jelly beans and acne" using
an unadjusted alpha level of .05 given that this claim is based on a
*single test* of the hypothesis that green jelly beans cause acne rather
than *multiple tests* of this hypothesis.

<figure>
<img
src="https://cdn-images-1.medium.com/max/540/1*nJz_1rvgtOvvlEzbpubPBg.png" />
<figcaption>Retrieved from <a
href="https://xkcd.com/882/">https://xkcd.com/882/</a></figcaption>
</figure>

For a list of quotes from others that are consistent with my position on
individual testing, please see Appendix
B [here](https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/p-values-in-exploratory-analyses).

To be clear, I'm not saying that an alpha adjustment is *never*
necessary. It is necessary when *at least one* significant result would
be sufficient to support a *joint* hypothesis that's composed of several
constituent hypotheses that each undergo testing (i.e., disjunction
testing). For example, an alpha adjustment would be necessary to
conclude that "jelly beans of one or more colours cause acne" because,
in this case, a single significant result for at least one of the 20
colours of jelly beans would be sufficient to support this claim, and so
a familywise error rate is relevant.

### Studywise Error Rates are Not Usually Relevant

I also argue against the automatic (mindless) use of what I call
*studywise error rates* --- the familywise error rate that is associated
with all of the hypotheses that are tested in a study. I argue that
researchers should only be interested in studywise error rates if they
are interested in testing the associated joint studywise hypotheses, and
researchers are not usually interested in testing studywise hypotheses
because they rarely have any theoretical relevance. As I explain in my
paper, "in many cases, the joint studywise hypothesis has no relevance
to researchers' specific research questions, because its constituent
hypotheses refer to comparisons and variables that have no theoretical
or practical basis for joint consideration."

<figure>
<img
src="https://cdn-images-1.medium.com/max/981/1*aYxQY8dW0TrKc3MoPlUUmA.jpeg" />
<figcaption>Sometimes it doesn’t make sense to combine different
hypotheses as part of the same family!</figcaption>
</figure>

For example, imagine that a researcher conducts a study in which they
test gender, age, and nationality differences in alcohol use. Do they
need to adjust their alpha level to account for their multiple testing?
I argue "no" unless they want to test a studywise hypothesis that, for
example: "*Either* (a) men drink more than women, (b) young people drink
more than older people, *or* (c) the English drink more than Italians."
If the researcher does not want to test this potentially atheoretical
joint hypothesis, then they should not be interested in controlling the
associated familywise error rate, and instead they should consider each
individual hypothesis separately. As I explain in my paper, "researchers
should not be concerned about erroneous answers to questions that they
are not asking."

<figure>
<img
src="https://cdn-images-1.medium.com/max/936/1*-DFqzDI8yS18WZqCfI5mpg.jpeg" />
<figcaption>Thomas Wiecki <a
href="http://twitter.com/twiecki">@twiecki</a> <a
href="https://twitter.com/twiecki/status/1418462017476759552?s=20">https://twitter.com/twiecki/status/1418462017476759552?s=20</a></figcaption>
</figure>

For a list of quotes that support my position on studywise error rates,
please see Appendix
A [here](https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/p-values-in-exploratory-analyses).

For further information, please see:

Rubin, M. (2021). When to adjust alpha during multiple testing: A
consideration of disjunction, conjunction, and individual testing.
*Synthese*. <https://doi.org/10.1007/s11229-021-03276-4> or
[Open Access](https://drive.google.com/file/d/1Rl23xQxSs06-uAb0uhUKTxdTFCLrFbrG/view?usp=sharing)

For more of my work in this area, please see:
<https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=574f8484f984){width="1"
height="1"}