Critical metascience takes a step back to question some common assumptions, approaches, problems, and solutions in metascience.

Mark Rubin

Felipe Romero presented his work - “The conceptual origins of metascience: Fashion or revolution?” - in The Popper Seminar at the London School of Economics on 30th May 2023. Here, I highlight a few points that he raised and provide some comments along the way. ::: {#youtube2-AKWCCdK-jgc .youtube-wrap attrs=“{"videoId":"AKWCCdK-jgc","startTime":null,"endTime":null}” component-name=“Youtube2ToDOM”} Ein Fehler ist aufgetreten.

What's Special About Metascience?

In this new article, I consider questionable research practices in the field of metascience. A questionable metascience practice (QMP) is a research practice, assumption, or perspective that’s been questioned by several commentators as being potentially problematic for metascience and/or the science reform movement. I discuss 10 QMPs that relate to criticism, replication, bias, generalization, and the characterization of science.

Questionable Metascience Practices

The inflation of Type I error rates is thought to be one of the causes of the replication crisis. Questionable research practices such as
<em>
 p
</em>
-hacking are thought to inflate Type I error rates above their nominal level, leading to unexpectedly high levels of false positives in the literature and, consequently, unexpectedly low replication rates. In this article, I offer an alternative view.

Type I Error Rates are Not Usually Inflated

The Centre for Open Science’s symposium on “Critical Perspectives on the Metascience Reform Movement” took place on 7
<sup>
 th
</sup>
March 2024. It was organised by Sven Ulpts, and it includes presentations by Bart Penders, Tom Hostler, Stephan Guttinger, Sarahanne Field, Nicole Nelson, and Berna Devezer.

Critical Perspectives on the Metascience Reform Movement

Abstract   Popper’s (1983, 2002) philosophy of science has enjoyed something of a renaissance in the wake of the replication crisis, offering a philosophical basis for the ensuing science reform movement. However, adherence to Popper’s approach may also be at least partly responsible for the sense of “crisis” that has developed following multiple unexpected replication failures.

The Replication Crisis is Less of a “Crisis” in the Lakatosian Approach than it is in the Popperian and Naïve Methodological Falsificationism Approaches

The December 2022 issue of the journal
<em>
 Industrial and Organizational Psychology
</em>
includes a very interesting discussion of the pros and cons of open science. There’s a target article titled “Open science, closed doors: The perils and potential of open science for research in practice” (Guzzo et al., 2022) and nine commentaries on this article. You can find all 10 papers here.

Opening Up Open Science to Epistemic Pluralism

I really enjoyed Mel Andrews’ recent essay: "Philosophy in the Trenches and Laboratory Benches of Science" and its main point that “every laboratory needs a philosopher.” This point is made in the context of a concern about “the industrialisation of science”:  This concern about the industrialisation of science reminded me of something that statistician Ronald Fisher wrote many years ago when he argued that the Neyman-Pearson approach to

The Industrialisation of Science

During multiple testing, researchers often adjust their alpha level to control the familywise error rate for a statistical inference about a joint union alternative hypothesis (e.g., “
<em>
 H1
</em>
or
<em>
 H2
</em>
”). However, in some cases, they do not make this inference.

Redundant Multiple Testing Corrections

In an article published last week in Synthese, philosopher of science Pekka Syrjänen asked “does a theory become better confirmed if it fits data that was not used in its construction versus if it was specifically designed to fit the data?” The first approach is called
<em>
 prediction,
</em>
and the second approach is called
<em>
 accommodation
</em>
. The debate over the epistemic advantages of prediction and accommodation has been bubbling away for

Prediction vs Accommodation: Which is Better and When?

<strong>
 <strong>
  An Intellectual Vocation
 </strong>
</strong>
In his book “The Soul of a University” (2018), Chris Brink describes the story of G.H. Hardy, a Cambridge Mathematician whose principled stance on his academic research was that it had no practical use whatsoever: “No discovery of mine” Hardy proudly wrote, “has made, or is likely to make, directly or indirectly, for good or ill, the least difference to the amenity of the world”

Tom Hostler

<strong>
 An Intellectual Vocation
</strong>

Scientific reform, post-academic research, and academic identity

Abstract   I discuss Van Drimmelen’s (2023) Metascience2023 presentation on researchers’ decision making during the research process.

The Preregistration Prescriptiveness Trade-Off and Unknown Unknowns in Science

Sabina Leonelli’s new book – “Philosophy of Open Science” – will be published later this year. However, there is an open access preprint available on the PhilSci Archive here. In the book, Leonelli provides “a constructively critical reading” of the standard approach to open science which, she argues, is focused on sharing “objects” such as data and materials.

Sabina Leonelli’s “Philosophy of Open Science”

Preregistration Distinguishes Between Exploratory and Confirmatory Research? Previous justifications for preregistration have focused on the distinction between “exploratory” and “confirmatory” research. However, as I discuss in this recent presentation, this distinction faces unresolved questions. For example, the distinction does not appear to have a formal definition in either statistical theory or the philosophy of science.

Preregistration, Severity, and Deviations

Tom Hostler has just published an article titled “The invisible workload of open research” in the
<em>
 Journal of Trial and Error’s
</em>
special issue on the “Consequences of the scientific reform movement: Is the scientific reform movement headed in the right direction?” It’s an insightful and thought-provoking piece that uncovers the potential workload costs of open research, considers why these costs may be ignored by university management,

Open Science and Overwork

A conference called “The promises and pitfalls of preregistration” was hosted by the Royal Society in London from 4th-5th March 2024. Here, I discuss the presentations by Chris Donkin and Stephan Lewandowsky, both of which consider some of the potential “pitfalls” of preregistration.

The Pitfalls of Preregistration

An often overlooked source of the “replication crisis” is the tendency to treat the replication study as a definitive verdict while ignoring the statistical uncertainty inherent in both the original and replication studies. This simplistic view fosters misleading dichotomies and erodes public trust in science.

Forcing a Deterministic View on Probabilistic Phenomena: Implications for the Replication Crisis

Critical Metascience

### Preregistration Distinguishes Between Exploratory and Confirmatory Research?

Previous justifications for preregistration have focused on the
distinction between "exploratory" and "confirmatory" research. However,
as I discuss in this [recent
presentation](https://www.youtube.com/watch?v=Gw_vn71esmA), this
distinction faces unresolved questions. For example, the distinction
does not appear to have a formal definition in either statistical theory
or the philosophy of science. In addition, critics have questioned
related concerns about the "double use" of data and "circular reasoning"
(Devezer et al., 2021; Rubin, 2020, 2022; Rubin & Donkin, 2022; Szollosi
& Donkin, 2021; see also Mayo, 1996, pp. 137, 271-275; Mayo, 2018, p.
319).

::: {.captioned-image-container}
<figure>
<a
href="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5875ff05-9e2c-4274-bab4-edeeefa9a073_1019x764.png"
class="image-link image2 is-viewable-imag" target="_blank"
data-component-name="Image2ToDOM"></a>
<div class="image2-inset">
<img
src="https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5875ff05-9e2c-4274-bab4-edeeefa9a073_1019x764.png"
class="sizing-normal"
data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5875ff05-9e2c-4274-bab4-edeeefa9a073_1019x764.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:764,&quot;width&quot;:1019,&quot;resizeWidth&quot;:590,&quot;bytes&quot;:278805,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false}"
srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5875ff05-9e2c-4274-bab4-edeeefa9a073_1019x764.png 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5875ff05-9e2c-4274-bab4-edeeefa9a073_1019x764.png 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5875ff05-9e2c-4274-bab4-edeeefa9a073_1019x764.png 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5875ff05-9e2c-4274-bab4-edeeefa9a073_1019x764.png 1456w"
sizes="100vw" data-fetchpriority="high" width="590" height="442" />
<div class="image-link-expand">
<img
src="data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIyMCIgaGVpZ2h0PSIyMCIgdmlld2JveD0iMCAwIDI0IDI0IiBmaWxsPSJub25lIiBzdHJva2U9ImN1cnJlbnRDb2xvciIgc3Ryb2tlLXdpZHRoPSIyIiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIGNsYXNzPSJsdWNpZGUgbHVjaWRlLW1heGltaXplMiAiPjxwb2x5bGluZSBwb2ludHM9IjE1IDMgMjEgMyAyMSA5Ij48L3BvbHlsaW5lPjxwb2x5bGluZSBwb2ludHM9IjkgMjEgMyAyMSAzIDE1Ij48L3BvbHlsaW5lPjxsaW5lIHgxPSIyMSIgeDI9IjE0IiB5MT0iMyIgeTI9IjEwIj48L2xpbmU+PGxpbmUgeDE9IjMiIHgyPSIxMCIgeTE9IjIxIiB5Mj0iMTQiPjwvbGluZT48L3N2Zz4="
class="lucide lucide-maximize2" />
</div>
</div>
</figure>
:::

### Preregistration Improves the Transparent Evaluation of Severity

Lakens and colleagues provide a more coherent justification for
preregistration based on Mayo's (1996, 2018) error statistical approach
(Lakens, 2019, 2024; Lakens et al., 2024; see also Vize et al., 2024).
Specifically, Lakens (2019) argues that "preregistration has the goal to
allow others to transparently evaluate the capacity of a test to falsify
a prediction, or the severity of a test" (p. 221).

A hypothesis passes a severe test when there is a high probability that
it would not have passed, or passed so well, if it was false (Mayo,
1996, 2018). A test procedure's error probabilities play an important
role in evaluating severity. In particular, "pre-data, the choices for
the type I and II errors reflect the goal of ensuring the test is
capable of licensing given inferences severely" (Mayo & Spanos, 2006, p.
350). For example, a test procedure with a nominal pre-data Type I error
rate of α = 0.05 is capable of licensing specific inferences with a
minimum "worst case" severity of 0.95 (i.e., 1 -- α; Mayo, 1996, p.
399).

Importantly, "biasing selection effects" in the experimental testing
context (e.g., *p*-hacking) can lower the capability of a test procedure
to license inferences severely by increasing the error probability with
which the procedure passes hypotheses. From this error statistical
perspective, preregistration allows a more transparent evaluation of the
capability of a test procedure to perform severe tests. In particular,
preregistration reveals a researcher's *planned* hypotheses, methods,
and analyses and enables a comparison with their *reported* hypotheses,
methods, and analyses in order to identify any biasing selection effects
in the experimental testing context that may increase the test
procedure's error probabilities and lower its capability for severe
tests.

### What Type of Severity?

Mayo's (1996, 2018) error statistical conceptualization of severity is
not the only one out there! Other types of severity have been proposed
by Bandyopadhyay and Brittan (2006), Hellman (1997, p. 198), Hitchcock
and Sober (2004, pp. 23-25), Horwich (1982, p. 105), Lakatos (1968, p.
382), Laudan (1997, p. 314), Popper (1962, 1983), and van Dongen et al.
(2023). Furthermore, preregistration may not facilitate the transparent
evaluation of these other types of severity. In my article, I illustrate
this point by showing that, although preregistration can facilitate the
transparent evaluation of Mayoian severity, it does not improve the
transparent evaluation of Popperian severity.

::: {.captioned-image-container}
<figure>
<a
href="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8566fc83-e8d9-4739-9629-ecbb054af961_1920x1080.png"
class="image-link image2 is-viewable-imag" target="_blank"
data-component-name="Image2ToDOM"></a>
<div class="image2-inset">
<img
src="https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8566fc83-e8d9-4739-9629-ecbb054af961_1920x1080.png"
class="sizing-normal"
data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8566fc83-e8d9-4739-9629-ecbb054af961_1920x1080.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:568,&quot;bytes&quot;:2508144,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false}"
srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8566fc83-e8d9-4739-9629-ecbb054af961_1920x1080.png 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8566fc83-e8d9-4739-9629-ecbb054af961_1920x1080.png 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8566fc83-e8d9-4739-9629-ecbb054af961_1920x1080.png 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8566fc83-e8d9-4739-9629-ecbb054af961_1920x1080.png 1456w"
sizes="100vw" loading="lazy" width="568" height="319" />
<div class="image-link-expand">
<img
src="data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIyMCIgaGVpZ2h0PSIyMCIgdmlld2JveD0iMCAwIDI0IDI0IiBmaWxsPSJub25lIiBzdHJva2U9ImN1cnJlbnRDb2xvciIgc3Ryb2tlLXdpZHRoPSIyIiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIGNsYXNzPSJsdWNpZGUgbHVjaWRlLW1heGltaXplMiAiPjxwb2x5bGluZSBwb2ludHM9IjE1IDMgMjEgMyAyMSA5Ij48L3BvbHlsaW5lPjxwb2x5bGluZSBwb2ludHM9IjkgMjEgMyAyMSAzIDE1Ij48L3BvbHlsaW5lPjxsaW5lIHgxPSIyMSIgeDI9IjE0IiB5MT0iMyIgeTI9IjEwIj48L2xpbmU+PGxpbmUgeDE9IjMiIHgyPSIxMCIgeTE9IjIxIiB5Mj0iMTQiPjwvbGluZT48L3N2Zz4="
class="lucide lucide-maximize2" />
</div>
</div>
</figure>
:::

I show that a valid measurement of Popperian severity can be made using
a potentially *p*-hacked result, a potentially HARKed hypothesis, and
potentially biased background knowledge. In addition, I show that
Popper's \"requirement of sincerity\" can be transparently evaluated
during a public critical rational discussion among scientists.
Preregistration does not facilitate a transparent evaluation in either
case because neither evaluation requires knowledge of the researcher's
planned approach or unreported biasing selection effects.

### Preregistration When Deviations are Allowed

I also argue that a preregistered test procedure that allows deviations
does not provide a more transparent evaluation of Mayoian severity than
a non-preregistered procedure. In particular, I consider deviations that
are intended to maintain or increase the validity of a test procedure in
light of unexpected issues that arise in particular samples of data
(e.g., a violation of the assumption of homogeneity). I argue that a
test procedure that allows these sample-based validity-enhancing
deviations in its implementation will suffer an unknown inflation of its
Type I error rate due to the [forking paths
problem](https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/the-garden-of-forking-paths?authuser=0)
(Gelman & Loken, 2013, 2014). Consequently, the test procedure will have
an unknown reduction of its capability to license inferences with
Mayoian severity.

::: {.captioned-image-container}
<figure>
<a
href="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feedd61bc-577e-411a-8326-90d77b80f4fa_693x662.png"
class="image-link image2 is-viewable-imag" target="_blank"
data-component-name="Image2ToDOM"></a>
<div class="image2-inset">
<img
src="https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feedd61bc-577e-411a-8326-90d77b80f4fa_693x662.png"
class="sizing-normal"
data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/eedd61bc-577e-411a-8326-90d77b80f4fa_693x662.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:662,&quot;width&quot;:693,&quot;resizeWidth&quot;:421,&quot;bytes&quot;:750415,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false}"
srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feedd61bc-577e-411a-8326-90d77b80f4fa_693x662.png 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feedd61bc-577e-411a-8326-90d77b80f4fa_693x662.png 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feedd61bc-577e-411a-8326-90d77b80f4fa_693x662.png 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feedd61bc-577e-411a-8326-90d77b80f4fa_693x662.png 1456w"
sizes="100vw" loading="lazy" width="421" height="402" />
<div class="image-link-expand">
<img
src="data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIyMCIgaGVpZ2h0PSIyMCIgdmlld2JveD0iMCAwIDI0IDI0IiBmaWxsPSJub25lIiBzdHJva2U9ImN1cnJlbnRDb2xvciIgc3Ryb2tlLXdpZHRoPSIyIiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIGNsYXNzPSJsdWNpZGUgbHVjaWRlLW1heGltaXplMiAiPjxwb2x5bGluZSBwb2ludHM9IjE1IDMgMjEgMyAyMSA5Ij48L3BvbHlsaW5lPjxwb2x5bGluZSBwb2ludHM9IjkgMjEgMyAyMSAzIDE1Ij48L3BvbHlsaW5lPjxsaW5lIHgxPSIyMSIgeDI9IjE0IiB5MT0iMyIgeTI9IjEwIj48L2xpbmU+PGxpbmUgeDE9IjMiIHgyPSIxMCIgeTE9IjIxIiB5Mj0iMTQiPjwvbGluZT48L3N2Zz4="
class="lucide lucide-maximize2" />
</div>
</div>
</figure>
:::

I conclude that preregistration does not improve the transparent
evaluation of severity in Popper's philosophy of science or when
deviations are allowed.

### The Article

Rubin, M. (2024). Preregistration does not improve the transparent
evaluation of severity in Popper's philosophy of science or when
deviations are allowed. *arXiv.*
<https://doi.org/10.48550/arXiv.2408.12347>

### References

Bandyopadhyay, P. S., & Brittan, G. G. (2006). Acceptibility, evidence,
and severity. *Synthese, 148*, 259-293.
<https://doi.org/10.1007/s11229-004-6222-6>

Gelman, A., & Loken, E. (2013). *The garden of forking paths: Why
multiple comparisons can be a problem, even when there is no "fishing
expedition" or "p-hacking" and the research hypothesis was posited ahead
of time.* Department of Statistics, Columbia University. Retrieved from
<http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf> 

Gelman, A., & Loken, E. (2014). The statistical crisis in science.
*American Scientist, 102*, 460.
<http://dx.doi.org/10.1511/2014.111.460> 

Devezer, B., Navarro, D. J., Vandekerckhove, J., & Ozge Buzbas, E.
(2021). The case for formal methodology in scientific reform. *Royal
Society Open Science, 8*(3). <https://doi.org/10.1098/rsos.200805>

Hellman, G. (1997). Bayes and beyond. *Philosophy of Science, 64,*
191--221. <https://doi.org/10.1086/392548>

Hitchcock, C., & Sober, E. (2004). Prediction versus accommodation and
the risk of overfitting. *British Journal for the Philosophy of
Science*, *55*(1), 1-34. <https://doi.org/10.1093/bjps/55.1.1>

Horwich, P. (1982). *Probability and evidence.* Cambridge University
Press.

Lakatos, I. (1968). Changes in the problem of inductive logic. *Studies
in Logic and the Foundations of Mathematics*, 51, 315-417.
[https://doi.org/10.1016/S0049-237X(08)71048-6](https://doi.org/10.1016/S0049-237X%2808%2971048-6)

Lakens, D. (2019). The value of preregistration for psychological
science: A conceptual analysis. *Japanese Psychological Review, 62*(3),
221--230. <https://doi.org/10.24602/sjpr.62.3_221>

Lakens, D. (2024). When and how to deviate from a preregistration.
*Collabra: Psychology*, 10(1): 117094.
<https://doi.org/10.1525/collabra.117094>

Lakens, D., Mesquida, C., Rasti, S., & Ditroilo, M. (2024). The benefits
of preregistration and Registered Reports. *Evidence-Based Toxicology,
2:*1, Article 2376046, <https://doi.org/10.1080/2833373X.2024.2376046>

Laudan, L. (1997). How about bust? Factoring explanatory power back into
theory evaluation. *Philosophy of Science, 64*(2), 306-316.
<https://doi.org/10.1086/392553>

Mayo, D. G. (1996). *Error and the growth of experimental knowledge*.
University of Chicago Press.

Mayo, D. G. (2018). *Statistical inference as severe testing: How to get
beyond the statistics wars*. Cambridge University Press.

Mayo, D. G., & Spanos, A. (2006). Severe testing as a basic concept in a
Neyman--Pearson philosophy of induction. *The British Journal for the
Philosophy of Science, 57*(2), 323-357.
<https://doi.org/10.1093/bjps/axl003>

Popper, K. R. (1962). *Conjectures and refutations: The growth of
scientific knowledge.* Routledge.

Popper, K. R. (1983). *Realism and the aim of science: From the
postscript to the logic of scientific discovery.* Routledge.

Rubin, M. (2020). Does preregistration improve the credibility of
research findings? *The Quantitative Methods for Psychology, 16*(4),
376--390. <https://doi.org/10.20982/tqmp.16.4.p376>

Rubin, M. (2022). The costs of HARKing. *British Journal for the
Philosophy of Science, 73*(2), 535-560.
*[](https://doi.org/10.1093/bjps/axz050)*
<https://doi.org/10.1093/bjps/axz050>

Rubin, M. (2024). Inconsistent multiple testing corrections: The fallacy
of using family-based error rates to make inferences about individual
hypotheses. *Methods in Psychology, 10,* Article 100140*.*
<https://doi.org/10.1016/j.metip.2024.100140>

Rubin, M., & Donkin, C. (2022). Exploratory hypothesis tests can be more
compelling than confirmatory hypothesis tests. *Philosophical
Psychology. [](https://doi.org/10.1080/09515089.2022.2113771)*
<https://doi.org/10.1080/09515089.2022.2113771>

Szollosi, A., & Donkin, C. (2021). Arrested theory development: The
misguided distinction between exploratory and confirmatory research.
*Perspectives on Psychological Science, 16*(4), 717-724.
https://doi.org/10.1177/1745691620966796

van Dongen, N., Sprenger, J., & Wagenmakers, E. J. (2023). A Bayesian
perspective on severity: Risky predictions and specific hypotheses.
*Psychonomic Bulletin & Review*, *30*(2), 516-533.
<https://doi.org/10.3758/s13423-022-02069-1>

Vize, C., Lynam, D., Miller, J., & Phillips, N. L. (2024, May 28). On
the use and misuses of preregistration: A reply to Klonsky (2024).
*PsyArXiv*. <https://doi.org/10.31234/osf.io/g7dn2>

[Share](https://markrubin.substack.com/p/preregistration-severity-and-deviations?utm_source=substack&utm_medium=email&utm_content=share&action=share){.button
.primary}

[Subscribe now](https://markrubin.substack.com/subscribe?){.button
.primary}
