Critical metascience takes a step back to question some common assumptions, approaches, problems, and solutions in metascience.

Mark Rubin

Felipe Romero presented his work - “The conceptual origins of metascience: Fashion or revolution?” - in The Popper Seminar at the London School of Economics on 30th May 2023. Here, I highlight a few points that he raised and provide some comments along the way. ::: {#youtube2-AKWCCdK-jgc .youtube-wrap attrs=“{"videoId":"AKWCCdK-jgc","startTime":null,"endTime":null}” component-name=“Youtube2ToDOM”} Ein Fehler ist aufgetreten.

What's Special About Metascience?

In this new article, I consider questionable research practices in the field of metascience. A questionable metascience practice (QMP) is a research practice, assumption, or perspective that’s been questioned by several commentators as being potentially problematic for metascience and/or the science reform movement. I discuss 10 QMPs that relate to criticism, replication, bias, generalization, and the characterization of science.

Questionable Metascience Practices

The inflation of Type I error rates is thought to be one of the causes of the replication crisis. Questionable research practices such as
<em>
 p
</em>
-hacking are thought to inflate Type I error rates above their nominal level, leading to unexpectedly high levels of false positives in the literature and, consequently, unexpectedly low replication rates. In this article, I offer an alternative view.

Type I Error Rates are Not Usually Inflated

The Centre for Open Science’s symposium on “Critical Perspectives on the Metascience Reform Movement” took place on 7
<sup>
 th
</sup>
March 2024. It was organised by Sven Ulpts, and it includes presentations by Bart Penders, Tom Hostler, Stephan Guttinger, Sarahanne Field, Nicole Nelson, and Berna Devezer.

Critical Perspectives on the Metascience Reform Movement

Abstract   Popper’s (1983, 2002) philosophy of science has enjoyed something of a renaissance in the wake of the replication crisis, offering a philosophical basis for the ensuing science reform movement. However, adherence to Popper’s approach may also be at least partly responsible for the sense of “crisis” that has developed following multiple unexpected replication failures.

The Replication Crisis is Less of a “Crisis” in the Lakatosian Approach than it is in the Popperian and Naïve Methodological Falsificationism Approaches

The December 2022 issue of the journal
<em>
 Industrial and Organizational Psychology
</em>
includes a very interesting discussion of the pros and cons of open science. There’s a target article titled “Open science, closed doors: The perils and potential of open science for research in practice” (Guzzo et al., 2022) and nine commentaries on this article. You can find all 10 papers here.

Opening Up Open Science to Epistemic Pluralism

I really enjoyed Mel Andrews’ recent essay: "Philosophy in the Trenches and Laboratory Benches of Science" and its main point that “every laboratory needs a philosopher.” This point is made in the context of a concern about “the industrialisation of science”:  This concern about the industrialisation of science reminded me of something that statistician Ronald Fisher wrote many years ago when he argued that the Neyman-Pearson approach to

The Industrialisation of Science

During multiple testing, researchers often adjust their alpha level to control the familywise error rate for a statistical inference about a joint union alternative hypothesis (e.g., “
<em>
 H1
</em>
or
<em>
 H2
</em>
”). However, in some cases, they do not make this inference.

Redundant Multiple Testing Corrections

In an article published last week in Synthese, philosopher of science Pekka Syrjänen asked “does a theory become better confirmed if it fits data that was not used in its construction versus if it was specifically designed to fit the data?” The first approach is called
<em>
 prediction,
</em>
and the second approach is called
<em>
 accommodation
</em>
. The debate over the epistemic advantages of prediction and accommodation has been bubbling away for

Prediction vs Accommodation: Which is Better and When?

<strong>
 <strong>
  An Intellectual Vocation
 </strong>
</strong>
In his book “The Soul of a University” (2018), Chris Brink describes the story of G.H. Hardy, a Cambridge Mathematician whose principled stance on his academic research was that it had no practical use whatsoever: “No discovery of mine” Hardy proudly wrote, “has made, or is likely to make, directly or indirectly, for good or ill, the least difference to the amenity of the world”

Tom Hostler

<strong>
 An Intellectual Vocation
</strong>

Scientific reform, post-academic research, and academic identity

Abstract   I discuss Van Drimmelen’s (2023) Metascience2023 presentation on researchers’ decision making during the research process.

The Preregistration Prescriptiveness Trade-Off and Unknown Unknowns in Science

Sabina Leonelli’s new book – “Philosophy of Open Science” – will be published later this year. However, there is an open access preprint available on the PhilSci Archive here. In the book, Leonelli provides “a constructively critical reading” of the standard approach to open science which, she argues, is focused on sharing “objects” such as data and materials.

Sabina Leonelli’s “Philosophy of Open Science”

Preregistration Distinguishes Between Exploratory and Confirmatory Research? Previous justifications for preregistration have focused on the distinction between “exploratory” and “confirmatory” research. However, as I discuss in this recent presentation, this distinction faces unresolved questions. For example, the distinction does not appear to have a formal definition in either statistical theory or the philosophy of science.

Preregistration, Severity, and Deviations

Tom Hostler has just published an article titled “The invisible workload of open research” in the
<em>
 Journal of Trial and Error’s
</em>
special issue on the “Consequences of the scientific reform movement: Is the scientific reform movement headed in the right direction?” It’s an insightful and thought-provoking piece that uncovers the potential workload costs of open research, considers why these costs may be ignored by university management,

Open Science and Overwork

A conference called “The promises and pitfalls of preregistration” was hosted by the Royal Society in London from 4th-5th March 2024. Here, I discuss the presentations by Chris Donkin and Stephan Lewandowsky, both of which consider some of the potential “pitfalls” of preregistration.

The Pitfalls of Preregistration

An often overlooked source of the “replication crisis” is the tendency to treat the replication study as a definitive verdict while ignoring the statistical uncertainty inherent in both the original and replication studies. This simplistic view fosters misleading dichotomies and erodes public trust in science.

Forcing a Deterministic View on Probabilistic Phenomena: Implications for the Replication Crisis

Critical Metascience

An often overlooked source of the "replication crisis" is the tendency
to treat the replication study as a definitive verdict while ignoring
the statistical uncertainty inherent in both the original and
replication studies. This simplistic view fosters misleading dichotomies
and erodes public trust in science. By analyzing the media coverage of
the Open Science Collaboration's "Reproducibility Project: Psychology"
(RPP), [this commentary](https://doi.org/10.1177/10755470241239947)
draws attention to this common neglect of statistical uncertainty and
highlights the need to discuss replication studies in the context of
cumulative evidence.[1](#footnote-1){#footnote-anchor-1 .footnote-anchor
component-name="FootnoteAnchorToDOM" target="_self"}

::: {.captioned-image-container}
<figure>
<a href="https://doi.org/10.1177/10755470241239947"
class="image-link image2" target="_blank"
data-component-name="Image2ToDOM"></a>
<div class="image2-inset">
<img
src="https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2b7b34c6-5ea6-4edf-a9e9-89e97ff3f4e7_659x663.png"
class="sizing-normal"
data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2b7b34c6-5ea6-4edf-a9e9-89e97ff3f4e7_659x663.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:663,&quot;width&quot;:659,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:48259,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://doi.org/10.1177/10755470241239947&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null}"
srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2b7b34c6-5ea6-4edf-a9e9-89e97ff3f4e7_659x663.png 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2b7b34c6-5ea6-4edf-a9e9-89e97ff3f4e7_659x663.png 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2b7b34c6-5ea6-4edf-a9e9-89e97ff3f4e7_659x663.png 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2b7b34c6-5ea6-4edf-a9e9-89e97ff3f4e7_659x663.png 1456w"
sizes="100vw" data-fetchpriority="high" width="659" height="663" />
<div>

</div>
</div>
</figure>
:::

### Method Uncertainty vs. Statistical Uncertainty {#method-uncertainty-vs.-statistical-uncertainty}

*Method uncertainty* encompasses the variability of outcomes arising
from differences in research and analysis methods. The most recognized
source of method uncertainty is the varying extent of control of
potentially confounding factors. Additionally, method uncertainty can
also arise from the criteria for selection into the study and the
implementation of statistical procedures, such as how variables are
coded.

The commentary centers on the second kind of uncertainty, statistical
uncertainty, which holds a prominent position in biology and human
sciences but is unfortunately often overlooked in interpretations of
results. Unlike phenomena studied in physical sciences, which are
governed by universal and exact laws, those involving complex living
organisms are characterized by fuzzy boundaries between categories,
intricate feedback loops, and complex natural variation (Mayr, 1985;
Nelson, 2016). This added layer of complexity, along with the sampling
variation and "noise" it introduces --modeled as "statistical
uncertainty" -- is what renders biomedicine, psychology, and social
sciences "soft", as they explore phenomena whose regularities are
amorphous and situational.

### Forcing a Deterministic Frame on Continuous Probabilistic Phenomena

Unfortunately, our desire for certainty misleads us to approach
statistics as if it could eliminate all uncertainty and to treat
hypothesis testing as if it could transform a continuous probabilistic
phenomenon into a deterministic dichotomy (statistically significant vs.
statistically insignificant, Gelman, 2016; Gelman & Loken, 2014).
However, neither dichotomizing results (based on whether single
*p*-values exceed a cutoff value) nor using isolated *p*-values for
decision or judgment is justified. This is because the *p*-value is not
only a function of what would be purely random error under the test
hypothesis, but also depends on all other assumptions used to compute
them (Greenland, 2019). The latter assumptions cannot be assured in soft
sciences, which typically deal with small effect sizes and deviations
from assumptions that to lead uncontrolled biases (Greenland, 2017).
Consequently, there is a low ratio of true signal (effect) to random
noise and bias, with the bias resulting in more erroneous decisions than
indicated by the statistical theory. The dichotomous deterministic usage
of *p*-values thus can be misleading, contributing to the "replication
crisis" in two key ways.

Firstly, even in the presence of a true effect, the likelihood of a pair
of (original and replication) studies yielding "inconsistent" results is
a function of power and is exaggerated at typical power levels: For a
power level lower than 50%, which is typical in soft sciences (Bakker et
al., 2012; Button et al., 2013; van Zwet et al., 2024), the likelihood
of apparently conflicting results can be as high as 50% in the presence
of a true effect. At a 20% power level, the rate of observing
conflicting results is still 32% given a true effect. Surely, a 32% to
50% rate of conflict is a gross exaggeration when no actual conflict
exists.

Secondly, this dichotomous deterministic frame also exacerbates the
"replication crisis" through journals' selective publication of
statistically significant results ("the publication bias"). When
statistical power is low, only the most extreme results will clear the
"statistically significant" bar (Amrhein et al., 2019; Gelman & Carlin,
2014; van Zwet et al., 2021), resulting in a literature biased with
inflated effect sizes. The inflated effect sizes in published literature
in turn misguide the design of replication studies and lead to
underpowered studies. Moreover, as initial results in literature tend to
be outliers, results of replication studies tend to regress to the mean.
Together, underpowered studies and regression to the mean further
increase the likelihood of initially positive findings to "fail to
replicate" in subsequent studies (Maxwell et al., 2015; van Zwet et al.,
2021, 2024).

A re-examination of the RPP results can illustrate this problem (Amhrein
et al.,2019): assuming a more realistic 50% power (instead of the 92%
used in the RPP), the "low replication rate" of 36% observed in the RPP
could be observed even if only 30% of the original studies were false
positives. This should caution us against the hasty conclusion that most
or all the original studies are false positives, and warn us that many
"replication failures" arise from the statistical variability inherent
in both the replication and original studies.

To demonstrate this problem, the commentary includes an analysis of the
media coverage over the RPP (59 non-duplicated news reports and 15
opinion pieces published in 2015 from a total of 52 sources). The
analysis reveals a clear pattern: while almost all reports and opinion
pieces cover method uncertainty, only one report and two opinion pieces
touch upon statistical uncertainty. Notably, among ten experts who
expressed reservations about RPP outcomes, only one interviewed expert
pointed to statistical uncertainty as an explanation for "failed"
replication, further highlighting statistical uncertainty as a major
blind spot in pertinent public discussion. Additionally, the analysis
documented several noteworthy instances where reports implicitly
associated "non-replication" with false positives and even insinuated
fraud. Such labeling is not only premature but also undermines public
trust in science, which is one of the biggest challenges facing science
today.

### Discussion

Replication projects and follow-up efforts are a partial solution to
rebuilding science's credibility, but cannot succeed without tackling
the problem of statistical misconceptions. As demonstrated, blind spots
regarding statistical uncertainty in replication projects exaggerate
irreplicability and ultimately undermine trust. Thus, to reach
trustworthy conclusions, we not only need high-quality data from sound
studies, but we also need to be patient for studies (the data points) to
accumulate enough to provide a clear picture of actual effects and their
variations across study settings. There is no short cut, and hastily
drawing conclusions based on a single replication study is just as
unwise as blindly trusting the original study.

### References

Amrhein, V., Trafimow, D., & Greenland, S. (2019). Inferential
statistics as descriptive statistics: There is no replication crisis if
we don't expect replication. *The American Statistician, 73*(sup1),
262--270. <https://doi.org/10.1080/00031305.2018.1543137>

Bakker, M., Dijk, A. van, & Wicherts, J. M. (2012). The rules of the
game called psychological science. *Perspectives on Psychological
Science, 7*(6), 543--554. <https://doi.org/10.1177/1745691612459060>

Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J.,
Robinson, E. S., & Munafò, M. R. (2013). Power failure: Why small sample
size undermines the reliability of neuroscience. *Nature Reviews
Neuroscience, 14*(5), 365--376. <https://doi.org/10.1038/nrn3475>

Gelman, A. (2016). The problems with *p*-values are not just with
*p*-values. *The American Statistician, supplemental materials to ASA
Statement on p-Values and Statistical Significance, 70,* 1--2.
<http://www.stat.columbia.edu/~gelman/research/published/asa_pvalues.pdf>

Gelman, A., & Carlin, J. (2014). Beyond power calculations.
*Perspectives on Psychological Science, 9*(6), 641--651.
<https://doi.org/10.1177/1745691614551642>

Gelman, A., & Loken, E. (2014). The garden of forking paths: Why
multiple comparisons can be a problem, even when there is no "fishing
expedition" or "p-hacking" and the research hypothesis was posited ahead
of time. [https://statmodeling.stat.
columbia.edu/2021/03/16/the-garden-of-forking-paths-why-multiple-compari-
sons-can-be-a-problem-even-when-there-is-no-fishing-expedition-or-p-hacking-
and-the-research-hypothesis-was-posited-ahead-of-time-2/](https://statmodeling.stat.%20columbia.edu/2021/03/16/the-garden-of-forking-paths-why-multiple-compari-%20sons-can-be-a-problem-even-when-there-is-no-fishing-expedition-or-p-hacking-%20and-the-research-hypothesis-was-posited-ahead-of-time-2/)

Greenland, S. (2017). The need for cognitive science in methodology.
*American Journal of Epidemiology, 186*, 639--645.
[https://academic.oup.com/aje/arti-
cle/186/6/639/3886035](https://academic.oup.com/aje/arti-%20cle/186/6/639/3886035)

Greenland, S. (2019). Valid *p*-values behave exactly as they should:
Some misleading criticisms of *p*-values and their resolution with
S-values. *The American Statistician, 73(Suppl. 1),* 106--114.
[https://doi.org/10.1080/00031305.2018.152
9625](https://doi.org/10.1080/00031305.2018.152%209625)

Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015). Is psychology
suffering from a replication crisis? What does "failure to replicate"
really mean? *American Psychologist, 70(6)*, 487-498.
<https://doi.org/10.1037/a0039400>

Mayr, E. (1985). How biology differs from the physical sciences. In D.
Depew & B. Weber (Eds.), *Evolution at a crossroads: The new biology and
the new philosophy of science* (pp. 43--63). A Bradford Book.

Nelson, R. R. (2016). The sciences are different and the differences
matter. *Research Policy, 45*(9), 1692--1701.
<https://doi.org/10.1016/j.respol.2015.05.014>

Van Zwet, E., Schwab, S., & Greenland, S. (2021). Addressing
exaggeration of effects from single RCTs. *Significance, 18*(6), 16--21.
<https://doi.org/10.1111/1740-9713.01587>

### The Article

Ting. C., & Greenland, S. (2024). Forcing a deterministic frame on
probabilistic phenomena: a communication blind spot in media coverage of
the "replication crisis." *Science Communication*.
<https://doi.org/10.1177/10755470241239947>

For an open access version, please click
[here](https://www.researchgate.net/profile/Carol-Ting/publication/379598105_Title_Forcing_a_Deterministic_Frame_on_Probabilistic_Phenomena_A_Communication_Blind_Spot_in_Media_Coverage_of_the_Replication_Crisis_Forcing_a_deterministic_frame_on_probabilistic_phenomena_A_communi/links/6612b4b07476d47e4445a29b/Title-Forcing-a-Deterministic-Frame-on-Probabilistic-Phenomena-A-Communication-Blind-Spot-in-Media-Coverage-of-the-Replication-Crisis-Forcing-a-deterministic-frame-on-probabilistic-phenomena-A-communi.pdf).

### Related Work

Ting, C., & Montgomery, M. (2023). Taming human subjects: researchers'
strategies for coping with vagaries in social science experiments.
*Social Epistemology,* 1-17.
<https://doi.org/10.1080/02691728.2023.2177128>

### About the Author

Carol Ting is an assistant professor at the Department of Communication,
University of Macau. Her research interest focuses on social science
methodology and, in particular, factors complicating research
replication/reproducibility. Her recent work was published in Social
Epistemology, the International Journal of Social Research Methodology,
and the Social Science Journal. For more information about Carol's work,
please see: <https://comm.fss.um.edu.mo/prof-carol-ting/>

::: {.captioned-image-container}
<figure>
<a
href="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa80bf486-3e2f-4d92-a62f-3e57fa235676_400x652.jpeg"
class="image-link is-viewable-img image2" target="_blank"
data-component-name="Image2ToDOM"></a>
<div class="image2-inset">
<img
src="https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa80bf486-3e2f-4d92-a62f-3e57fa235676_400x652.jpeg"
class="sizing-normal"
data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a80bf486-3e2f-4d92-a62f-3e57fa235676_400x652.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:652,&quot;width&quot;:400,&quot;resizeWidth&quot;:228,&quot;bytes&quot;:43703,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null}"
srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa80bf486-3e2f-4d92-a62f-3e57fa235676_400x652.jpeg 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa80bf486-3e2f-4d92-a62f-3e57fa235676_400x652.jpeg 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa80bf486-3e2f-4d92-a62f-3e57fa235676_400x652.jpeg 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa80bf486-3e2f-4d92-a62f-3e57fa235676_400x652.jpeg 1456w"
sizes="100vw" loading="lazy" width="228" height="371" />
<div class="image-link-expand">
<img
src="data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIyMCIgaGVpZ2h0PSIyMCIgdmlld2JveD0iMCAwIDI0IDI0IiBmaWxsPSJub25lIiBzdHJva2U9ImN1cnJlbnRDb2xvciIgc3Ryb2tlLXdpZHRoPSIyIiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiIGNsYXNzPSJsdWNpZGUgbHVjaWRlLW1heGltaXplMiAiPjxwb2x5bGluZSBwb2ludHM9IjE1IDMgMjEgMyAyMSA5Ij48L3BvbHlsaW5lPjxwb2x5bGluZSBwb2ludHM9IjkgMjEgMyAyMSAzIDE1Ij48L3BvbHlsaW5lPjxsaW5lIHgxPSIyMSIgeDI9IjE0IiB5MT0iMyIgeTI9IjEwIj48L2xpbmU+PGxpbmUgeDE9IjMiIHgyPSIxMCIgeTE9IjIxIiB5Mj0iMTQiPjwvbGluZT48L3N2Zz4="
class="lucide lucide-maximize2" />
</div>
</div>
</figure>
:::

:::: {.footnote component-name="FootnoteToDOM"}
[1](#footnote-anchor-1){#footnote-1 .footnote-number
contenteditable="false" target="_self"}

::: {.footnote-content}
Summary prepared by Carol Ting, who takes sole responsibility for all
errors within it.

[Share](https://markrubin.substack.com/p/forcing-a-deterministic-view-on-probabilistic?utm_source=substack&utm_medium=email&utm_content=share&action=share){.button
.primary}

[Subscribe now](https://markrubin.substack.com/subscribe?){.button
.primary}
:::
::::
