Critical metascience takes a step back to question some common assumptions, approaches, problems, and solutions in metascience.

Mark Rubin

Felipe Romero presented his work - “The conceptual origins of metascience: Fashion or revolution?” - in The Popper Seminar at the London School of Economics on 30th May 2023. Here, I highlight a few points that he raised and provide some comments along the way. ::: {#youtube2-AKWCCdK-jgc .youtube-wrap attrs=“{"videoId":"AKWCCdK-jgc","startTime":null,"endTime":null}” component-name=“Youtube2ToDOM”} Ein Fehler ist aufgetreten.

What's Special About Metascience?

In this new article, I consider questionable research practices in the field of metascience. A questionable metascience practice (QMP) is a research practice, assumption, or perspective that’s been questioned by several commentators as being potentially problematic for metascience and/or the science reform movement. I discuss 10 QMPs that relate to criticism, replication, bias, generalization, and the characterization of science.

Questionable Metascience Practices

The inflation of Type I error rates is thought to be one of the causes of the replication crisis. Questionable research practices such as
<em>
 p
</em>
-hacking are thought to inflate Type I error rates above their nominal level, leading to unexpectedly high levels of false positives in the literature and, consequently, unexpectedly low replication rates. In this article, I offer an alternative view.

Type I Error Rates are Not Usually Inflated

The Centre for Open Science’s symposium on “Critical Perspectives on the Metascience Reform Movement” took place on 7
<sup>
 th
</sup>
March 2024. It was organised by Sven Ulpts, and it includes presentations by Bart Penders, Tom Hostler, Stephan Guttinger, Sarahanne Field, Nicole Nelson, and Berna Devezer.

Critical Perspectives on the Metascience Reform Movement

Abstract   Popper’s (1983, 2002) philosophy of science has enjoyed something of a renaissance in the wake of the replication crisis, offering a philosophical basis for the ensuing science reform movement. However, adherence to Popper’s approach may also be at least partly responsible for the sense of “crisis” that has developed following multiple unexpected replication failures.

The Replication Crisis is Less of a “Crisis” in the Lakatosian Approach than it is in the Popperian and Naïve Methodological Falsificationism Approaches

The December 2022 issue of the journal
<em>
 Industrial and Organizational Psychology
</em>
includes a very interesting discussion of the pros and cons of open science. There’s a target article titled “Open science, closed doors: The perils and potential of open science for research in practice” (Guzzo et al., 2022) and nine commentaries on this article. You can find all 10 papers here.

Opening Up Open Science to Epistemic Pluralism

I really enjoyed Mel Andrews’ recent essay: "Philosophy in the Trenches and Laboratory Benches of Science" and its main point that “every laboratory needs a philosopher.” This point is made in the context of a concern about “the industrialisation of science”:  This concern about the industrialisation of science reminded me of something that statistician Ronald Fisher wrote many years ago when he argued that the Neyman-Pearson approach to

The Industrialisation of Science

During multiple testing, researchers often adjust their alpha level to control the familywise error rate for a statistical inference about a joint union alternative hypothesis (e.g., “
<em>
 H1
</em>
or
<em>
 H2
</em>
”). However, in some cases, they do not make this inference.

Redundant Multiple Testing Corrections

In an article published last week in Synthese, philosopher of science Pekka Syrjänen asked “does a theory become better confirmed if it fits data that was not used in its construction versus if it was specifically designed to fit the data?” The first approach is called
<em>
 prediction,
</em>
and the second approach is called
<em>
 accommodation
</em>
. The debate over the epistemic advantages of prediction and accommodation has been bubbling away for

Prediction vs Accommodation: Which is Better and When?

<strong>
 <strong>
  An Intellectual Vocation
 </strong>
</strong>
In his book “The Soul of a University” (2018), Chris Brink describes the story of G.H. Hardy, a Cambridge Mathematician whose principled stance on his academic research was that it had no practical use whatsoever: “No discovery of mine” Hardy proudly wrote, “has made, or is likely to make, directly or indirectly, for good or ill, the least difference to the amenity of the world”

Tom Hostler

<strong>
 An Intellectual Vocation
</strong>

Scientific reform, post-academic research, and academic identity

Abstract   I discuss Van Drimmelen’s (2023) Metascience2023 presentation on researchers’ decision making during the research process.

The Preregistration Prescriptiveness Trade-Off and Unknown Unknowns in Science

Sabina Leonelli’s new book – “Philosophy of Open Science” – will be published later this year. However, there is an open access preprint available on the PhilSci Archive here. In the book, Leonelli provides “a constructively critical reading” of the standard approach to open science which, she argues, is focused on sharing “objects” such as data and materials.

Sabina Leonelli’s “Philosophy of Open Science”

Preregistration Distinguishes Between Exploratory and Confirmatory Research? Previous justifications for preregistration have focused on the distinction between “exploratory” and “confirmatory” research. However, as I discuss in this recent presentation, this distinction faces unresolved questions. For example, the distinction does not appear to have a formal definition in either statistical theory or the philosophy of science.

Preregistration, Severity, and Deviations

Tom Hostler has just published an article titled “The invisible workload of open research” in the
<em>
 Journal of Trial and Error’s
</em>
special issue on the “Consequences of the scientific reform movement: Is the scientific reform movement headed in the right direction?” It’s an insightful and thought-provoking piece that uncovers the potential workload costs of open research, considers why these costs may be ignored by university management,

Open Science and Overwork

A conference called “The promises and pitfalls of preregistration” was hosted by the Royal Society in London from 4th-5th March 2024. Here, I discuss the presentations by Chris Donkin and Stephan Lewandowsky, both of which consider some of the potential “pitfalls” of preregistration.

The Pitfalls of Preregistration

An often overlooked source of the “replication crisis” is the tendency to treat the replication study as a definitive verdict while ignoring the statistical uncertainty inherent in both the original and replication studies. This simplistic view fosters misleading dichotomies and erodes public trust in science.

Forcing a Deterministic View on Probabilistic Phenomena: Implications for the Replication Crisis

Critical Metascience

[Tom Hostler](https://www.researchgate.net/profile/Thomas-Hostler-2) has
just published an article titled "[The invisible workload of open
research](https://doi.org/10.36850/mr5)" in the *Journal of Trial and
Error's* special issue on the "[Consequences of the scientific reform
movement: Is the scientific reform movement headed in the right
direction](https://journal.trialanderror.org/special-issue-consequences-of-the-scientific-reform-movement)?"
It's an insightful and thought-provoking piece that uncovers the
potential workload costs of open research, considers why these costs may
be ignored by university management, and makes some useful
recommendations for moving forward. Here's a few quotes to whet your
appetite.

::: {.captioned-image-container}
<figure>
<a href="https://doi.org/10.36850/mr5" class="image-link image2"
target="_blank" data-component-name="Image2ToDOM"></a>
<div class="image2-inset">
<img
src="https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45026131-ae6b-4dac-8fbf-6f228053a4d9_1333x796.png"
class="sizing-normal"
data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/45026131-ae6b-4dac-8fbf-6f228053a4d9_1333x796.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:796,&quot;width&quot;:1333,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:184638,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://doi.org/10.36850/mr5&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null}"
srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45026131-ae6b-4dac-8fbf-6f228053a4d9_1333x796.png 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45026131-ae6b-4dac-8fbf-6f228053a4d9_1333x796.png 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45026131-ae6b-4dac-8fbf-6f228053a4d9_1333x796.png 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45026131-ae6b-4dac-8fbf-6f228053a4d9_1333x796.png 1456w"
sizes="100vw" data-fetchpriority="high" width="1333" height="796" />
<div>

</div>
</div>
</figure>
:::

#### **The Problem**

> The benefits of open research practices have been widely discussed
> (Munafò et al., 2017), and generally speaking researchers have
> positive attitudes towards adopting open research practices, finding
> them worthwhile (Eynden et al., 2016; Lowndes et al., 2017), and
> recommending their use to others (Sarafoglou et al., 2022). However,
> high time cost is repeatedly identified as one of the main barriers to
> the adoption of open practices (Eynden et al., 2016; Gownaris et al.,
> 2022; Tenopir et al., 2011) and time is the main thing that
> researchers lack, with workloads at capacity across the sector, having
> already reached "untenable" levels (Long et al., 2020; Papadopoulos,
> 2017).

In short, implementing open science reforms take work, and work takes
time, which academics have very little of!

#### **University Management are Not Motivated to Help!**

> The theory of academic capitalism suggests that the responsibility for
> addressing potential discrepancies between modelled and actual
> workload will not be taken up voluntarily by university management,
> who may at best ignore such issues, or at worse tacitly approve of
> them as a form of capitalist efficiency (Lyons & Ingersoll, 2010). In
> other words, if university management can choose not to incorporate
> open research into workload models, then they won't.

I don't think many working academics need to refer to the theory of
academic capitalism to appreciate this point! But to quell any doubt,
Hostler refers to empirical work by Kenny and Fluck
([2022](https://doi.org/10.1007/s10734-021-00747-y)) and Miller
([2019](https://doi.org/10.1080/1360080X.2019.1635328)). For example,
Kenny and Fluck (2022) found that:

> Management knows that academics will do research (and other work) in
> their own time and exploit that....What is written on paper in terms
> of hours re admin or research does not correspond to the actual time
> spent. It all looks fine on paper but has enormous hidden workloads.

Hence, Hostler concludes:

> There is a high potential for expectations to engage in open research
> to become widespread, but without additional time on academics'
> workloads dedicated to the activity.

#### **Some Open Research Advocates Do Not Acknowledge Open Research Workload**

> Suggestions that concerns about the workload of open research are an
> example of a "myth" (Bastiaansen, 2019) or a "misconception" that can
> be corrected by "positive advocacy" (Hagger, 2022) are unhelpful to
> having honest conversations about the practical negative consequences
> of conducting open research.

This is an important point. Some open science advocates engage in a
pretty hard sell, and this can include omitting or downplaying some of
the costs of open science. However, like Hostler says, it's worth having
"honest conversations" about the pros and cons of open science in order
to set more realistic expectations.

#### **A Key Recommendation**

I've just picked out one of Hostler's recommendations below to keep
things brief, but note that he makes several more in his article:

> When promoting open research reforms in universities or conducting
> open practices themselves \[open research advocates\] should advocate
> not only for investments in open research infrastructure and training
> but to receive extra time in workload allocations to acknowledge the
> additional burden of open research.

Getting open research work acknowledged in university workload models is
likely to be easier said than done! As Hostler explains, open research
work often consists of many small tasks, rather than a few large tasks,
and so it's easily overlooked in workload models. Nonetheless,
cumulatively, open research workload can be substantial and, as Hostler
notes, if it's ignored, we risk "death by a thousand 10-minute tasks"
(Bozeman et al., 2021).

::: {.captioned-image-container}
<figure>
<a
href="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F926ef13c-9658-42d0-a89f-5ea0abfb214f_583x500.jpeg"
class="image-link is-viewable-img image2" target="_blank"
data-component-name="Image2ToDOM"></a>
<div class="image2-inset">
<img
src="https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F926ef13c-9658-42d0-a89f-5ea0abfb214f_583x500.jpeg"
class="sizing-normal"
data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/926ef13c-9658-42d0-a89f-5ea0abfb214f_583x500.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:500,&quot;width&quot;:583,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:75314,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null}"
srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F926ef13c-9658-42d0-a89f-5ea0abfb214f_583x500.jpeg 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F926ef13c-9658-42d0-a89f-5ea0abfb214f_583x500.jpeg 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F926ef13c-9658-42d0-a89f-5ea0abfb214f_583x500.jpeg 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F926ef13c-9658-42d0-a89f-5ea0abfb214f_583x500.jpeg 1456w"
sizes="100vw" loading="lazy" width="583" height="500" />
<div class="image-link-expand">
<img
src="data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIxNiIgaGVpZ2h0PSIxNiIgdmlld2JveD0iMCAwIDI0IDI0IiBmaWxsPSJub25lIiBzdHJva2U9IiNGRkZGRkYiIHN0cm9rZS13aWR0aD0iMiIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIiBjbGFzcz0ibHVjaWRlIGx1Y2lkZS1tYXhpbWl6ZTIgIj48cG9seWxpbmUgcG9pbnRzPSIxNSAzIDIxIDMgMjEgOSI+PC9wb2x5bGluZT48cG9seWxpbmUgcG9pbnRzPSI5IDIxIDMgMjEgMyAxNSI+PC9wb2x5bGluZT48bGluZSB4MT0iMjEiIHgyPSIxNCIgeTE9IjMiIHkyPSIxMCI+PC9saW5lPjxsaW5lIHgxPSIzIiB4Mj0iMTAiIHkxPSIyMSIgeTI9IjE0Ij48L2xpbmU+PC9zdmc+"
class="lucide lucide-maximize2" />
</div>
</div>
</figure>
:::

#### The Article

Hostler, T. J. (2023). The invisible workload of open research. *Journal
of Trial & Error.* <https://doi.org/10.36850/mr5> 

::: {.subscription-widget-wrap attrs="{\"url\":\"https://markrubin.substack.com/subscribe?\",\"text\":\"Subscribe\",\"language\":\"en\"}" component-name="SubscribeWidgetToDOM"}
::: {.subscription-widget .show-subscribe}
::: {.preamble}
Thanks for reading Critical Metascience ! Subscribe for free to receive
new posts and support my work.
:::

::: {.fake-input-wrapper}
::: {.fake-input}
:::

::: {.fake-button}
:::
:::
:::
:::
