Adam Day

There’s this phrase used by some technologists: ‘epistemic security’. Epistemic security has to do with things like the spreading of misinformation on social media. I.e. if we spend enough time on social media reading unreliable information, how does that influence us? That’s a matter of epistemic security. Epistemic security is interesting because if you start a conversation about epistemic security at a party, they don’t invite you back.

The Papermill Alarm

I remember finding ChatGPT usage in a research paper for the first time. It was April 2023, and a Twitter user pointed out that you could identify bots by their use of the phrase “as an AI language model…”. It’s a hallmark of ChatGPT usage. So I searched Google Scholar for the phrase  … and found just one paper.

Detecting genAI: beside the point?

TL;DR: The current version of the Papermill Alarm detects signals in 98.9% of the Hindawi retractions conducted over the last 12 months.** It’s never nice to see the harms caused by papermills, but it is good to see independent verification of the Papermill Alarm’s predictions.

Here’s a question: Why are there 2 multi-squillion-euro detectors at CERN’s Large Hadron Collider?

Predicting retractions with the Papermill Alarm

It’s 1 January 2021. Dozing in bed, floating in that warm &amp; fuzzy limbo between dreams and reality, and that’s where it hits him. That lightbulb moment. (At last!) He sits bolt-upright in bed, punches the air and shouts: “You know what, honey, I’m gonna start a papermill!”. She snorts awake; reluctantly conscious. Oh no. Not again.

 Not another hare-brained scheme…

“…and, by the end of next year, we’ll publish 20,000 fake papers!”.

Impossible to ignore

TL;DR: We can detect individuals with a high probability of being involved in milling papers. The question is: how should we respond?  A few years ago, my bike was stolen. It was an organised job. The thieves arrived after dark, cut a hefty lock clean off, and my bike disappeared silently forever. I imagined it dismantled and sold off into some vast black market network. Poor bike. The police admitted that they weren’t going to do anything.

Keystone: papermill detection with network analysis

TL;DR: the rate of Papermill Alarm alerts is significantly higher in gold OA content than it is in subscription content. If you’re concerned about papermills, Clear Skies offers the Papermill Alarm: the world’s leading papermill detection tool. Get in touch for a demo. I’m a big believer in Open Science.

Papermills prefer Open Access

TL;DR: peer-review times appear to have been growing for a long time. The effects of COVID-lockdowns on peer-review are surprising.

A few months ago, I was invited to referee a research paper. So, I guess the editor thought that I was one of the 2 best people in the entire world to review this thing. That’s how it works, right? Flattered, I flagged the editor’s email to signal its importance and I got straight to work!

Is science getting slower?

This is

TL;DR: Join me at ConTech Live to hear about a recent project with Open Credo to see if we could detect unusual co-authorships in a dataset created by Anna Abalkina. Sign up here! Papermilling has a few definitions which you see here and there.

(The?) 3 kinds of papermills

TL;DR: There’s been a recent major upgrade to the private Papermill Alarm. It looks like we can predict the inclusion of journals on the Chinese Academy of Sciences (CAS) lists. You know how data scientists make predictions?

Cast a wide net

There’s a quote attributed to Ernest Rutherford: “That which is not measurable is not science. That which is not physics is stamp collecting”. I think his point was that a lot of scientific work is just

 documenting

things. In Rutherford’s day, there was a lot of exciting new

 creative

work happening in physics, so perhaps physics seemed special to him.

We need a chat about ChatGPT

Recently, I gave a presentation on the APIs for Papermill Detection offered by Clear Skies Ltd. I also touch on a newer service called the Clear Skies Standard Report. More on that in future posts… :)  Here’s the video:  https://medium.com/media/ee5a04aaed9fc53d2748e64516178ebe/href Would you like to know more?

Video: APIs for Papermill detection (ConTech API Series)

The first time I flew over the North Atlantic was quite an experience. Through the clouds, I could see some little white boats out sailing in the sea. It was puzzling: from 30,000 feet, those boats must have been huge for me to be able to see them at all.

How to find all the papermills (step 1)

This post is about The Papermill Alarm: an API for detecting potential papermill-products. There’s a field of study called ‘stylometry’ where we look at the statistical properties of someone’s writing and use that to model their ‘style’. People write in idiosyncratic ways.

The Papermill Alarm: Patterns in PubMed

The Papermill Alarm draws a line between papers which have the characteristics of papermill-products and those that don’t. Given data like that shown below, where red dots represent papermill-products and green dots represent normal papers, where should we draw that line?

Where to draw the line? Precision and recall

So, we’ve established that papermills like to use templates. We see templates in referee reports and in the text of cookie-cutter research papers. There’s an important insight here: Templates are used in legit academic behaviour as well as in industrial research fraud. So we need to be careful — we can’t assume that templates are necessarily a sign of research fraud.

Thank you for your warm work… earnestly

Last week, a paper I wrote on the subject of peer-review fraud was published in the journal

 Scientometrics

(free link here, preprint here)

 .

It was an interesting project to work on. I found a lot of examples where one referee would write a report during peer-review and then another referee would write an identical report in some other peer-review of some other paper.

Papermills in peer-review

Also… what is an API?

The Papermill Alarm API, is a service which you can send some article metadata to and which will return an alert telling you if the paper

 looks like

past papermill-products. Anyone can use it, but it definitely helps to have the support of an IT or data professional.

How to use the Papermill Alarm API

I once saw a brilliant presentation about how simple data analysis can detect credit card fraud**. The presentation showed a

 pattern

in how people use their credit cards. Given a large number of people who had been victims of credit card fraud, this pattern showed there was just 1 store in-particular where they had all used their cards. There was no

 observational evidence

of someone at that store stealing card details.

Patterns and Evidence (and how to find papermills before peer-review)

This is Ralph. How tall is Ralph? It seems simple, you could just hold a ruler up to the screen. But when you look at the ruler and use it to measure Ralph, are you actually measuring

 Ralph

, or are you measuring the ruler and using that as a

 proxy

? How accurate is your measurement? Are you including fur in the measurement? What if Ralph were to stand on his hind legs, like a mighty bear — how tall would he be then?

The definition of plagiarism

Stories by Adam Day on Medium

I once saw a brilliant presentation about how simple data analysis can
detect credit card fraud\*\*. The presentation showed a *pattern* in how
people use their credit cards. Given a large number of people who had
been victims of credit card fraud, this pattern showed there was just 1
store in-particular where they had all used their cards. There was no
*observational evidence* of someone at that store stealing card details.
No camera footage, no eye-witnesses, but the pattern was obvious in
the data.

The pattern is *circumstantial evidence*: a useful lead that tells
law-enforcement where to look for the *observational evidence*
they need.

<figure>
<img
src="https://cdn-images-1.medium.com/max/424/1*dRQZ1u6E11FXrouAsZdRVA.png" />
<figcaption><a
href="https://commons.wikimedia.org/wiki/File:Fingerprint-146242.svg">image
CC-0</a></figcaption>
</figure>

With misconduct detection, we are also often looking at patterns rather
than direct evidence of misconduct. Take plagiarism detection:

- We detect a pattern (which is an overlap in text between 2 documents)
- There are various levels to which it's ok to copy text with proper
 attribution and so simply finding overlapping text between 2 documents
 isn't evidence of plagiarism on its own (but I still think it's a good
 place to start!).
- Finding the pattern saves a lot of time by telling us when to look
 much more carefully at a pair of documents to get evidence of
 misconduct.

### But where to start with papermills?

Recently, I shared [a simple method to identify most of the existing
published papermill
content](https://publisherad.medium.com/how-to-find-all-the-papermills-step-1-f00c624facb1).
This method is particularly nice, because --- while we are detecting a
pattern (text duplication again) the investigation required to confirm
misconduct is as simple as it gets.

But, from a publisher's point of view, it would be much better to
identify papermills early on --- before peer-review. So I recently
released a simple API that does just that. The Papermill Alarm
identifies unusual patterns in the text of new submissions and issues a
warning if those patterns appear consistent with papermills.

> Here it is:
> <https://rapidapi.com/clear-skies-clear-skies-default/api/papermill-alarm/>

For example, you may have seen [Elisabeth Bik's excellent work on the
'tadpole'
papermill](https://scienceintegritydigest.com/2020/02/21/the-tadpole-paper-mill/).
This shows a very obvious pattern in the titles of several papers which
appeared to come from a papermill. [You can see similar patterns in the
many papers highlighted by Smut Clyde's insightful
work](https://forbetterscience.com/tag/smut-clyde/).

The Papermill Alarm

- receives article metadata (from you)
- detects patterns like those described above (and more)
- and returns a warning when it finds them.

Checking your papers is simple. The Papermill Alarm highlights papers
that you might want to check in-detail and greatly reduces the amount of
time you might have to spend looking for such cases.

That being said, I don't want to overstate what a tool like this can do.

- It doesn't find observational evidence, it detects patterns that are
 consistent with papermills.
- While results are already extremely useful, it's still an
 early-iteration of something with a lot of potential for development.

I'll say more about testing the Papermill Alarm in a future post. In the
meantime, if you have any questions at all, [get
in touch](https://twitter.com/AdamSci12)!

\*\* P.S. I found the presentation I mentioned at the start. It's
possible that one of the balding heads in the audience is mine!
<https://youtu.be/6eOr7lp4MPg?t=805>
<https://www.slideshare.net/tdunning/cheap-learningdunning9182015>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=469b3f206055){width="1"
height="1"}