This post provides an update on some recent new features in Oversight and walks through an investigation. Given the pace of development of Oversight, I might update it in future as things change. The problem is time. It’s time spent on misconduct investigations. Such investigations are often complex and sometimes inconclusive — fraud is intentionally hidden after all.
Oversight, by Clear Skies, is the first complete index of journal integrity. Oversight takes complex data on research integrity and enables a straightforward and intuitive understanding of every journal in the world. We rank journals based on analysis of data from our extensively field-tested integrity tools.
TL;DR: The Papermill Alarm has grown to become a comprehensive mix of methods for detecting organised research fraud and simplifying a complex problem. But there’s something new. Something bigger and better from Clear Skies.
“Disruption” is what happens when a different technology develops to a point where it does what yours does well enough to meet the needs of your customers. That means that, when customers switch, you can’t compete without changing your product completely. There’s a brilliant essay by Michael Nielsen from 2009 which predicted that disruption in academic publishing was imminent. It’s still worth a read, even now.
It’s 1 January 2021. Dozing in bed, floating in that warm & fuzzy limbo between dreams and reality, and that’s where it hits him. That lightbulb moment. (At last!) He sits bolt-upright in bed, punches the air and shouts: “You know what, honey, I’m gonna start a papermill!”. She snorts awake; reluctantly conscious. Oh no. Not again. Not another hare-brained scheme… “…and, by the end of next year, we’ll publish 20,000 fake papers!”.
TL;DR: AI-generated-text detection ceased to be a good indicator of research fraud at the exact time that it became a good tool for generating fraudulent research. I remember finding ChatGPT usage in a research paper for the first time. It was April 2023, and a Twitter user pointed out that you could identify bots by their use of the phrase “as an AI language model…”. It’s a hallmark of ChatGPT usage.
TL;DR: The current version of the Papermill Alarm detects signals in 98.9% of the Hindawi retractions conducted over the last 12 months.** It’s never nice to see the harms caused by papermills, but it is good to see independent verification of the Papermill Alarm’s predictions. Here’s a question: Why are there 2 multi-squillion-euro detectors at CERN’s Large Hadron Collider?
TL;DR: We can detect individuals with a high probability of being involved in milling papers. The question is: how should we respond? A few years ago, my bike was stolen. It was an organised job. The thieves arrived after dark, cut a hefty lock clean off, and my bike disappeared silently forever. I imagined it dismantled and sold off into some vast black market network. Poor bike. The police admitted that they weren’t going to do anything.
TL;DR: the rate of Papermill Alarm alerts is significantly higher in gold OA content than it is in subscription content. If you’re concerned about papermills, Clear Skies offers the Papermill Alarm: the world’s leading papermill detection tool. Get in touch for a demo. I’m a big believer in Open Science.
TL;DR: thousands of different authors use the same template emails to communicate with journals. Why? So, we’ve established that papermills like to use templates. We see templates in referee reports and in the text of cookie-cutter research papers. There’s an important insight here:Templates are used in legit academic behaviour as well as in industrial research fraud.
There’s this phrase used by some technologists: ‘epistemic security’. Epistemic security has to do with things like the spreading of misinformation on social media. I.e. if we spend enough time on social media reading unreliable information, how does that influence us? That’s a matter of epistemic security. Epistemic security is interesting because if you start a conversation about epistemic security at a party, they don’t invite you back.