Fighting paper mills

(This article was first published in Norwegian in The Magazine Research Ethics no. 1, 2020)

«It was new to me that an entire article could be fake! It is very sad. The situation was bad enough in China already, but it turns out to be much worse than I thought», says «Morty».

Paper mills is the nickname given to shady companies that appear to produce articles from start to finish; from creating fake data and writing scientific papers to publishing in a reputable international journal.

The phenomenon has been known for a few years, but in recent months the true scope has been uncovered. Together with a small group of unpaid self-employed sleuths fighting for research integrity, «Morty» has helped finding one article after another, all connected to the same paper mill in China.

«The situation is unclear. We believe that what we have found so far is only a small part of a comprehensive, systematic falsification of research literature», says «Morty».

Until now he has considered it would be far too time-consuming and probably detrimental to his career to go public with his efforts to clean up other people's research. So, he is hiding behind the pseudonym «Morty» when he tweets about findings and suspicions, and also in this article.

Scientific detectives

Morty works at a major medical research institution in Norway. Since 2013 he has spent much of his spare time on a kind of scientific detective work, sitting on his computer looking for papers with problematic images.

These are usually not «images» the way we use the term in everyday speech, but illustrations and figures that show biomedical research data: proteins examined by a method called Western blotting, characteristics of cells measured by flow cytometry and plotted into histograms, and cells and tissue samples shown in microscope images.

Morty explains that duplications where the exact same image is used multiple times, is the most obvious form of problematic image use. For example, the same image can be used to illustrate different experiments.

«If the image itself has not been changed in any way, this may only be an honest mistake where the scientists inadvertently have used the wrong image», he says.

But there is also falsification, fabrication and outright misconduct. For example, a section of a microscope image can be both rotated, flipped and enlarged, and then placed into another image again.

«In such cases there is no doubt,» Morty sums up.

Photoshopped proteins

The researcher came across the Chinese paper mill (see fact box) accidentally.

«Ironically, I found the first two examples while looking at randomly selected papers to investigate if the problem of data duplications has been reduced,» he says.

He posted his findings on PubPeer, a discussion forum for researchers (see fact box). He also discussed the examples with his small group of image sleuths, an informal team that mainly communicates via Twitter. (Check out relevant links for the Twitter accounts.)

«Another team member, «Smut», immediately found several similar examples,» says Morty.

The articles that Morty and his partners believe have been produced by the same paper mill have several distinctive features. But the most obvious one is that they all share the similar Western blot images.

Sitting in front of his computer looking for more papers to add to the paper mill spreadsheet, Morty soon finds something.

«Check this out. This immediately looks shady. The information from this Western blot is meaningless. Those bands should not be as fat, they are overexposed. They are also diffuse, that's a bad sign and makes me suspicious right away.»

The proteins are expressed as black bands in different sizes, shapes and shades. In principle, none of them should be identical.

With eyes accustomed to scanning endless pages, Morty and his colleagues quickly discover repeating patterns and shapes. Using Photoshop they can also determine if the figures have another distinctive feature:

«If we adjust the contrast, the background becomes more apparent. In the paper mill articles we see that the background is quite similar, both from figure to figure as well as across different articles. That is unnatural,» Morty explains.

The article he is scrolling through now is published in the Journal of Cellular Biochemistry, a relatively reputable journal. He just needs to refine his gaze to determine that the protein picture he is investigating has been manipulated.

«This will be added to the list,» he concludes.

Has scanned 60.000 papers

On the other side of the Atlantic, in San Francisco, one of Morty's allies also sits in front of her computer staring intently at articles that could potentially origin from a Chinese paper mill.

It's almost midnight, but Elisabeth Bik still has several tasks to finish before she turns her computer off. One of them is an interview with The Magazine Research Ethics. She has quit her job as a scientist to become an unpaid full-time image sleuth, but the day still isn't long enough.

«It's tough living without a fixed income, but I can do more for science now than when I was an employee,» Bik says.

In 2016, she and two other researchers published an article on the occurrence of inappropriate image duplication in more than 20,000 biomedical research articles. These articles came from 40 journals in the period 1995-2014. Problematic images are alarmingly common, they concluded.

«Almost four percent of the articles contained problematic images, especially related to Western blots. At least half of these images had features that suggested intentional manipulation,» says Bik.

Since then, Bik's eyes have scanned endless numbers of images. She estimates that she is now up to 60,000 reviewed articles, and a total of 2,000 duplicated images.

«I guess I have a good eye for this. Many say we need software to screen articles for duplicates. But to me, in many cases, a duplication is as obvious as determining that the thing you see in a picture is the Eiffel Tower.»

Ringing the alarm

While Morty and the other two in the small image sleuth group are anonymous, Elisabeth Bik is the one who personifies their work. She admits to sometimes thinking "shoot, I should have done this anonymously". Like the time that a Chinese top scientist with close ties to the government was responsible for the articles she was questioning on PubPeer.

Instead of creating difficulties for Bik, the authorities surprisingly allowed Chinese media to discuss the matter. The researcher himself replied to a few of the fifty posts on PubPeer and explained that there had been a mistake.

«But most of the cases seemed to involve image manipulation, and he did not comment to those. I contacted about 30 journals. So far none of the articles have been retracted,» says Bik.

The paper mill case that has been uncovered now is of a different magnitude.

«I am ringing the alarm,» Bik wrote on Twitter in late February, telling her followers about more than 400 articles found by then. She also pointed out that they had found dozens of other articles that may be part of other paper mills. «I'm worried there might be hundreds of papers more,» she continued.

At the time of writing, the number of articles from the large paper mill has reached 439.

Wasted time and money

Morty's interest in errors and fraud in research was prompted when the PubPeer online forum was established in 2012. At the same time, he encountered obstacles in his own research.

«In relevant articles I sometimes noticed strange things, like the same picture being used twice. I repeatedly tried to reproduce the research results of others unsuccessfully. I spent time and money on this, and became frustrated and angry,» he says.

Over the years, he has realized the situation is much worse than he had thought.

«The disclosures of the paper mills show that we need fundamental changes in scientific publishing,» he claims.

Morty believes the system has failed completely: The institutions have not checked their research. The peer reviewers should have stopped the articles before they were published, and the quality assurance by the scholarly publishers hasn't been good enough.

(Read the reply from Wiley: Wiley increases its screening activity)

The National Library of Medicine in the United States is the gatekeeper for the articles that find their way into the largest medical research databases, MedLine and PubMed. They've let too many articles pass, according to Morty.

(Read the reply from The National Library of Medicine: NLM says it takes its role seriously)

A «WADA for science»

Elisabeth Bik believes inappropriate images in general is a major problem.

«New research is based on previous research by referring to published articles. The articles are like bricks in a building. If any of the bricks are not solid, the entire building may collapse,» she illustrates.

The paper mill case has made her more concerned about research in general, and now she fears that a large, unknown proportion of the research literature is completely fabricated. Meanwhile, the image sleuths come across increasingly sophisticated manipulation that is difficult to detect.

Both Bik and Morty believe that large efforts are needed to put a stop to the paper mills in China and to get the entire research field back on track. They hope their contribution can raise awareness.

«I want to show how little publishers do to address the situation. Several years ago, I reported possible errors in a total of 800 articles. Only one in three has been corrected or retracted. I usually spend a few seconds discovering these things, so it shouldn't take five years to reach a conclusion,» says Bik.

She calls for more courageous editors who not only take whistle blowers more seriously but also take responsibility for cleaning up. Published articles must be reviewed with a critical eye. But most importantly - the journals must hire professional people to scan articles for problematic images before publishing.

«In this way issues can be resolved - or papers can be rejected, before other scientists see them and try to replicate them,» says Bik.

Morty believes that a supervisory body at the top level is needed. He draws parallels to the WADA in sports, the World Anti-Doping Agency.

«We need an international organization that is independent, but that has money and power, which can both demand withdrawals and kick out the junk journals.»

In order to overcome the trend that the Chinese paper mills represent, one must use tough measures, he believes.

«The journals should stop accepting all articles from China for a period of time, to signal to the authorities in China that they need to clean up. An approach like this will also affect some honest researchers, but the situation is so bad that drastic measures are needed.»

 

Wiley increases its screening activity

The academic publisher Wiley will make sure more images are examined before publishing.

Wiley owns several of the journals that have published articles from the Chinese paper mills.

The publisher emphasizes its commitment to maintaining an accountable research literature and protecting public trust in science.

«We take research integrity seriously, investigating all claims of false data or images according to COPE guidelines,» Wiley states in an e-mail to The Magazine Research Ethics.

In some of the journals Wiley publishes, images are already screened. Wiley is now planning a screening service with at least two of the journals involved in the paper mill-case: Journal of Cellular Biochemistry and Journal of Cellular Physiology.

(Go back to the main article)

NLM says it takes its role seriously

National Library of Medicine says it takes seriously its role in providing access to trusted health information.

The National Library of Medicine (NLM) is in charge of MEDLINE, a bibliographic database that contains more than 25 million references to journal articles in life sciences. NLM also maintains the free search engine PubMed, which primarily accesses MEDLINE.

The Magazine Research Ethics asked NLM to comment on its role in keeping the scientific literature sound and what is done to uncover falsification, fabrication and misconduct. We also asked if NLM agrees with «Morty's» claim that NLM has let too many articles pass.

In an e-mail NLM replied that it «takes seriously its role in providing access to trusted health information to researchers, clinicians, and the general public». It also «relies on a robust set of procedures to assure the quality of the materials it collects and makes available through services such as PubMed and PubMed Central (PMC)».

According to the e-mail, expert consultants review the scientific and editorial quality of a journal for PMC selection, and make recommendations to NLM. «The scientific quality assessment for PMC focuses on the scientific rigor of the articles examined.»

According to NLM, journals selected for PMC are subject to review by a federal advisory committee comprised of external experts in biomedical science and scholarly communication.

«The recommendations are based on the journal's scientific content, quality, impact, editorial policies, and scope.»

We also asked NLM to comment on the discovery of the Chinese paper mills but got no reply.

(Go back to the main article)