Both fake studies were of very high quality and conformed to the best standards of medical research. The two compared methods – a single dose of antibiotics before surgery versus a starter doze plus smaller doses over a 24-hour follow up period – are highly debated by medical scientists at the moment. Theoretically, both papers should have been of equal interest for reviewers, but that was not true.

The “one method better” paper received 98% positive reviews from reviewers of the American edition of Journal of Bone and Joint Surgery (JBJS). The “no difference” paper was recommended for publication by only 71%. Strikingly, these reviewers also gave the entirely identical methods section a full point advantage (on a scale of one to ten) in the positive paper. “There's no good explanation for that,” says Leopold.

Reviewers were also different in how well they found five intentional small mistakes planted in the papers. For the positive result paper, they caught only 0.3 errors per reviewer as compared to 0.7 errors per reviewer for the negative result paper.

The positive result bias is “a major problem for evidence-based medicine,” Leopold said while presenting his work at the Sixth International Congress of Peer Review and Biomedical Publication in Vancouver, British Columbia. It can skew journals towards good reviews of drugs, which may affect recommended treatments.

The study “just goes to show that peer review is done by biased, subjective people," says Liz Wager, managing director of the Sideview consultancy in Princes Risborough, UK, and chair of the UK-based Committee on Publication Ethics. “Everyone wants the new stuff to work – they want to believe.”