by Demetris Koutsoyiannis & Zbigniew W. Kundzewicz
Having published some hundreds of papers over decades and having been co-editors of Hydrological Sciences Journal (HSJ) for a long time (ZWK for 18 years, DK for 12 years), we have been interested in the peer-review system. Trying to diagnose its pathologies and suggest remedies, we have written several editorials published in HSJ (Kundzewicz and Koutsoyiannis, 2005, 2006; Koutsoyiannis and Kundzewicz, 2007, 2009). We also collaborated with editors of other hydrological journals and compiled joint editorials, published in several journals simultaneously (Blöschl et al., 2014; Koutsoyiannis et al., 2016; Quinn et al., 2018).
There are three different peer review systems, from the transparency viewpoint:
- double-anonymous (double-blind) system, where the identity of the reviewers is not disclosed to the authors and the identity of the authors is not disclosed to the reviewers;
- single-anonymous (single-blind) system, where the identity of the reviewers is not disclosed to the authors, while the identity of the authors is disclosed to the reviewers;
- eponymous, fully transparent, system, where the identity of the reviewers is disclosed to the authors and the identity of the authors is disclosed to the reviewers;
System (1) is rather a delusion or even a joke, because very often the identity of the authors can be easily guessed, based on references. System (3) is the most ethical and responsible—yet very rare. Hence, far more common (if not virtually exclusive) is the system (2).
Anonymity has been so prevalent in scientific transactions that many colleagues only know the Greek word anonymous (ἀνώνυμος) and not its antonym, eponymous (ἐπώνυμος).
Overall, we have supported the need for a change in culture in the peer-review process toward enhanced transparency.
Recently, we had an interesting experience, not as editors but as authors of a paper challenging conventional wisdom (Koutsoyiannis and Kundzewicz, 2020; graphical abstract in Fig. 1). Trusting that this case deserves being disseminated, we describe this experience below as a didactic case, whose lessons could point to the direction where—in our opinion—progress lies.
Fig. 1. Graphical abstract of Koutsoyiannis and Kundzewicz (2020): “Atmospheric temperature and CO₂: Hen-or-egg causality?”
Initially, we submitted our paper to Science of the Total Environment (STOTEN). It was swiftly rejected based on two negative anonymous reviews. This was no surprise to us: it is well known that anonymous reviewers tend to kill papers presenting new ideas. We challenged the editor’s decision as we felt that we could rebut every one of the critical review comments. As usual, we received a negative reply by the editor: “Unfortunately, we can no longer consider the paper for publication in STOTEN”. We made our challenge quite strong, by rendering the manuscript public in ResearchGate along with the reviews. We asked the STOTEN editor “to let the reviewers know that our paper and their reviews are uploaded in ResearchGate and that we will be happy to post our rebuttal there if the reviewers agree to post replies to our rebuttal”.
We are pretty sure that the editor and the reviewers visited our posts (the manusript and the reviews) on ResearchGate. (Some of the sections of the final published paper contain replies to these latter comments, as we have specified on ReserachGate). This platform allows posting comments—eponymously of course. But no comments were posted (or received in private communication). Again, no surprise. These reviewers could easily kill the paper in an anonymous mode—but they would have difficulties in justifying their criticism if they were eponymous.
Indeed, eponymity is more demanding. Eponymous reviewers should articulate their claims and should be ready to receive criticism on them by the authors. In other words, eponymity needs more effort and courage. But it is cooperative, democratic, equitable, ethical, productive and responsible.
In our view, in an era where the quest for transparency has become extremely important, it is time for a radical change in scientific ethics. Thus, when we are tempted to submit an anonymous review, a good question to ask ourselves is this: If I cannot be an eponymous reviewer, is it accurate to be called a reviewer? (And if yes, who is actually that reviewer? Myself or my anonymous, perhaps frightened, clone?)
Some modern scholars use Clarivate’s PUBLONS to secure recognition for their anonymous reviews. We suggest that they ask themselves another question: What is the ethical and aesthetic value of seeking credit for anonymous transactions?
After the unpleasant experience with STOTEN, we decided not to submit our paper to another journal employing blind review. We were attracted by the novel, eponymous, review system of the emerging, open-access, journal Sci published by the Multidisciplinary Digital Publishing Institute (MDPI) group (Rittman and Vazquez, 2019; Jacob et al., 2019). In this innovative, community-driven, so-called post-publication peer-review system, a decision for (pre-)publication was made by an editor and then reviewers were invited (or volunteered, as everything is public and transparent). After one or two review rounds, a final decision was made and, if this was positive, the paper would be included in a journal issue.
We decided to submit our paper to Sci for two reasons. First, to avoid another likely rejection after another blind review, with the reason of rejection being au fond the fact that we challenge conventional wisdom. And second, to support the innovative progressive peer-review system of Sci. We were aware, of course, that Sci, as a new journal, is not included in the Clarivate’s Web of Science and Science Citation Index. We did not use this as a criterion for our choice.
Our experience with Sci was positive. The public and eponymous reviews were constructive and recognized the usefulness and importance of our paper. In the second round, the reviewers fully accepted the way we handled their comments and our reasoning about it. In fact, we considered even the negative (anonymous) review comments of the rejected submission to STOTEN and we strengthened our paper against them. Eventually, our paper was accepted and published. The timing of the several stages of the process is shown in Figs 2. and 3.
Unfortunately, the acceptance of our paper coincided in time with a major change of Sci’s peer review pattern to the conventional (single-blind) system (Vazquez et al. 2020). The change is also visually reflected in the journal’s web platform: compare Fig. 2 (before) and Fig. 3 (after).
In our case, the rules of the game changed during the game. We communicated our complaints to the editors and the publisher, explaining that we would not submit our paper in that journal if it was run with the conventional peer-review system. We also expressed our disappointment that a big step of progress was followed by a big step of regression. We understood the publisher’s reasoning that there exists a clear conflict—if Sci continued with this system, Clarivate Analytics would not include it in its indices. As we wrote, for us this is not important, but we understand that for most colleagues—particularly the youngsters—it is. Indeed, bibliometric indices spanned over recognized sources from the Clarivate’s Web of Science, and citation counts in particular, are the currency in which scientists are evaluated nowadays. Given that Clarivate has invested a lot in anonymity, through its PUBLONS system, it is explainable (yet saddening) that it discourages an innovative progressive system in favour of the traditional, anonymous, transactions.
Fig. 2. The Sci’s site with our paper in its original (post-publication peer review) setting with full recognition of reviewers’ contributions.
Fig. 3. The Sci’s site in its changed setting, with the final version of our paper; yet the eponymous reviewers’ contributions are accessible by hitting “Review Reports”.
Nonetheless, this Sci’s step back was, fortunately, partly counterbalanced by a subsequent arrangement by the journal not to fully abolish eponymous open review. Thus, it runs now with two alternative options. As explained in the instructions for authors, “Authors are given the option for open peer review or single-blind peer review.”
We welcome this arrangement and, thanks to it, we will continue to support Sci for its first option.
References
Blöschl, G. Bardossy, A., Koutsoyiannis, D. Kundzewicz, Z.W. Littlewood, I.G., Montanari, A., and Savenije, H.H.G., 2014. Joint Editorial—On the future of journal publications in hydrology, Hydrological Sciences Journal, 59 (5), 955–958, doi: 10.1080/02626667.2014.908041.
Jacob, C., Rittman, M., Vazquez, F., and Abdin, A.Y., 2019. Evolution of Sci‘s Community-Driven Post-Publication Peer-Review. Sci , 1, 16, doi: 10.3390/sci1010016.v1.
Koutsoyiannis, D., Blöschl, G., Bardossy, A., Cudennec, C., Hughes, D., Montanari, A., Neuweiler, I., and Savenije, H.H.G., 2016. Joint Editorial: Fostering innovation and improving impact assessment for journal publications in hydrology, Hydrological Sciences Journal, 61 (7), 1170–1173, doi: 10.1080/02626667.2016.1162953.
Koutsoyiannis, D., and Kundzewicz, Z.W., 2007. Editorial—Quantifying the impact of hydrological studies, Hydrological Sciences Journal, 52 (1), 3–17, 2007, doi: 10.1623/hysj.52.1.3.
Koutsoyiannis, D., and Kundzewicz, Z.W., 2009. Editorial—Recycling paper vs recycling papers, Hydrological Sciences Journal, 54 (1), 3–4, doi: 10.1623/hysj.54.1.3.
Koutsoyiannis, D., and Kundzewicz, Z.W., 2020. Atmospheric temperature and CO₂: Hen-or-egg causality?, Sci, 2 (4), 83, doi: 10.3390/sci2040083.
Kundzewicz, Z.W., and Koutsoyiannis, D., 2005. Editorial—The peer-review system: prospects and challenges, Hydrological Sciences Journal, 50 (4), 577–590, doi: 10.1623/hysj.2005.50.4.577.
Kundzewicz, Z.W., and Koutsoyiannis, D., 2006. Pathologies, improvements and optimism, Hydrological Sciences Journal, 51 (2), 357–363, doi: 10.1623/hysj.51.2.357.
Quinn, N., Blöschl, G., Bardossy, A., Castellarin, A., Clark, M., Cudennec, C., Koutsoyiannis, D., Lall, U., Lichner, L., Parajka, J., Peters-Lidard, C.D., Sander, G. Savenije, H.H.G., Smettem, K., Vereecken, H., Viglione, A., Willems, P., Wood, A., Woods, R., Xu, C.-Y., and Zehe, E., 2018. Invigorating hydrological research through journal publications, Hydrological Sciences Journal, 63 (8), 1113–1117, doi: 10.1080/02626667.2018.1496632.
Rittman, M. and Vazquez, F., 2019. Sci—An Open Access Journal with Post-Publication Peer Review. Sci, 1, 1, doi: 10.3390/sci1010001.v1
Vazquez, F.; Lin, S.-K.; Jacob, C., 2020. Changing Sci from Post-Publication Peer-Review to Single-Blind Peer-Review. Sci, 2, 82, doi: 10.3390/sci2040082.
We have sent the address of this blog by email to about a thousand colleagues and we receive some interesting feedback by email.
Here is my reply to the first email by an editor.
—-
Thank you very much for sharing your thoughts. Glad that our email triggered this communication.
In my view things are simple. Don’t we call that “peer review”? If I am afraid to say my opinion eponymously, am I peer? And if I have hesitation to be eponymous, isn’t it conflict of interest? So, I may tell the editor that there is conflict of interest and he may find another reviewer who does not have conflict of interest.
When you speak about early career scientists, have you thought about the responsibility of the late career scientists in training the former not to be courageous and transparent?
I have been always eponymous since I was an early career scientist. Believe it or not, I never had any problem of the type you are writing.
Dimetris, thanks for this most interesting discussion. Here is my own experience with the peer-review system … not impressed in the slightest. It was this experience that put me off scientific journals.
w.
http://wattsupwiththat.com/2013/03/30/dr-michael-mann-smooth-operator/
In the six years and 13 submissions it took me to publish the very straight-forward error analysis of climate models, my conclusion is that the major problem lays with journal editors.
They typically seem to lack the intellectual and professional courage to set aside obviously incompetent and partisan reviews. Editorial failing is the largest impediment to fair peer review.
Many Journal editors also lack the moral courage to publish valid critical work that falsifies fashionable theories, at times violating the ethical standards of their own journal. They flee the conflict. I came to call this the ABM — ‘Anyone But Me’ — syndrome.
I have 60 MB of evidence documenting these traits.
The one instance of fully transparent and signed peer-review produced a review extraordinary only for its spiteful dishonesty. In my experience, review method 3 — revealed author, revealed reviewer — is an invitation to personal acrimony.
Much better is total anonymity. Yes, one may guess an anonymous author or an anonymous reviewer, but a guess is not the same as knowing. Better to completely avoid the entry of personality.
Willis, thanks very much for posting and for the link. I believe quitting is not the best practice and I hope you reconsider your relationship with scientific journals.
Demetris, you say quite reasonably:
“I believe quitting is not the best practice and I hope you reconsider your relationship with scientific journals.”
I was burned very badly when Michael Mann used the journal reviewers or editors to steal my ideas and publish them under his name … and I have not seen one thing to make me think this wouldn’t happen again. This is not the Royal Society, this is the Garden of Thieves.
Me, I would be very interested in a journal which published the SIGNED reviews along with the paper, for both approved and rejected papers. In some ways the rejected papers would be much more valuable and interesting than the accepted ones, as they would keep people from repeating the same mistakes. We are missing a huge opportunity for scientific progress.
w.
Thanks, Pat. My own record is rejections from 8 journals for one item. So, it appears that you have beaten me…
I believe people are different and this is true also for the editors. There are problems with all players, not with a single part. I believe by improving the entire system through transparency, responsibility and consistency with ethical values, all players would be benefited.
Here I am copying an entry from simultaneous discussion at ResearchGate.
—
Demetris Koutsoyiannis added a reply (30 minutes ago)
Thanks, Russell, for your comments. Here is my feedback. First, let me point out that anonymity and secrecy are corruptible—and corrupting. Even though anonymous reviewers feel their anonymity is safe, it is not. Perhaps someday a good hacker will post lists of anonymous reviewers along with their names (cf. wikileaks, climategate, etc.). Perhaps a judge will someday order a journal to reveal the reviewers’ names. Even if this does not happen, an author smarter than average often knows with confidence the names of the anonymous reviewers. For example, I believe I know who most of the anonymous reviewers of my papers are.
As we write on the blog, the anonymity of the authors “is rather a delusion or even a joke, because very often the identity of the authors can be easily guessed, based on references.”. What would you propose if I submit a follow-up of a work that I have published? Should I hide that I was the author of that earlier work? And if I try, is the reviewer so silly that would not spot it? Note, with current content similarity systems (e.g. IThenticate, Turnitin, etc.) it is trivial to associate characteristic phrases with names. I support full simplicity and full transparency: Everybody has a name, let it be known in every scientific transaction.
I fully agree with you that “the current system of scientific publishing is moving toward a business model that prioritizes profit over science”. I am supporting the “diamond open-access” model (open-access publication without payment). As an example, after some discussions with publishers, I gave up the idea of publishing my new book with a commercial publisher. I have already uploaded a full preview on my group’s website and on ResearchGate:
D. Koutsoyiannis, Stochastics of Hydroclimatic Extremes – A Cool Look at Risk, 330 pages, Edition 0, National Technical University of Athens, Athens, 2020.
http://www.itia.ntua.gr/2000/
https://www.researchgate.net/publication/345981795
Here is my reply to another editor, slightly edited to anonymize it. 🙂
—-
Thank you very much for your reply and for sharing your thoughts about the peer review system. You are very kind.
You say “the eponymous system might be helpful for your case but I see it may create more issues than it may solve, as the major concern on the conflict of interest“. I respectfully disagree 100%. See my reasoning about the problems you mention here: http://blog.itia.ntua.gr/2020/12/11/challenging-conventional-wisdom-and-the-conventional-peer-review-system/#comment-816. In my view the reason that we are still stuck with anonymous transactions is the insufficient ethical standards of the scientific community (as indicated by the fact we note in the blog, i.e. that colleagues know what “anonymous” is, but do not know its antonym) and the intervention of political correctness.
Apparently, it is very easy to tear a paper down if you know you will never have to account for what you have said. Isn’t this an ethical problem? Personally, I am proud that I have a big rejection record, see http://www.itia.ntua.gr/en/search/?tags=rejected. All these papers but one were eventually published somewhere else and most of them are my most cited ones. So there are good omens that our “hen-or-egg” paper would have a good impact.
What you advise us as a better way to get through the peer review symplegades (i.e. to reach the editor first), I believe we have actually done. We reached the editor, when we knew who he is and when we had the evidence that our paper is “non-traditional” (offered by the rejecting reviewers). We wrote to him that we could rebut every one of the review comments.
Finally, I agree with you 100% when you say “it is important to hear different people’s opinions, including both positive and negative ones“. Actually, as you may see in the Acknowledgments of our published paper, we acknowledge the negative reviews we received in STOTEN as they helped us to strengthen our paper against their comments.
Here is (part of) my reply to another editor.
—
I am glad that we agree on eponymous reviewing. I also agree on the obstacles you write, stemming from the human nature: egoism, fear, desire to be always right. But I think all of us humans have also good sides, not only these bad sides–and all these are not static but subject to change. A certain system may encourage and promote some of them, and discourage and demote some other.
I believe eponymous reviewing encourages and promotes the good sides. Thus, it helps us to become better people and at the same time helps the scientific community to become more cooperative, democratic, equitable, ethical, productive and responsible.
Demetris,
I completely agree, and I myself ALWAYS identify myself to the authors when providing a review. A major reason is that I feel that the role of the review system should be primarily to be constructive, rather than destructive, and open communication helps to promote that.
I further believe that all of the review and rebuttal process (i.e. comments and responses) should be posted along with the eventually published manuscript. This would help to ensure that the (interested) readers are able to grasp more of the nuances of the topic being discussed, and further that if the authors alter their manuscript to incorporate reviewer suggestions, that the reviewers get some kind of credit for their contributions, where significant. I myself have seen authors make radical alterations to their papers in response to reviews, without providing acknowledgement, which is a form of intellectual dishonesty.
So, all in all, an double open process would lead to a more rigorous and scientifically useful result.
I guess a concern some younger reviewers may have is that if they post critical comments they could be subject to retaliation from senior authors in future … this is a difficult issue, and one that the community needs to grapple with.
But again, if the critical comments are posted in a polite and constructive manner, one would hope that the community would not stoop to using tactics like retaliation, and accept the comments graciously and in the spirit of advancing science rather than personal egos.
Regards.
Hoshin
Hoshin, thank you so much for your thoughtful comments. Given your active involvement and tremendous experience as a researcher and editor, I trust they would be influential.
Demetris