https://www.nature.com/articles/d41586-018-07245-9 A nature article is describing current effort to use AI for improving the peer review system. The most surprising answer is, that AI is indeed the best practice method in doing so. There are two main reasons for this:
1. By formalizing the peer review process into a computer software, it can be reproduced more easier. That means, even if the created peer review software isn't used in reality, but normal human peer reviewer are monitoring a manuscript it make sense to program the software, because this will make more clear what peer review is about.
2. Large part of the peer review process, for example plagiarism checks can be realized with computer programs very well. That means, the AI is able to do a basic incoming control.
So let us describe what an AI peer review system is doing. The first and most important incoming check is to verify the identity of the user. That means, an online peer review platform whould allow only professors and phd student to submit a paper, but not other people. It's interesting to know, that such an identity checkup has nothing to do with Artificial Intelligence itself, but how peer review works. Peer review is about 10 million scientists worldwide who are able to submit a paper to a journal, while normal authors are not allowed in doing so.
Sure, a non-scientist can create with LaTeX a paper too, but he won't get a peer review for it. That means, all the papers in the world were written by experts but not by amateurs. Sure, it's possible to imagine a moderated preprint server in which everybody is allowed to upload content. Similar to zenodo. But Zenodo isn't a peer review platform but it's a self-publishing system.
The funny thing is, that academic peer review and be member of the population of 10 million researchers in the world is the same. That means, all the peer review papers in the world are created by phd students who can proove that they are really phd students but not amateurs. The hypothesis is, that escpecially an automated AI based incoming check won't change the situation, but it would replicate the working of the current system.
Or let me explain it the other way around. An automated peer review system doesn't open up the ivory tower for new users, but it holds non-scientists outside of the publication chain.
How important is the identity in peer review?
The surprising fact around peer review is, that the process isn't formalized yet. One possible attempt in doing is to introduce artificial Intelligence. This will result into the question, how exactly a software should work who is doing the peer review process autonomously.
The first aspect of peer review is, that it's never about the document itself, but it's about the researcher's identity. Perhaps it make sense to describe the situation in detail. Suppose, peer review is equal to anonymous peer review. Then, the workflow is the following:
10 human users with unknown names are submitting pdf documents to a website. The papers are evaluated by the peer reviewers, if the content is plagiarized, if the research topic is new and if the mistakes are made. Then the peer reviewers will reject 5 of the papers, and the remaining 5 papers are peer reviewed.
The funny thing is, that such a pipeline is the opposite of the reality. The reason is, that in the fictional example, the assumption was, that paper itself, but not the identity of the users is important. A more realistic peer review works with the following principle:
All the 10 users are validated. 5 of them have a phd title and as a result their papers get accepted, while 5 of them have no phd title and their paper gets rejected. Even if the non-phd students have written a more interesting paper, they have no chance to get accepted. Basically spoken an automated peer review software is doing an identity check if the submitting user has a phd title. If not, he gets rejected.