The Peer Review Process

Common Peer Review Processes

The life of a manuscript flows as follows:

Author(s) spend hours revising and finalizing a manuscript.
Authors submit to the right journal. Right is based on (a) the mission of the journal and (b) the match between the audience which the author wants to reach and the known audience of the journal.
Editor determines if this manuscript should be sent for external review. The editor may ask these questions: Does this manuscript align with my journal's mission and my reader's interests? Did we recently publish similar content?
External Reviewers are invited (usually 2-3).
External reviewers are provided the manuscript, review forms, instructions, and deadlines. Most journals have an online process for editorial review.

Types of Reviewers

There are at least three methods by which journals use reviewers.

For board review, all reviewers are members of the journal's editorial board. The board is chosen based on the needs of the journal and the expertise of the potential reviewers.
For pool review, the journal editors identify professionals who might have expertise in the area of focus for this journal and who have agreed to review as need. A list is maintained of these potential reviewers, and invitations are sent to a few for each manuscript. Invitees can accept the invitation, or decline, if this is not a good time for reviewing, or ask to be removed from the pool.
Mixed method of review uses a combination of the board review and the pool review approach where every manuscript is reviewed by an editorial board member and one or two pool members.

The Journal of Student-Run Clinics has a mixed method review system with reviews being conducted by both editorial board and student/faculty reviewer pool members.

Selection of Reviewers

Factors considered include: content expertise, clarity of writing, balanced judgment, responsiveness to the needs of the journal.

Mechanisms to identify reviewers:

Contacts at professional meetings
Editorial boards
Personal acquaintances
Literature searches
Members of professional societies
Manuscript bibliography. If you have published something on X, you are likely to be invited as a reviewer on another paper discussing X.
Authors may be asked to suggest reviewers
Pool reviewers declining a specific invitation may be asked to suggest other potential reviewers
Often the same reviewers are used for reviewing the revised manuscript to see if the constructive suggestions have been considered by the authors and the paper improved as a result of the feedback.

The Journal of Student-Run Clinics recruits student reviewers based on an application process to the editorial board. Student reviewers are selected based on involvement in student-run clinics, commitment, and
interests. Faculty reviewers are field experts identified through the mechanisms described above and are
recruited by the editorial board directly or through nomination by others.

Choosing & Tracking Reviewers:
Journals track how often they use their reviewers and the timeliness and quality of the reviews provided.
Some journals (e.g. Annals of Internal Medicine) give the reviewers an annual report on the quality of the
reviews done by each reviewer in comparison with the whole pool of reviewers.

Peer Review Format

Commonly, reviewers are asked specific questions with dichotomous or other rating scales for various sections of an article. Most reviews require written comments.

Reviewers may then be asked to provide a final disposition: accept, accept with revisions, reject with resubmission option, reject. Usually the format has a section for constructive suggestions to authors and a separate section for confidential comments to the editor.

This example comes from the journal Medical Education.

METHODS
Are these clearly explained? Y/N
Are the methods appropriate? Y/N
Is the sample size accurate? Y/N
Is the sample size representative? Y/N
Are the subjects fully described? Y/N
Is the statistical method appropriate? Y/N

Another example for openended questions comes from Academic Medicine.

Explain why the research design and analysis of the data is or is not appropriate for the research question.
Given the topic and research question, who in the academic medicine community would be interested in reading this paper? Who (if anyone) should read this paper?

Final decision for publication rests with the editors. The reviewers advise but may not have a vote.

The Journal of Student-Run Clinics has a two-level review process (detailed below) which uses a structured form for narrative comments about particular aspects of an article with an accompanying rubric with concept anchors to consider for guidance. Reviewers will provide their acceptance recommendation and can provide constructive suggestions and confidential comments. The entire process is conducted through an online interface.

Journal of Student-Run Clinics Review Process

While the Journal of Student-Run Clinics review process shares the fundamental goals and values of peer review for scholarly publishing, it is made unique through the explicit use of both student and faculty reviewers. When an article is received through the online submission system, an editor will assign it to one or more student
reviewers for evaluation. A reviewer may accept or decline this request. Conflict of interest or inability to
complete the task within one month may be reasons to decline. The paper is screened by the student reviewers
using the review form and rubric to determine relevance, general scientific and ethical validity, and
readability. Constructive comments to the author(s) and confidential comments to the editors may also be made.

If the submission passes the screening and is of an article type that requires additional review, it is sent to one or more faculty experts for peer review using the same review forrm -- again assessing relevance, validity, and readability and gathering constructive and confidential comments.

A reviewer may accept or decline this request. Editors will collect and analyze the reviews from both students and faculty and make a publication decision based on the reviewer recommendations. In the case that there are significantly conflicting reviews, additional reviewers may be requested at the editor's discretion.

Below is the review form and guide rubric. Each element of the rubric will be thoroughly discussed during subsequent training module lessons. Narrative comments will also be discussed in later lessons.

Review Form

Briefly (1-3 sentences) summarize the nature of the submission (i.e. type of article and main topic).

Briefly state why the submission would or would not be of interest to participants of student-run clinics.

Provide specific feedback on the strengths and weakness of the submission using examples when helpful. If the recommendation is "Revisions Required" list exactly what needs to be revised. Every critique should be specific and actionable so authors can easily identify and correct areas of improvement.

Summarize your rationale for your publication recommendation.

Additional comments to authors:

Note any limitations of your review, such as non-expertise in a statistical method (not shared with authors).

Additional comments to editors (not shared with authors):

Guide Rubric

Relevance	Topic and its direct relation to and implications for student-run clinics are clearly and fully defined
Relevance	Establishes extensive context of prior/existing research and examines how the study conducted relates to them in terms of originality and importance
Validity	Methods are thoroughly and clearly described and explained to facilitate replication
	Study design is clearly appropriate for measuring intended parameters and potential biases are well explored and accounted for
	Conclusion clearly logically flows from data provided and explicit, technically-correct statistical reasoning is provided
	Ethical review entities (i.e. institutional review board) are cited for ethical clearance and study is clearly conducted with highest standards of scientific conduct
Readability	Writing explains key terms and concepts that may be unfamiliar to those outside a given field and can largely be understood by all professional audiences
	No grammatical errors
	Concise and focused writing style without sacrificing accessibility
	Organization reflects a cohesive progression of ideas and the abstract and title are completely consistent with the manuscript content

The next training module lessons will discuss the relevance, validity, and readability pieces of the guide rubric as well as key points about assessing the different sections of a manuscript.

Next Lesson >>

Lessons

Recommended for first time student reviewers: lessons 1 through 9
Recommended for first time faculty reviewers: lessons 1, 3, 7, 8, and 9