Guide Rubric: Validity

In this section, we will look at the Methods, Results, Discussion, and Conclusion for the validity of a manuscript as described by the review criteria. We will also examine scientific conduct and authorship ethics.

Descriptive reports may not contain elements evaluated by this section of the rubric, in which case reviewers
are asked to use their best judgement for the relevant items.

Validity	Methods are thoroughly and clearly described and explained to facilitate replication
	Study design is clearly appropriate for measuring intended parameters and potential biases are well explored and accounted for
	Conclusion clearly logically flows from data provided and explicit, technically-correct statistical reasoning is provided
	Ethical review entities (i.e. institutional review board) are cited for ethical clearance and study is clearly conducted with highest standards of scientific conduct

Methods are thoroughly and clearly described and explained to facilitate replication

The variables being investigated are clearly identified and presented
The research design is defined and clearly described and is sufficiently detailed to permit the study to be replicated
The development and content of the instrument are sufficiently described or referenced and are sufficiently detailed to permit the study to be replicated
The psychometric properties and procedures are clearly presented
The data set is sufficiently described or referenced
The population is clearly defined, for both subjects (study participants) and stimulus cases (i.e. those selected to include in a teaching intervention), and is sufficiently described to permit the study to be replicated
The sampling procedures are sufficiently described
Data analysis procedures are sufficiently described and are sufficiently detailed to permit the study to be replicated

Study design is clearly appropriate for measuring intended parameters and potential biases are well explored and accounted for

The research question (research hypothesis where applicable) is clear, concise and complete (this may be found in the introduction section)
The design is appropriate for the research question
The measurement instrument is appropriate given the study's variables, the scoring method is clearly defined
The psychometric properties and procedures are clearly appropriate
The design has internal validity (causal effect can be determined from the experiment, potential confounding variables or biases are addressed)
The design has external validity (can be generalized, including subjects, settings and conditions)
The design allows for unexpected outcomes or events to occur
The design and conduct of the study are plausible
Observers or raters were sufficiently trained
Data quality control is described and adequate
Subject samples are appropriate to the research question
Stimulus samples are appropriate to the research question
Selection bias is addressed
Data analysis procedures conform to the research design's hypotheses, models or theory drives the data analysis
Statistical tests are appropriate
If statistical analysis involves multiple tests or comparisons, proper adjustment of significance level for chance outcomes was applied
Power issues were considered in statistical studies with small sample sizes
The statistics are reported correctly and appropriately
The number of analyses is appropriate

Conclusion clearly logically flows from data provided and explicit, technically-correct statistical reasoning is provided

In qualitative research that relies on words instead of numbers, basic requirement of data reliability, validity, trustworthiness, and absence of bias were fulfilled
The assumptions underlying the use of statistics are fulfilled by the data, such as measurement properties of the data and normality of the distributions
The assumptions underlying the use of statistics are considered, given the data collected
Measures of functional significance, such as effect size or proportion of variance accounted for, accompany hypothesistesting analysis
Tables, graphs, and figures are used judiciously and agree with the text
The amount of data presented is sufficient and appropriate
The conclusions are clearly statedÍ¾ key points stand out
The conclusions follow from the design, methods, and resultsÍ¾ justification of conclusions is well articulated
Interpretations of the results are appropriate, conclusions are accurate (not misleading)
The study limitations are discussed
Alternative interpretations of the findings are considered
Statistical differences are distinguished from meaningful differences
Personal perspectives or values related to the interpretation are discussed
Practical significance or theoretical implications are discussedÍ¾ guidance for future studies is offered

Ethical review entities (i.e. institutional review board) are cited for ethical clearance and study is clearly conducted with highest standards of scientific conduct

No plagiarism is detected
Ideas and materials of others are correctly attributed
Prior publication by the author(s) of substantial portions of the data or study is appropriately acknowledged
No conflict of interest is apparent
The text explicitly describes approval by an institutional review board for studies directly involving human subjects or data about them
Reference citations are complete and accurate

We will now further explore key concepts of research validity from a reviewer's perspective.

Research Design
Research designs vary, including fully controlled experimental studies and purely observational studies. Sometimes more than one method is used in a study.

The primary tasks for the reviewer are:

To recognize the fir or congruence of the research method to the research question and the field of study.
To consider the validity and plausibility of the work.
- Internal validity refers to the integrity of the study and potential sources of bias such as selection bias, attrition of subjects, intervention bias, strength of intervention, measurement bias among others.
- External validity refers to the issue of generalizability of the study to other settings, taking into consideration the subjects the settings and the conditions of the study.
- Plausibility takes into account issues such as proper randomization, small sample size leading to low statistical power, weak interventions, absent or inappropriate control groups.

Instrumentation, Data Collection & Quality Control

Decisions about data gathering and data scoring must be clear and logical to the reviewer.

Were the instruments appropriately selected or developed?
Were the data gathering methods appropriate and/or instruments used appropriately?
If data collection was done by human effort, were data collectors trained and was intra- and inter-rater reliability assessed?
Were actions taken to minimize differences between different users? Were incentives appropriately used to ensure completeness of data?
Were the data appropriately scored?

Population and Sample

Does the study sample include appropriate age groups, genders from appropriate populations, sufficient to answer the specific study question?
Was there explicit consideration to avoid selection bias by ensuring that the sample is truly representative of the population?

Data Analysis & Statistics

Research design dictates what statistical analysis should be used to analyze the data. Here are points to
consider:

Have appropriate statistics been used for between-group vs. within-group differences (i.e. t-tests, ANOVA vs. correlation coefficients)?
Were appropriate assumptions made about expected distribution of data (i.e. if the data are normally distributed, were two-tailed tests used)?
Were post hoc analyses specified before data were collected?
Beware of data dredging. It is a given, that with continued analyses, something might emerge as "significant." Multiple analyses of data, without an a priori research hypothesis, are problematic. For example, when 20 statistical tests are run, one "significant" result will emerge purely by chance. Thus, when multiple analyses are done, the significance level must be adjusted (i.e. Bonferroni correction).
Qualitative data flaws include inattention to triangulation, lack of detailed description of research observations, failure to use repetitive data analysis and interpretation, lack of independent data verification by colleagues and stakeholders, and absence of a priori expression of the investigator's personal orientation in the written report.

Reporting of Statistical Analyses

NOTE: As a reviewer, not understanding the statistical analysis used in the work is not necessarily reason for
recusing yourself from the review, but in such a case, you should alert the editor that you have no expertise in
this analysis.

Try to determine if the data satisfy the assumptions necessary for use of the statistical test(s) used. Sometimes the planned analysis may not be feasible after the data are collected (i.e. a planned correlation between two variables cannot be performed because the range of data is very tight or a t-test was planned to compare the means of two groups, but the results have a bimodal rather than a normal distribution, so the means and standard deviations may not meaningfully describe the data.
In the Results section, are statistical analyses reported that are not described in the Methods section? Expansion or addition of new analyses could be inappropriate if done without forethought and careful consideration. An uncontrolled proliferation of analyses or new analyses without adequate explanation is a red flag to the reviewer.
Statistical significance may not mean practical significance. A common situation in which a statistically significant finding may lack clinical significance may occur in a study where the sample size (n) is very large. Small differences between groups may achieve statistical significance but still not be meaningful. Inferential statistical tests only tell us the probability that chance alone produced the results. However they do not reveal the strength of the association among research variables (e.g. effect size).
Did the authors attempt to report what proportion of the variance can be explained or accounted by the specific independent variable? Common indices of explained variations are eta 2 in ANOVA and r2 in correlational analyses. If the independent variable does not account for a reasonable proportion of the variance, the study may not be worthy of publication. Although the result may be statistically significant, it is not caused by the variable tested and therefore may not be interesting to the audience.

Presentation of Results

The results of the study, and their relationship to the study question and discussion points, must be clear to the reader. Organization of the Results section ideally mirrors organization of the research questions in the introduction of the paper. When several research questions are addressed, the results may be best presented in a series of subsections, each of which addresses one question. This allows the reader to see if each research question has been answered appropriately and completely.
Did the authors critically select which data to present and carefully reflect on how best to present the data?
For qualitative research, the Results section may be organized by themes or by the method of collection of data.
Context of the study must be clearly reported in order to provide a framework for that data, such that reader can judge if the subsequent interpretation reflects the context accurately.
For quantitative data, the balance between descriptive statistics and inferential statistics must be maintained.
Narrative should be used to describe the key results clearly. Tables and graphs should be supplementary or supportive, not the sole presentation of the results. The text must be consistent with the data in the tables and/or figures.
Extrapolation of the results to the research question and discussion of implications of the work belong in the Discussion section of the paper, not in the Results section.

Discussion and Conclusion

Reviewers should be convinced that the interpretation of the results is justified. In addition, given the limitations and architecture of the study, reviewers should judge the generalizability and practical significance of the study.
Reviewers must evaluate whether each research hypothesis is refuted or confirmed and whether each conflicts with or aligns with previous research.
Qualitative approaches: did the author convince the reader that data are trustworthy? Are the data credible (internal validity) and transferable (external validity)? Techniques for enhancing data credibility include triangulation, member checking and peer debriefing. Have multiple data sources (triangulation) been used? Have interpretations been tested with those from whom the data were collected during interviews (member checking)? Have disinterested peers analyzed the data and confirmed or expanded the researcher's conclusions (peer debriefing)?

Scientific Conduct

Reviewers must be alert to issues of plagiarism, attribution errors and deliberate misrepresentation or omission of prior research. If the reviewer notices missing citations or attributions, it is quite helpful to note this in the review because often these are honest omissions.
Sometimes reviewers conduct a search on the topic of the manuscript to conduct a general review of the citations in the manuscript as well as pick up on prior work or duplication of publication.
Presence of a financial interest does not necessarily prevent publication of a study. However there must be a clear statement in the text about all potential conflicts of interest and how they have been handled.
The number of authors appears to be appropriate given the study

Authorship Ethics

These criteria for authorship were developed by the ICMJE. All authors should meet criteria, and all who meet criteria should become authors.

Each author must provide evidence of meeting three conditions:

Substantial contributions to conception and design, or acquisition of data, or analysis and interpretation of data
Drafting the article or revising it critically for important intellectual content
Providing final approval of the submitted version

Every author must remember: if you take credit, you also take responsibility.

Contributions not warranting authorship include:

Providing funding, technical advice, reagents, samples, or patient data
Providing students or technical personnel who perform studies
Routine collection of data
General supervision of the research group

Types of authorship abuse¹ include:

Coercion authorship: Intimidation to gain authorship
Gift authorship: Authorship awarded out of respect
Mutual support authorship: Two or more investigators placing their names on each other's papers
Duplication authorship: Publication of the same work in multiple journals
Ghost authorship: Papers written by individuals who are not included as authors or acknowledged
Denial of authorship: Publication of work carried out by others without providing them credit

References:

1. Strange K. Authorship: why not just toss a coin? Am J Physiol Cell Physiol. 2008 Sept.Í¾ 295(3):C567575.

Next Lesson >>

Lessons

Recommended for first time student reviewers: lessons 1 through 9
Recommended for first time faculty reviewers: lessons 1, 3, 7, 8, and 9