Grounds for Ambiguity: Justifiable Bases for Engaging in Questionable Research Practices

Abstract

The current study sought to determine research scientists’ sensitivity to various justifications for engaging in behaviors typically considered to be questionable research practices (QRPs) by asking them to evaluate the appropriateness and ethical defensibility of each. Utilizing a within-subjects design, 107 National Institutes of Health principal investigators responded to an invitation to complete an online survey in which they read a series of research behaviors determined, in prior research, to either be ambiguous or unambiguous in their ethical defensibility. Additionally, each behavior was paired with either an ostensibly sound or unsound reason for the behavior. Consistent with hypotheses, the results indicated that scientists perceived QRPs as more appropriate and defensible when paired with a justifiable motive relative to when paired with a clearly unethical motive, particularly for QRPs that are more ambiguous in their ethicality. In fact, ambiguous QRPs were perceived as categorically defensible when given a justifiable motive. This suggests scientists are sensitive to contextual factors related to QRPs’ appropriateness, which could inform how institutions develop appropriate training modules for research integrity.

Appendix

UU-QRPs with Unjustifiable Motives

• 1. A researcher is coauthoring a manuscript using data he collected and used previously for another study and separate paper. He does not indicate the origin of his data, because the hypotheses and variables analyzed in the current manuscript differ from those in the prior work.

• 2. A researcher submits a paper for publication without receiving direct permission from all coauthors, since as the lead author, she believed she could speak for all of her coauthors.

• 3. A researcher investigating the preliminary effects of a new pharmaceutical withholds journal submission of a manuscript at the request of the corporate sponsor, because the sponsor wants the paper’s publication to coincide with the release of the new drug.

• 4. A researcher’s coauthor wants to conduct his own analyses on some data recently collected by the research team, but the Principal Investigator refuses to share the raw dataset to ensure that he alone has access to the findings.

• 5. A researcher fails to replicate a finding that she found recently in another study. She decides to publish the results from the first study alone, without reporting the failed replication attempt, since she is confident that the first study represents the truth.

• 6. An employee of a corporate sponsor of the research analyzes the data for publication and is listed as one of the study’s co-authors. Though the sponsor’s involvement in the project is clearly indicated in the manuscript, the employee’s relationship to the sponsor is not disclosed.

UU-QRPs with Justifiable Motives

• 1. A researcher is coauthoring a manuscript using data he collected and used previously for another study and separate paper. He does not indicate the origin of his data, because his graduate school mentor used to do the same thing, leading him to infer that this is an acceptable practice.

• 2. A researcher submits a paper for publication without first receiving direct permission from all coauthors, because the coauthors had approved earlier, similar drafts.

• 3. A researcher investigating the preliminary effects of a new pharmaceutical withholds journal submission of a manuscript at the request of the corporate sponsor, because the sponsor wants additional evidence that the drug has no significant side effects.

• 4. A researcher’s coauthor wants to conduct his own analyses on some data recently collected by the research team, but the Principal Investigator refuses to share the raw dataset in order to protect participants’ confidential information.

• 5. A researcher fails to replicate a finding that she found recently in three previous studies. She decides to publish the results from the original three studies alone, without indicating the failed replication attempt, since she is confident that the first three studies represent the truth.

• 6. An employee of a corporate sponsor of the research analyzes the data for publication and is listed as one of the study’s co-authors. The sponsor’s involvement in the project is clearly indicated in the manuscript, and the employee’s relationship to the sponsor is disclosed.

AU-QRPs with Unjustifiable Motives

• 1. The results from a researcher’s most recent study are trending toward significance, which prompts the researcher to collect data from an additional ten participants beyond what was estimated from an a priori power analysis. The original data set was sufficiently powered to detect effects and no participants were excluded from analyses. Following this extra data collection, the results become conventionally significant and are then published, but without disclosing the post hoc modification of the research plan.

• 2. Upon a preliminary analysis of a data set, a researcher observes that she is about 20 observations away from what her original study design called for, but since her hypothesis is already supported by the existing data, she stops data collection to write up the results for publication immediately.

• 3. So as to meet a target journal’s standards for statistical significance, and because the journal requires reporting p values only to the hundredth place, a researcher reports the p value of a finding as p = 0.05, rounding down the actual significance observed, which was p = 0.053.

• 4. In a submitted manuscript, a researcher acknowledges a statistician for her technical assistance without asking the statistician for permission to do so, simply assuming the statistician would appreciate the acknowledgement.

• 5. A researcher publishes statistically significant results, but statistical significance was attained only by excluding data from two participants after the researcher becomes convinced that these participants probably should not have been included in the study to begin with.

• 6. A researcher conducts a study in which two outcome measures out of four yield significant results. She reports the results of only the two significant measures, and fails to mention the two others, so as to present the results as more consistent and impressive than the entirety of her data supports.

• 7. A researcher obtains unexpected results from what she predicted for a study and drafts a manuscript reporting these results as having been predicted from the start, reporting the results in the discussion as support for predicted relationships.

• 8. A researcher conducts a study with two independent variables, with an a priori prediction that there should be a statistical interaction between the two factors. However, the statistical interaction is not conventionally significant. He omits the results of this analysis in his paper in order to conduct post hoc tests, which provide support for more specific predictions, and does not disclose that he predicted a statistical interaction between the two variables.

• 9. Several of a researcher’s colleagues have questioned why she has not yet submitted for publication a paper they had all recently coauthored. Because she is under pressure from her department chair to publish in high-impact journals, she is delaying submission because she hopes to discuss the paper at an upcoming conference with the editor of a journal she has in mind. She hopes that by delaying publication, it will increase the likelihood of publication in a high-impact journal.

• 10. A researcher conducts two different but conceptually similar studies that she intends to make into two different papers. For convenience, she re-uses the literature review from the first-drafted paper for the second without disclosing the reuse of this material.

• 11. So as to garner more publications, a researcher separates out several aspects of a large study he recently conducted, making each an independent manuscript, but without disclosing the relationship of the papers to each other in each manuscript.

• 12. At the suggestion of a corporate sponsor of the research, a researcher changes her planned methods of statistical analysis, even though she believes her original methods more accurately represent the data.

AU-QRPs with Justifiable Motives

• 1. The results from a researcher’s most recent study are trending toward significance, which prompts the researcher to collect data from an additional ten participants beyond what was estimated from an a priori power analysis. Ten participants from the initial sample were excluded from analysis after failing an attention check, thus necessitating the additional ten participants for the study to attain adequate statistical power. The results then become conventionally significant and are published; the researcher discloses details regarding post hoc modification of the research plan.

• 2. Upon a preliminary analysis of a data set, a researcher observes that she is about 20 observations away from what her original study design called for, but since her hypothesis is already supported by the existing data, she stops data collection to preserve unused research funds for a follow-up study that will determine the reliability of the original findings.

• 3. Because a target journal limits reporting p values to the hundredth place, a researcher reports the p value of a finding as p = 0.05, rounding down the actual significance observed, which was p = 0.053. He acknowledges the finding was marginal but did not reach conventional statistical significance.

• 4. In a submitted manuscript, a researcher acknowledges a statistician for her technical assistance without asking the statistician for permission to do so, unaware that it is impolite to acknowledge someone without her permission.

• 5. A researcher publishes statistically significant results, but statistical significance was attained only by excluding data from two participants after a statistical test for outliers confirmed the appropriateness of removing these participants’ data.

• 6. A researcher conducts a study in which two outcome measures out of four yield significant results. She reports the results of only the two significant measures, but mentions the other two in a description of her methods, despite not reporting findings from them, because these other measures had insufficient reliability.

• 7. A researcher obtains unexpected results from what she predicted for a study and drafts a manuscript reporting these results as not having been predicted from the start, offering a tentative post hoc explanation for the findings in the paper’s discussion.

• 8. A researcher conducts a study with two independent variables, with an a priori prediction that there should be a statistical interaction between the two factors. However, the statistical interaction is not conventionally significant. He reports in the manuscript that although the statistical interaction was not significant, he conducted post hoc tests, which supported his more specific predictions, but indicated that such findings should be interpreted cautiously in light of the non-significant statistical interaction.

• 9. Several of a researcher’s colleagues have questioned why she has not yet submitted for publication a paper they had all recently coauthored. She hopes to collect data from a few more studies to confirm the initial results, which she hopes would help place the paper in a high-impact journal.

• 10. A researcher conducts two different but conceptually similar studies that she intends to make into two different papers. As she had seen her graduate school mentor do several times, she re-uses the literature review from the first-drafted paper for the second without disclosing the reuse of this material.

• 11. Consistent with what often occurs in his field, a researcher separates out several aspects of a large study he recently conducted, making each an independent manuscript, but without disclosing the relationship of the papers to each other.

• 12. At the suggestion of a journal manuscript peer reviewer, a researcher changes her planned methods of statistical analysis, even though she believes her original methods more accurately represent the data.