In my first job as a statistician in biomedical research, I was shocked by how much mental effort and time were devoted to the process of writing, editing, and submitting manuscripts to research journals. I understood that publication was necessarily an important part of research. The research had little, if any, value if it wasn’t shared with other researchers and the public. But it was astonishing how considerations around getting published — the potential reaction of peer reviewers and editors, the chances of getting the paper accepted — were top of mind at nearly every step, in so many of the decisions; they consumed so much time and thought that otherwise could have been spent on the research itself. One principal investigator even gave me and a faculty statistician a mini lecture on his personal theory of manuscript titles, enumerating the different types and each of their pros and cons. Most investigators weren’t quite that absurd. Still, it was common for the question, “How will this look (to reviewers or journal editors)?” to be more central in a decision about a study or analysis than the question, “What makes sense scientifically?”
The research publication game is on my mind because most of my recent work has been to help a study team prepare and submit a paper based on a cohort of patients that finished enrolling 7 months ago. I prepared the code for the statistical analysis in advance, so the results were available with a few button clicks as soon as the study enrollment concluded. Drafting and revising a manuscript started almost immediately, but as of today, the paper is not yet accepted by a journal. Publication may be several months away still. This is not even a prolonged timeline for a manuscript, relatively speaking, but it has been tedious and exhausting at times. It prompted me to make some observations about the writing of journal manuscripts for collaborative, quantitative research.
The taboo on describing negative or inconclusive study results
First, there’s the widely acknowledged pressure that is felt, and partly generated, by investigators to publish positive findings. This might be what frustrates me the most in biomedical research. Sometimes it’s as if investigators have had the brain tissue responsible for acknowledging disappointing or ambiguous study results surgically excised; the possibility of such results simply doesn’t register.
Recently I’ve witnessed some more egregious examples of suppressing undesirable analyses. In these cases, not only was an investigator clearly pushing for a choice of analysis or presentation that would throw more favorable light on their own research program, contrary to a more objective view of the data, but the investigator also was willing to state all of this out loud. The investigator said, without apparent qualms, that they didn’t want to publish a certain graph or statistic because it would make, for example, a certain measurement favored by their research team look bad. In other cases, I’ve suspected ulterior motives were driving a suggestion or decision, but the investigator wouldn’t have admitted those motives unless really pressed; most investigators feel an obligation to maintain at least an appearance of scientific integrity.
It’s said that journal editors are partly to blame because they’re biased towards publishing research with positive findings. It’s probably true that bias exists in editorial decisions, but I’ve personally witnessed little, if any. Among the 20 or so papers to which I’ve contributed, and that were submitted to a journal, I don’t recall any that appeared to get rejected by a journal because of the results of the study. In all of the cases that I remember of the journal rejecting or requiring heavy revision of a manuscript, either the study was not addressing a particularly interesting or relevant question; or it was using data that were inadequate in quantity or quality for the study to produce reliable, definitive results; or there was questionable reasoning in the conception or interpretation of the study.
In particular, I’ve seen a journal require heavy revision of a manuscript because the presentation and interpretation of the study results were so obviously trying to draw attention away from, and deny, disappointing results for pre-specified analyses. Putting a positive spin on the results can have the opposite of the intended effect, if the intent is to get the manuscript accepted for publication.
It’s possible that I don’t yet have enough experience to understand the difficulty of publishing study results that are disappointing or a mix of positive and negative, but I’m inclined to think that many investigators overestimate the importance of study results; that they could be more successful if they put as much, or more, energy into the conception and execution of a study as they put into the framing and interpretation of the results; and that the reason for their pointing to reviewer and editor reactions to their manuscripts as the cause of their being careful to frame things positively is that blaming reviewers and editors requires less mental effort than changing one’s approach to planning and executing research. Careful, scientific thinking and planning is difficult and requires patience and focus. Many MD investigators are hyperactive, impatient types accustomed to multitasking and making a quick succession of decisions in their clinical practice. It might be that prolonged planning or discussion early in the life of a study makes them restless, and they prefer just to see some data and run with it.
The Instagram style of research
Of course, even if there were no bias towards positive study results, either genuine or merely perceived, in the acceptance of manuscripts for publication (or in consideration for funding), there would remain a tendency for investigators to color their presentation of study results in a self-serving manner. There would remain a natural, human concern for one’s public image. For the same reason that people curate their Instagram and Facebook accounts to appear happy, beautiful, and interesting, researchers will tend to curate the records and products of their work so that they appear productive and clever.
I’ve encountered investigators who seemed so driven by image alone that I would characterize their work more as marketing than research. One investigator was fixated on a figure from another research group’s paper, not because it was particularly enlightening about the disease under study, but because it got a lot of attention and was frequently shown in others’ research talks, etc. This investigator asked me to create a figure like it from our data, even though our data were quite different and such a figure couldn’t be expected to be informative or pertinent to the question we were trying to answer. I doubt that the investigator understood, or cared much about, what information the figure actually contained; what mattered was that it represented the image of attention-grabbing research.
The complications of collaborative writing
Nearly all research manuscripts are written collaboratively. The research itself rarely can be carried out by anyone alone. Necessarily, for any given study, there are multiple people who have substantially contributed, and input from each of them is needed just to describe the study and the analysis results accurately. Additionally, it seems common to request input from people who have not contributed substantially to the study itself. This seems to be partly a matter of politics and professional courtesy. For example, a study might use data collected from patients seen in a particular department of a research hospital. Although there may be only 2–3 people directly involved in carrying out the study, the entire department was involved in treating the patients, and whatever is published about the study may be seen to reflect on the entire department. Thus, the study authors often send the manuscript draft around to others in the department and solicit feedback. Although these peripheral contributors usually provide no more than a few brief comments, they will be included among the authors for the publication, and whoever has primary responsibility for the paper has to exercise judgment in whether to incorporate and reconcile the feedback.
Composing a journal manuscript would be a complicated, arduous task even if one person possessed all the necessary information and were writing alone. Collaborative writing is even more complicated. There’s the question of when the lead author should solicit input, and from whom. There’s the problem of keeping track of and reconciling different versions of the manuscript while multiple people concurrently suggest edits. And there’s the difficulty of avoiding either incorporating suggestions that would have been better ignored, or failing to incorporate suggestions that would have improved the paper.
An additional complication in academic research is that often the task of actually writing or assembling the text and managing edits is assigned to a research trainee with little experience, for training purposes. This means that sometimes the person working most directly with the manuscript is not the person with the best understanding of the study, or with the most skill or interest in writing. This is not to say that trainees shouldn’t be given this role. It can, however, make the process less efficient.
What’s conspicuous to me about this collaborative writing process is that as complicated as it is, and as much time as it consumes, there seems to be little or no training about how to manage it effectively. Even some highly experienced research groups go about it in a mostly unsystematic, slightly chaotic manner. Further, it seems rare to use any modern tools for collaboration. Nearly always, a Word file is emailed around repeatedly, with each person appending their initials to the end of the file name when they make edits.
Is there not a better way to do this? For one thing, manuscripts should be developed only in outline form until the content and its order of presentation are nailed down! Distilling a study’s essential methods and results, and giving the information a logical, digestible structure, takes time and will determine the quality of the end product much more than the exact choice of words or placement of punctuation, yet I see people go straight to editing these kinds of details in early-stage drafts, even before the main point of the study is clear. I think it would be ideal to keep the manuscript in the form of a bulleted list, with only fragments of English, for as long as possible so that no one is tempted to fiddle with details until everyone is satisfied with the content and structure.
Hopes for the future of publishing research
If biomedical research were really about science and improving patient care, and not about marketing (curating an image of one’s research), media attention, and advancing one’s career, then I would expect the people involved to find a way to spend less time on writing and revising papers. I haven’t seen much effort in this direction.
When it comes to sharing research methods and findings, there are hardly any technological limits. Why does it continue to be done basically the same way as it was done hundreds of years ago, with researchers composing paragraphs of text (despite most academics being pretty bad writers) and a small set of figures and tables that are carefully arranged into static pages? Why not present the information from a study in a structured, dynamic, web-based format that requires hardly any English composition and allows one to quickly find the information one is looking for? The “publication” process could start as soon as the study design and analysis plan are finalized, with the study webpage continually updated. Fellow researchers or other informed readers could comment on the study methods even before the data are collected. “Publishing” the study results would be as simple as inserting the numbers and graphics into designated slots.
What I have in mind is similar to how clinical studies and their results are posted on clinicaltrials.gov, except that it would have more features and would be the primary means of presenting one’s study, rather than being mainly a device for accountability.
There should be a space where researchers publicly comment on one another’s research findings. Rather than authors writing a Discussion section for their own paper — a section that many times seems pointless, and never is actually a discussion — there should be a true discussion held in the form of online comments, replies, etc., among the interested parties. Peer review should not be a secretive, one-time process involving only a few peers. It should be open, ongoing, and transparent.