…The apparent endemicity of bad
research behaviour is alarming…
—Dr. Richard Horton
…The myth of a separate mode based on rigorous objectivity and arcane, largely mathematical knowledge, vouchsafed only to the initiated, may provide some immediate benefits in bamboozling a public to regard us as a new priesthood…the myth of an arcane and
enlightened priesthood of scientists…
—Stephen Jay Gould
Dr. Richard Horton is the editor-in-chief of one of the most well respected peer-reviewed medical journals in the world The Lancet. In a comment titled Offline: What is medicine’s 5 sigma? (Vol 385, April 11, 2015 AD) he noted that as per Chatham House rules he would not name who it is who stated this to him, “A lot of what is published is incorrect.” Chatham House refers to The Royal Institute of International Affairs which is a London based non-governmental organization that analyses major international issues and current affairs. Technically its “rules” pertain to the manner whereby to hold polemical debates and discussion panels according to which attendees can quote speakers but cannot reveal their identity: in order words, anonymous quotations. At a particular meeting the rules were kick up a notch as it was requested that no photos of slides be taken, some altogether “pleaded that their comments especially remain unquoted,” pertaining to “a chilling state where severe restrictions on freedom of speech are placed on anyone on the government’s payroll.”
This was within the context of a symposium on “the reproducibility and reliability of biomedical research” which took place at the Wellcome Trust.
But “Why the paranoid concern for secrecy and non-attribution?” asks Dr. Richard Horton and he replies by noting that “one of the most sensitive issues in science today: the idea that something has gone fundamentally wrong with one of our greatest human creations” but what is that? The scientific method itself or rather, how it is referred, etc. to the effect that “much of the scientific literature, perhaps half, may simply be untrue.”
This is, in part, something upon which I touched in my series Scientific Cenobites as Dr. Horton puts it, “Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness” he anonymously quotes that “poor methods get results.”
He notes that “Journal editors” such as himself (the exalted referees) “aid and abet the worst behaviours. Our acquiescence to the impact factor fuels an unhealthy competition to win a place in a select few journals”—he refers to journals as “miscreants.”
Just because the scientific method and the institutions which have arisen around it are impersonal does not mean that scientists are impersonal. Dr. Richard Horton notes “scientists too often sculpt data to fit their preferred theory of the world. Or they retrofit hypotheses to fit their data” and more to the point, “Our love of ‘significance’ pollutes the literature with many a statistical fairy-tale.”
Then there is the follow the money aspect as “Universities are in a perpetual struggle for money” and certain “National assessment procedures, such as the Research Excellence Framework, incentivise bad practices. And individual scientists, including their most senior leaders, do little to alter a research culture that occasionally veers close to misconduct.”
Along with such incentives towards bad practices is the problem that “no-one is incentivised to be right. Instead, scientists are incentivised to be productive and innovative.” He asks whether “a Hippocratic Oath for science” would help. Well, there is the Hippocratic Oath which is taken by personages within the medical realm and yet, there are such personages who are somehow allowed to call themselves “doctor” and who murder healthy human babies for a living—they must have taken the Hypocritical Oath.
“Following several high-profile errors” within the particle physics community, Oxford Professor of Particle Physics, Tony Weidberg, “worried we set the bar for results in biomedicine far too low.”
In any regard, “The conclusion of the symposium was that something must be done. Indeed, all seemed to agree that it was within our power to do that something. But as to precisely what to do or how to do it, there were no firm answers”—likely no incentive—thus, “The bad news is that nobody is ready to take the fi rst step to clean up the system.”
This brings us to John P. A. Ioannidis’ paper “Why Most Published Research Findings Are False,” PLoS Med 2(8): e124, August 30, 2005 AD, herein it is noted that “The probability that a research claim is true may depend on study power and bias…a research finding is less likely to be true” in part based on “analytical modes” and “when there is greater financial and other interest and prejudice.” Note that “analytical modes” means interpretations and this is touched upon thusly, “Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.”
In fact, “Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true.”
Note that “p-value” refers to the probability of obtaining the observed sample results (or a more extreme result) when the null hypothesis is actually true.
The paper note:
…unfortunately, there is a widespread notion that medical research articles should be interpreted based only on p-values. Research findings are defined here as any relationship reaching formal statistical significance, e.g., effective interventions, informative predictors, risk factors, or associations. “Negative” research is also very useful. “Negative” is actually a misnomer, and the misinterpretation is widespread. However, here we will target relationships that investigators claim exist, rather than null findings.
Moreover, “Unfortunately, in some areas, the prevailing mentality until now has been to focus on isolated discoveries by single teams and interpret research experiments in isolation.”
Also:
The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.
Conflicts of interest and prejudice may increase bias…Scientists in a given field may be prejudiced purely because of their belief in a scientific theory or commitment to their own findings. Many otherwise seemingly independent, university-based studies may be conducted for no other reason than to give physicians and researchers qualifications for promotion or tenure. Such nonfinancial conflicts may also lead to distorted reported results and interpretations. Prestigious investigators may suppress via the peer review process the appearance and dissemination of findings that refute their findings, thus condemning their field to perpetuate false dogma.
The paper defines “bias as the combination of various design, data, analysis, and presentation factors that tend to produce research findings when they should not be produced.”
It moreover states:
What is less well appreciated is that bias and the extent of repeated independent testing by different teams of investigators around the globe may further distort this picture and may lead to even smaller probabilities of the research findings being indeed true…the proportion of probed analyses that would not have been “research findings,” but nevertheless end up presented and reported as such, because of bias…Bias can entail manipulation in the analysis or reporting of findings. Selective or distorted reporting is a typical form of such bias…with increasing bias, the chances that a research finding is true diminish considerably.
We are also told of “the potential for transforming what would be ‘negative’ results into ‘positive’ results, i.e., bias…Conflicts of interest and prejudice may increase bias.”
As it stands:
…even in the most stringent research designs, bias seems to be a major problem. For example, there is strong evidence that selective outcome reporting, with manipulation of the outcomes and analyses reported, is a common problem.
Not surprisingly but shockingly, the paper notes, “Claimed Research Findings May Often Be Simply Accurate Measures of the Prevailing Bias” some research ends up “measuring nothing else but the net bias that has been involved in the generation of this scientific literature.”
It is also noted, “History of science teaches us that scientific endeavor has often in the past wasted effort in fields with absolutely no yield of true scientific information, at least based on our current understanding”—can you say “Darwinism”?
Of course, “Diminishing bias through enhanced research standards and curtailing of prejudices may also help. However, this may require a change in scientific mentality that might be difficult to achieve.”
~~~~~~~~~~~~~~~~~~~~~~~~
A plea: I have to pay for server usage and have made all content on this website free and always will. I support my family on one income and do research, writing, videos, etc. as a hobby. If you can even spare $1.00 as a donation, please do so: it may not seem like much but if each person reading this would do so, even every now and then, it would add up and really, really help. Here is my donate/paypal page.
Due to robo-spaming, I had to close the comment sections. However, you can comment on my Facebook page and/or on my Google+ page.
Twitter: #Science, #Lancet, #journals
Facebook: #Lancet, #journals, #Science