Can Medicine Be Cured Page 7
After the war, one of Austin Bradford Hill’s students on the diploma course in Public Health at the London School of Hygiene and Tropical Medicine was a young Scot called Archie Cochrane. Brilliant, charismatic and stylish, Cochrane had taken a first in Natural Sciences at Cambridge, studied psychoanalysis in Vienna, and worked in a field ambulance unit during the Spanish Civil War. As a prisoner of the Germans during the Second World War, he had conducted a randomized controlled trial on yeast as a treatment for the fluid retention common in malnourished prisoners. Inspired by Hill, Cochrane embarked on a career as an epidemiologist, making major contributions to the understanding of lung disease in miners (pneumoconiosis). Cochrane wrote a short book in 1972 called Effectiveness and Efficiency: Random Reflections on Health Services, which became an unexpected bestseller. In this book, he argued that the NHS should take evidence from randomized controlled trials to identify which medical treatments worked. Cochrane believed that only treatments of proven effectiveness should be offered by the NHS, and that all such treatments should be costed and delivered equitably.
The rigorous statistical analysis pioneered by Richard Doll, Archie Cochrane and Austin Bradford Hill formed the intellectual and scientific basis for evidence-based medicine, which became the new orthodoxy in the 1990s; it was welcomed by many as a breath of fresh air. The New York Times declared evidence-based medicine ‘the idea of the year’ in 2001, and the phrase ‘evidence-based’ is now applied to such disparate activities as social science, public policy and even politics. The ideas behind evidence-based medicine were not new. Decades before the term was coined, Richard Asher – who was friendly with both Richard Doll and Archie Cochrane – championed the concept, even if he didn’t use this phrase. In his 1961 essay ‘Apriority’, Asher defined the expression a priori as ‘the arguments, reasoning, speculations, notions, traditions, or other support for conclusions which have not been backed by any kind of practical experiment’. From this, he derived his term ‘apriority’ to describe a kind of lazy thinking, particularly the notion of treatments that have a theoretical reason for why they might work, but for which there was no evidence that they did work: ‘Many things which in theory ought to be highly effective turn out in practice to be completely useless.’
The evidence-based medicine movement began when a group of young, sceptical doctors began to question the received wisdoms of the time – what they disdainfully called ‘expert-based medicine’. Brian Haynes, a professor of clinical epidemiology and biostatistics at McMaster University in Hamilton, Ontario (the spiritual home of evidence-based medicine), cited a lecture on Freud at medical school in the late 1960s as his Damascene moment. He asked the lecturer whether there was any evidence that Freud’s theories were true. The lecturer replied honestly that there was no such evidence: ‘I had an intense tingle in my body as I wondered how much of my medical education was based on unproved theories.’ David Sackett – also of McMaster University – is widely regarded as the ‘father’ of evidence-based medicine. In the late 1960s, he helped establish a new kind of medical school at McMaster, where students studied medicine by starting with a specific patient problem, such as breathlessness, and then learning the relevant anatomy, physiology, pharmacology, and so on. This ‘problem-based learning’ was combined with statistics and epidemiology, and has become a widely copied model of medical education. Sackett later wrote a bestselling textbook on critical appraisal of research called Clinical Epidemiology: A Basic Science for Clinical Medicine. Yet another doctor at McMaster, Gordon Guyatt, coined the phrase ‘evidence-based medicine’ in 1991 to reflect their internal medicine residency programme, which trained doctors to manage patients on the basis of what the evidence showed worked, not on what the authorities told them to do. Sackett moved to Oxford in 1994, where he became director of the Centre for Evidence-Based Medicine. Unusually for a medical academic, he visited many British district general hospitals, and always began by doing a ‘post-take’ ward round of the patients admitted the night before with the junior on-call doctors. Sackett showed them how evidence could be useful at the clinical coalface: ‘The young physicians realized that they could challenge their seniors in a way that was not possible with expert-based medicine. It was liberating and democratizing.’ The epidemiologist Iain Chalmers, along with the obstetrician Murray Enkin, created a database of perinatal trials, which formed the basis for their landmark book Effective Care in Pregnancy and Childbirth (1989). This book led to the abandonment of many dangerous practices in obstetrics and neonatology. In 1993, Chalmers established the Cochrane Centre (named in honour of Archie Cochrane). The centre conducts systematic reviews of medical interventions and diagnostic tests; these reviews are published in the Cochrane Library. The Cochrane Centre has 30,000 volunteer experts. I am proud to have been one of them.
Evidence-based medicine (EBM) took off, according to David Sackett, for two reasons: it was supported by senior doctors who were secure enough to be challenged, and it empowered young doctors. A 1996 editorial in the British Medical Journal written by Sackett and others successfully rebutted the objections to evidence-based medicine: namely, that it was old hat, impossible to practise, cookbook medicine, the creature of managers and purchasers, and concerned only with randomized trials. EBM was defined by Sackett as ‘integrating individual clinical expertise and the best external evidence’. Who could argue with such common sense? EBM introduced the concept of a hierarchy of evidence: at the top was the meta-analysis, or systematic review, which gathers data from all the trials on a given treatment. Below this was the randomized controlled trial (RCT), the gold standard for studies of new drugs. At the bottom of the evidence hierarchy were uncontrolled trials, anecdotal reports and expert opinion. Although its founders would not admit to it, there was nothing new or magical about all this; the new orthodoxy gathered together long-established ideas about sound statistical design, the elimination of logical errors in clinical trials and, most importantly, scientific integrity.
The Cardiac Arrhythmia Suppression Trial (CAST) which began in 1987, was an early triumph for the new approach. The trial was designed to determine whether drugs that prevented abnormal heart rhythms (antiarrhythmics) reduced mortality after a myocardial infarction (heart attack). Sudden death after a heart attack is often caused by such rhythm disturbances, so it seemed plausible that these drugs might reduce deaths. The trial showed that these drugs did not prevent sudden death: they actually increased mortality. It was reckoned – using the alarmist statistics so beloved of evidence-based medicine researchers – that these drugs killed more people every year than were killed during the entire Vietnam War. The routine use of these drugs was a classic example of Richard Asher’s ‘apriority’, and a blow to the mechanistic reasoning that had dominated medicine; by ‘mechanistic’ I mean the use of a therapy that seems biologically plausible, but for which there is no evidence of benefit. This trial undermined the authority of experts who had up to that point strongly recommended these antiarrhythmic drugs. The drugs may have reduced arrhythmias, but that was a meaningless surrogate metric compared to the prevention of sudden death, an outcome that was hence called ‘patient-oriented evidence that matters’ (POEMs).
Evidence-based medicine introduced easily understandable statistical concepts, notably ‘number needed to treat’ (NNT). This is a simple way of communicating the effectiveness of a medical treatment, usually a drug. The NNT is the average number of patients who need to take a drug to prevent one bad outcome, such as a heart attack or a stroke. A good example is a study published in the New England Journal of Medicine in 1998, which examined the benefit of the cholesterol-lowering drug pravastatin (one of a family of drugs known as ‘statins’) in patients with known coronary heart disease: this is called ‘secondary prevention’. The researchers randomized over 9,000 patients to either pravastatin or placebo, and followed them up for 6 years. They reported an impressive-sounding 24 per cent relative reduction in risk of death from heart disease in the group taking
the statin compared to the group taking the placebo: ‘Over a period of 6.1 years, we estimate that 30 deaths, 28 non-fatal myocardial infarctions, and 9 strokes were avoided in 48 patients for every 1,000 randomly assigned to treatment with pravastatin.’ Translating this into ‘number needed to treat’, it sounds far less exciting: 21 patients need to take the drug for 6 years to prevent an ‘adverse event’ in 1 patient; 20 of the 21 will not benefit in any way. In studies of statins in ‘primary prevention’ – where the subjects do not have heart disease – the NNT is in the hundreds. In the West of Scotland Coronary Prevention Study (1995), men aged between 55 and 65, with a serum cholesterol of greater than 6.5 mmol/litre were randomized to pravastatin or placebo for 5 years. The study reported a 28 per cent reduction in deaths from heart disease, but the raw figures tell a different story: ‘Treating 1,000 middle-aged men… with pravastatin for 5 years will result in 20 fewer nonfatal myocardial infarctions, 7 fewer deaths from cardiovascular causes, and 2 fewer deaths from other causes’: 111 men need to take this drug for 5 years to prevent 1 death; 110 of these men will not benefit. ‘Number needed to treat’ is much easier to explain to patients than relative and absolute risk, and clearly shows also that most patients taking preventive drugs such as statins and antihypertensives (for high blood pressure) will not gain from years or even decades of taking these drugs, and are far more likely to experience side effects than avoid death from a heart attack. Real patients, however, are seldom told these facts before being prescribed these drugs.
Evidence, unfortunately, is very expensive to produce, mainly because clinical trials are so costly that only Big Pharma companies can afford to run them. Some drug trials are indeed paid for by government-funded agencies such as the Medical Research Council, but these constitute a minority. Three-quarters of trials published in four of the major general medical journals (Annals of Internal Medicine, the Lancet, the New England Journal of Medicine and JAMA) are industry funded. If Big Pharma is paying for the evidence, it is hardly surprising if the evidence it produces shows its product in the best possible light; these trials are deliberately designed to maximize the commercial possibilities of the drug. The journals benefit greatly, too: for example, Merck Pharmaceuticals ordered one million reprints of its VIGOR study in 2000 (on the safety of its anti-inflammatory drug Vioxx) from the New England Journal of Medicine; these reprints were given to doctors as ‘educational’ material, enriching the journal by several hundred thousand dollars. Evidence-based medicine, which started as a noble, almost evangelical crusade, is now largely a construct of the pharmaceutical industry. In his book The Philosophy of Evidence-based Medicine, Jeremy Howick makes the point that pharma’s annexation of EBM does not necessarily invalidate its methodology:
Imagine for the sake of argument that the EBM philosophy was violently rejected in favour of the view that palm reading experts possessed the unassailable authority to decide whether an intervention had its putative effects. Special interests would then presumably focus on influencing palm reading experts, which could turn out to be far cheaper than conducting several large randomized trials. In brief, the problem that special interests corrupt medical research is a real problem independent of methodology.
Medical research is now so sick that it has itself become a patient, and the object of critical study. The meta-researcher John Ioannidis trained as an epidemiologist during the golden age of evidence-based medicine in the 1990s, working at Harvard, Tufts, Johns Hopkins University and the National Institutes of Health (NIH). As a teenager in Athens, he had achieved some national celebrity as a mathematics prodigy, and these talents proved very useful when he began to critically appraise contemporary medical research. The closer he looked, the more shocked he became: every bit of the process was riddled with error and logical fallacy. Most clinical trials asked the wrong question, recruited too few patients (and those they recruited were often atypical and unrepresentative), analysed the data incorrectly and came to the wrong conclusions. In 2005, Ioannidis published a paper in the journal PLoS (Public Library of Science) Medicine, an online journal whose guiding principle is to publish research which is methodologically sound, regardless of its perceived importance. The paper was provocatively entitled: ‘Why most published research findings are false’. This is the summary of the paper:
a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true.
The paper is the most cited and downloaded ever published by PLoS Medicine. Astonishingly, most medical researchers privately agreed with Ioannidis: he had simply articulated, in a highly technical, statistical way, what everybody knew. The statistician Douglas Altman (author of the famous BMJ paper ‘The scandal of poor medical research’) said: ‘You can question some of the details of John’s calculations, but it is hard to argue that the essential ideas aren’t absolutely correct.’
Ioannidis then published an analysis of the forty-nine most highly cited research papers in medicine over the previous thirteen years. Forty-five of the forty-nine papers described new treatments. Thirty-four of these studies were repeated, and in fourteen (41 per cent) the original claim was shown to be wrong or grossly exaggerated. Ioannidis turned his attention next to ‘nutritional epidemiology’, and its vast output of publications linking dietary factors with cancer. Ioannidis and his colleague Jonathan Schoenfeld selected 50 common ingredients from random recipes in The Boston Cooking-School Cook Book, and found that 40 (80 per cent) were the subject of 264 studies published in medical journals on their cancer risk: ‘Thirty-nine per cent of studies concluded that the studied ingredient conferred an increased risk of malignancy; 33 per cent concluded that there was a decreased risk, 5 per cent concluded that there was a borderline statistically significant effect, and 23 per cent concluded that there was no evidence of a clearly increased or decreased risk.’ When Schoenfeld and Ioannidis dissected these studies, they found that ‘the great majority of these claims were based on weak statistical evidence’, and almost none of these alleged associations was found to be significant when subjected to meta-analysis. Ioannidis told the Washington Post: ‘I was constantly amazed at how often claims about associations of specific foods with cancer were made, so I wanted to examine systematically the phenomenon. I suspected that most of this literature must be wrong. What we see is that almost everything is claimed to be associated with cancer, and a large portion of these claims seem to be wrong indeed.’
Ioannidis concluded – more in sorrow than in anger – in a 2016 paper for the Journal of Clinical Epidemiology that ‘evidence-based medicine has been hijacked’. The paper takes the unusual form of a personal letter to his mentor, David Sackett, who had died in 2015:
The industry runs a large share of the most influential randomized trials. They do them very well… It is just that they often ask the wrong questions with the wrong short-term surrogate outcomes, the wrong analysis, the wrong criteria for success and the wrong inferences…
… corporations should not be asked to practically perform the assessments of their own products. If they are forced to do this, I can’t blame them, if they buy the best advertisement (i.e. ‘evidence’) for whatever they sell.
Sackett, however, was well aware of this hijacking and had written a spoof article many years earlier in 2003 for the British Medical Journal, in which he announced the foundation of HARLOT plc that specialized in How to Achieve positive Results without actually Lying to Overcome the Truth:
HARLOT plc will provide a comprehensive package of services to discriminating trial sponsors who don’t wa
nt to risk the acceptance and application of their products and policies amid the uncertainties of dispassionate science. Through a series of blind, wholly owned subsidiaries, we can guarantee positive results for the manufacturers of dodgy drugs and devices who are seeking to increase their market shares, for health professional guilds who want to increase the demand for their unnecessary diagnostic and therapeutic services, and for local and national health departments who are seeking to implement irrational and self-serving health policies.
Ioannidis uses the term ‘The Medical Misinformation Mess’ to encompass all of these issues. Most doctors, and nearly all patients, are unaware of this mess. Even those doctors who are aware generally lack the critical skills needed to evaluate the evidence; they are statistically illiterate. Paul Glasziou, professor of evidence-based medicine at Bond University, Australia, has argued that teaching such critical appraisal skills should be a core part of medical education: ‘A twenty-first-century clinician who cannot critically read a study is as unprepared as one who cannot take a blood pressure or examine the cardiovascular system.’ Medical education, however, does not encourage scepticism. The Czech polymath and contrarian Petr Skrabanek (1940–94) taught these skills at Trinity College Dublin Medical School during the 1980s and early 1990s, and lamented that ‘my course on the critical appraisal of evidence for medical students can be compared to a course on miracles by a Humean sceptic for prospective priests in a theological seminary’. Medical education overvalues training and rote memorization at the expense of education, scholarship and the cultivation of the critical faculty.
The most important thing I learned during the three years I spent as a research fellow is that nearly all papers in medical journals are dross. Ioannidis, along with other meta- researchers such as Glasziou and Sir Iain Chalmers, estimate that about 85 per cent of medical research is useless and wasted. This global waste amounts to £170 billion annually. As far back as 1954, Richard Asher warned that ‘the danger of crooked statistics is that its fallacies are less likely to be noticed because of the mixture of awe, suspicion, and reverence with which statistical thinking is regarded by most of us’. Alvan Feinstein warned about excessive reliance on randomized controlled trials and meta-analyses in 1997, and predicted the advent of authoritarian practice guidelines: