For those in the field of clinical research and those with a vested interest in it, randomized control trials (RCTs) are kind of a big deal. RCTs are the gold standard by which empirically based treatments (EBTs) are vetted. More and more, controversial treatment guidelines and algorithms used by insurance companies require RCTs when determining the necessity of a given treatment. But RCTs can also be performed in a way that meets the criteria of an RCT but is completely meaningless. Some RCTs are nothing but an exercise in procedure. This means that it can be easy to set up an RCT just for the purpose of obtaining the scientific credibility that is attributed to them.
To shed light on the problem, the prestigious British Medical Journal published a satire article entitled, “Parachute use to prevent death and major trauma when jumping from aircraft: randomized controlled trial” The article was quite funny and got a great deal of popular and professional attention. The fictional study the article discusses, “[whether] if using a parachute prevents death or major traumatic injury when jumping from an aircraft.”
The study compared “(j)umping from an aircraft (airplane or helicopter) with a parachute versus an empty backpack.” Setup as a rigorous RCT, the researchers concluded that, “(p)arachute use did not reduce death or major traumatic injury when jumping from aircraft in the first randomized evaluation of this intervention.”
The article describes the procedure, makes it clear that it was done in a way that was methodologically sound, and worked with a sufficiently large data set. So how could they come to such a ridiculous conclusion? Well, as the article points out, the researchers were “only able to enroll participants on small stationary aircraft on the ground.”
The way the article explains away their inability to test parachute use on people jumping from planes in actual flight is exceedingly telling. It shows that a circumstance that makes an RCT completely invalid can be disguised using jargon and authoritative language. This is how they told the reader that the airplanes from which participants jumped were parked on the ground;
“[ … ] participants in the study were similar to those screened but not enrolled with regard to most demographic and clinical characteristics. However, participants were less likely to be on a jetliner, and instead were on a biplane or helicopter (0% v 100%; P<0.001), were at a lower mean altitude (0.6 m, SD 0.1 v 9146 m, SD 2164; P<0.001), and were traveling at a slower velocity (0 km/h, SD 0 v 800 km/h, SD 124; P<0.001).”
There are a few lessons we can take away from this. First and foremost, technical RCTs can be set up that meet all of the criteria to be considered rigorous but omit one critical factor in measuring the phenomenon at issue. Like in the parachute story, taking the subjects up on an actual flight would be extremely time consuming and expensive.
The fictional researchers could push a lot more people off of a stationary biplane than they could out of a helicopter 100 feet in the air. It takes time for aircraft to get off the ground and land. There are large amounts of on the ground and in the air coordination that has to go on to fly and land planes safely. And airplane fuel is very expensive. So, the fictive researchers simply decided to forgo the hard part of the RCT.
The lesson here is that because RCTs are in such high demand, there is a great risk that research firms will simply leave out the most important, most difficult and expensive parts of their study and explain it away in such a way that most people will simply skim that part of the paper.
Todd Essig, a Forbes contributor, and leadership strategist writes, “[ … ] the real bottom line lesson is that EBTs need lots more building blocks in addition to RCTs. Otherwise too many dumb RCTs will inform clinical decision making. Too often studies conclude with something like the following from the “parachute study” just because it’s the conclusion from an RCT: “Our findings should give momentary pause to experts who advocate for routine use of parachutes for jumps from aircraft in recreational or military settings.”
The real danger is that we have no idea how many drugs and medical interventions are being foisted on the public which have been tested using RCTs that have had their critical clinical context completely omitted. We do know that insurance companies, drug merchants, and health care institutions have a vested interest in seeing that RCT write up and that it looks impressive. And with high demand- come those who would take shortcuts.
Essig warns that “An RCT taken out of the total clinical context is more of a lead standard than a gold standard.”
He cautions us to read carefully when someone announces that they have run a successful RCT. They could, he says, be full of hot air.