Size of the Effect in Treating COVID-19
With COVID-19 still causing thousands of deaths worldwide (over half a million as of this writing) the search for a vaccine and a treatment continues at hospitals around the globe. A breakthrough on treatment of some patients was announced back on June 16th when doctors at Oxford University posted research on the steroid dexamethasone. Their study indicated a reduction of the mortality rate by about one-third for patients suffering from the most severe symptoms of COVID-19. Researchers stopped the study before completion because the steroid was so successful and researchers wanted to share the good news with physicians so they could begin using the treatment. Even better news – dexamethasone is widely available and inexpensive.
What does this have to do with Lean Six Sigma? It turns out there are some key lessons in how this study was conducted and data analyzed which have direct parallels to some important LSS tools and concepts.
First – a little background: The study began in the UK in March 2020 at 175 different hospitals. Named the RECOVERY trial (Randomised Evaluation of COVid-19 thERapY), the study included over 11,500 individuals. Of those patients, the researchers randomly assigned a total of 4,321 to receive usual care and 2,104 received usual care plus dexamethasone.
In analyzing the results researchers grouped patients into three levels of severity: Those requiring ventilation (most severe), those requiring oxygen (less severe), and those not requiring any respiratory support (least severe). The question they sought to answer: Did the steroid dexamethasone reduce the mortality rate of patients in each group, and if so, to what degree?
In LSS terms, the medical researchers were conducting a Designed Experiment (often called Design of Experiments or DOE). One factor in the experiment was the treatment administered (usual care and usual care plus dexamethasone). A second factor was patient status (severity level: low / medium / high). The response was the mortality rate, which was measured after 28 days of treatment.
Here is a summary of the results:
Let’s start with the first row. For those patients who required ventilation – considered the highest level of severity – the mortality rate after 28 days of treatment with usual care was 41%. In other words, on average 41 out of 100 patients in this group died. By contrast, patients with the most severe symptoms who received dexamethasone had a mortality rate of 26.7% (i.e., 27 out of 100 died in this group). Which means an additional 14 people survived out of 100. That works out to an improvement of a little over one-third based on the ratio of mortality rates (26.7/41 = 0.65 or 35% improvement in the rate).
This difference in the mortality rate from 41% to 26.7% represents the size of the effect in this designed experiment. Again, doctors measured this effect for patients with the most severe symptoms. For medical researchers an effect of 35% is huge. This result makes it clear why researchers stopped the study. Using this steroid could save lives – many lives.
On LSS projects the size of the effect is measured in the units of the KPOV – key process output variable. That could be the process lead time in days, scrap rate in parts per week, productivity level in units per hour, error rate in mistakes per home loan, and so on. Pilot solutions are often implemented as part of the Improve Phase, and the size of the effect is measured by comparing the results from the pilot to the baseline from the Measure Phase.
Back to the study: What about the effect of dexamethasone on patients with less severe symptoms? Well, for those with a medium level of severity – requiring oxygen – the mortality rate dropped from 25% (usual care) to 20% (usual care plus dexamethasone). That translates to an improvement rate of 20% (20/25 = 0.8 or a 20% reduction). Hence, in a group of 100 patients with this severity level five fewer people given dexamethasone would die (20 instead of 25).
For patients with the lowest level of severity it turns out giving them dexamethasone resulted in no statistically significant effect. In other words, the mortality rate was the same whether patients were treated with dexamethasone or received the usual care.
At this point some of you may be asking: How do we know if these effects are the true effects on decreasing the mortality rate? Good question! As it turns out the actual effect may be higher – or lower – than the figures shown above. Oxford researchers included this additional information:
Let’s look at the results for the most severe patients who required ventilation. First, we have the improvement rate ratio of 0.65 which was calculated from the study data. Next, there is a 95% confidence interval for this same ratio. For the most severe patients the confidence interval ranges from a ratio of 0.48 (52% reduction) up to 0.88 (12% reduction). In other words, the true improvement rate could be higher or lower than the estimated ratio of 0.65.
Lastly, we have a p-value. What are the null hypothesis and alternative hypothesis?
- Null Ho: There is no effect on the improvement rate ratio for patients in this group
- Alternative Ha: There is an effect on the improvement rate ratio
The p-value = 0.0003 for patients requiring ventilation (most severe). This value is low (well below the commonly used alpha of 0.05) so we reject the null and conclude there is an effect. For patients in the medium level of severity there is also a low p (p=0.0021) so we can again reject the null hypothesis. But for the least severe patients the p-value = 0.1400 which is high. This indicates there was no effect on the mortality rate for this group of patients when given dexamethasone.
Finally, to the practical application of this research. Imagine you are the head physician in charge of the COVID-19 ward in a hospital. And you have a limited supply of dexamethasone. You would reserve most of that supply for patients with the most severe symptoms, a smaller quantity for patients with a medium level of symptoms, and none for those with the least severe symptoms.
You can learn more by reviewing this research paper on the Oxford University website:
