Dislaimer: the authors, Petrus Potgieter and Bronwyn Howell, are not clinicians but both have extensive experience in data analysis. Petrus has a doctorate in mathematics and Bronwyn has a doctorate in healthcare policy and economics. The piece was written on 8 April 2020.
As we write this, the economic life of the world has already been severely strained by the impact of the novel corona virus disease and by the steps taken by authorities to prevent its spread. Numbers of infections and fatalities are reported almost continuously on sites like Worldometer1, in minute detail. Public policy is said to be aimed at containment or "flattening the curve" but this will require data and metrics.
Two key numbers are of interest to the general public as well: the infection fatality rate (IFR, number of fatalities per number of infections) which is likely the same in all regions, when properly adjusted for demographics like age and other risk factors, and the population prevalence (fraction of the population currently infected) as well as the trajectory of the latter number. Measuring the IFR requires establishing not only the accurate number of deaths attributable to the virus (as opposed to possibly cases in which it is a comorbidity) but also the number of cases of infections (the prevalence of the disease). The public (like medical professionals) also has an interest in testing protocols that accurately identify persons for treatment, possible quarantine or perhaps for release from an obligation to self-isolate.
Estimates for the case fatality rate (CFR, the number of deaths per identified cases) are relatively widely reported. To illustrate how confusing a naive approach to using reported numbers to estimate the CFR can be, consider the 7 April numbers for France, Spain, the UK and Italy.
Country | Reported cases | Reported deaths |
---|---|---|
France | 98,010 | 8,911 |
UK | 51,608 | 5,373 |
Spain | 136,675 | 13,341 |
Italy | 132,547 | 16,523 |
Mental arithmetic is sufficient for noting that the reported fatalities amount to close to 10% of reported cases in the UK, France and Spain and more than that in Italy. Consider two more countries among the 10 with the highest total number of fatalities, the USA and Iran.
Country | Reported cases | Reported deaths |
---|---|---|
Iran | 60,500 | 3,739 |
USA | 367,650 | 10,943 |
In both cases, the number of deaths is significantly below 10% of the number of reported cases. Suppose that the criteria for testing are identical in all six countries. In that case, the differences in a naive estimate of the CFR can be accounted for by the hypothesis that France, Spain and the UK are at similar stages in the development of the epidemic whereas Italy is at a later stage (frankly and sadly, more of the infected have had time to pass away). The same hypothesis would also account for the USA having a lower naive CFR estimate since many might still be in early stages of the infection there. It would also account for the low naive CFR in Iran, presuming that the peak of the epidemic has passed there.
Nevertheless, more sophisticated estimates of the CFR reported elsewhere are lower than all of these numbers. A study in The Lancet2 estimates a CFR in China of 1.38% and an IFR of 0.66% which is close to other estimates3. Having a good estimate of the IFR as well as the prevalence would allow informed decision-making involving forecasts of the development or containment of the prevalence. Ongoing random testing would be one way of tracking the development of the disease.
Testing can serve at least three distinct purposes.
Underlying this is a binary classification problem where we have (at any moment in time) two groups of individuals: P, the positives who have the virus; and N, the negatives who do not have it. No test is able to perfectly distinguish these groups and generates, rather,
Naturally \(P + N = TP + FP + TN + FN\) and the mathematically inclined will see that \(P = TP + FN\) and \(N = TN + FP\). This is under the assumption that the test is conclusive. It is convenient to take all the numbers to be percentages.
In reality, only God knows what P and N are but in practice a high-standard test or clinical diagnosis is used to distinguish the two groups and a test is measured against it for a representative (or, available) subset. It is not unusual for this subset to consist of only between 20 and 100 persons which is understandable in view of the novelty of this disease and the practical difficulties of larger trials.
The accuracy of a test is defined as \(\frac{TP+TN}{P+N}\) which is the likelihood that an outcome of the test will be correct. Primary school arithmetic allows one to notice that the numerator \(TP+TN\) is split between TP and TN and making TP a bit lower and TN correspondingly higher, it would be unchanged. For this reason, it makes more sense to also consider the sensitivity \(\frac{TP}{P}\) (probability that a true positive will be correctly classified by the test) and the specificity \(\frac{TN}{N}\) (probability that a true negative will be correctly classified) of a test, individually.
The standard genetic (PCR or molecular) assay used to test for the novel corona virus SARS-CoV-2 has a sensitivity reportedly4 in the range of 45% to 70%. This means that of infected persons, between 30% and 55% will test negative on a first test. The specificity is much higher so the probability of a negative test result for a negative person is high. For this reason, infection can be excluded only after a number of negative tests on the same patient. So, the PCR test is rather ineffective for identifying patients for closer observation or for quarantine on a single test.
The molecular test is expensive however and can take 24 to 48 hours if a sample has to be sent away to a laboratory, which is the case for many testing sites as the equipment is bulky and tests need to be run in batches. Smaller point-of-care devices are however becoming available, some of which have been approved for emergency use in the USA by their FDA but not in many other countries. These allow for more rapid testing of potentially infected patients.
A different approach is to test for anti-bodies to the corona virus. These tests can be done inexpensively and at home, similar to HIV or pregnancy testing. In the first few days of infection, their sensitivity is lower than that of molecular testing and they will also test positive for patients who no longer have the virus but who still have anti-bodies. It is believed however that such patients have a degree of immunity to the disease. It should however be kept in mind that many patients who develop severe corona virus disease do not test positive for anti-bodies at all, possibly as part of the disease progression in those cases5.
At the time of writing, hundreds of tests6 have been developed but the FDA has issued a warning7 which has retarded the bringing to market of anti-body tests. We would argue that at least for the purpose of determining prevalence of the infection (or resistance, indicating previous exposure) the cheap anti-body tests are perfectly adequate. For, a test done for statistical purposes that misses new infections of the past few days is exactly as good as a test done a few days ago.
Studies in China8 have shown that anti-body tests on blood, serum or plasma are significantly better at detecting infections as soon as 7 days after onset of the illness than the molecular tests. The anti-body tests take 15 minutes and require no special equipment. Even in the first 7 days of the illness, the sensitivity reported by these researchers for the molecular test was 66.7% against 38.3% for the immunoassays. They found that during the second week of illness, sensitivity for the molecular test dropped to 54.0% whereas that of the immunoassays increased to 89.6,% so the anti-body tests were significantly better at this stage.
It is true that anti-body testing will provide positive results in patients who have healed. However, the range of the duration of the disease has been studied and it is clear that 5 or 6 weeks after a positive anti-body test, a patient should be free of the virus. This is at most one week different from the result for the molecular test so if the purpose of testing is to identify individuals who have had the disease then anti-body testing is a viable immediate substitute for molecular testing.
In fact, for determining how many people in a population are either currently infected or who have already had the disease, anti-body testing is the only suitable tool – unless molecular testing can be done universally and repeatedly, which is not realistic. Any fixed budget allows for far more immunoassays to be conducted than molecular tests so any deficiency in sensitivity of the former will likely be statistically overwhelmed by the efficacy of doing a larger number of tests.
For the purpose of public health management, the goals of testing should be clearly conceived. Gauging prevalence by screening a large random sample of the population using cheap immunoassays can inform political decisions as well as how to employ other tests. For instance, if prevalence is known to be very low then it might make sense to attempt to isolate individual cases in the population. Should prevalence be very high, it might make sense to isolate those with no exposure or immunity only and to treat patients based on disease symptoms only without further testing.
Stanford's John Ioannidis (professor of epidemiology and population health and expert in biomedical data science) has suggested that the corona virus epidemic could be a "once-in-a-century evidence fiasco" in a March 17 article9. He notes that the most useful information for answering a number of questions would be to know the current prevalence of infection in the population and to regularly measure it by random sampling. Anti-body testing is the best available tool for doing this. As with many things in life, certainty is a luxury which is not only unaffordable but also unavailable. Ioannidis points out that even estimates of the toll taken by the ordinary influenza vary widely with estimates in the current season ranging from 22,000 to 55,000 deaths in the USA.
Both authors concur with Ioannidis that not only is the threat posed by this disease badly understood but so is both the efficacy and cost of measures undertaken to contain it. Random testing using affordably immunoassays provide a cost-effective way of better understanding the threat and has started in Finland10. The state of New York is considering using anti-body tests to identify people who are safe to return to work11. The sooner other countries or regions follow suit, the better.
https://www.worldometers.info/coronavirus/↩
https://www.thelancet.com/journals/laninf/article/PIIS1473-3099(20)30243-7/fulltext↩
https://www.medrxiv.org/content/10.1101/2020.03.05.20031773v2↩
https://www.journalofhospitalinfection.com/article/S0195-6701(20)30100-6/fulltext↩
https://www.ncbi.nlm.nih.gov/pubmed/32198501↩
https://spectrum.ieee.org/the-human-os/biomedical/diagnostics/how-do-coronavirus-tests-work↩
https://www.mobihealthnews.com/news/home-covid-19-testing-services-pump-brakes-after-fda-warns-fraudulent-kits↩
https://www.jwatch.org/na51255/2020/03/31/serologic-tests-sars-cov-2-first-steps-long-road↩
https://www.statnews.com/2020/03/17/a-fiasco-in-the-making-as-the-coronavirus-pandemic-takes-hold-we-are-making-decisions-without-reliable-data/↩
https://www.reuters.com/article/us-health-coronavirus-finland-testing/finland-to-begin-randomised-coronavirus-antibody-testing-idUSKBN21P1KT↩
https://www.sciencemediacentre.org/expert-comment-on-different-types-of-testing-for-covid-19/↩