How did red dot® perform?
82,093
CXR exams were analysed over 19 months
AUC0.91
in the classification of Abnormal exams
AUC0.92
in the classification of CXRs with visible signs of COVID-19
Read the full case study:
Imaging and Covid-19
The impact of the Covid-19 pandemic was felt worldwide and early diagnosis was one of the main ways to keep infection rates under control. The core method for Covid-19 diagnosis is using real-time reverse transcriptase-polymerase chain reaction (RT-PCR), however at the height of the pandemic there was a shortage of PCR tests and delays in turn-around-times for reports [1]. Chest radiographs (CXR) are quick and inexpensive, and became an alternative method of COVID-19 screening for suspected patients [2]. This resulted in increasing pressures of imaging departments.
Covid-19 pandemic in India
Given the socioeconomic disparities and scarce healthcare resources, aggressive measures to combat the spread of Covid-19 such as personal protection, widespread contact tracing, isolation and mass movement restrictions were challenging for low-middle-income countries (LMICs) to adopt [3]. As a densely populated country with roughly 1.4 billion people, Covid infection rates in India quickly rose to a stark 850,000 by July 2020 [4].
Rising cases unsurprisingly swamped clinical services, including imaging departments where CXRs became the first-line imaging modality for Covid-19 screening. Accurate diagnosis of CXRs requires radiological expertise and chronological reporting becomes a tedious task when patient numbers become very large. This eluded the potential for AI solutions that can act as diagnostic assistances to be implemented and alleviate the burden on clinicians.
The AI solution
Behold’s red dot® V1 platform is a Class I CE marked software as a medical device (SaaMD). Red dot® processes adult ≥18 years frontal Chest X-Ray (CXR) examinations to help radiologists prioritise which chest plain film should be reviewed more urgently.
Partnership and pilot programme with Apollo Hospitals, India
In March 2020, Behold.ai announced a partnership with Apollo Radiology International, part of the Apollo Hospitals Group (a network of 71 hospitals in India) and a leading provider of interpretative and other clinical radiological services [5]. The Behold.ai red dot® service was implemented in a pilot programme for the quick triage of patients with suspected COVID-19. The goal of this programme was to speed up the diagnosis of this disease and allow a better allocation of resources, thus supporting India’s response to the pandemic.
This programme was supported by the Amazon Web Services (AWS) Diagnostic Development Initiative – the aim of this Initiative is to accelerate research and innovation on COVID-19 and infectious diseases [6]. Through the support of projects on early disease detection, diagnostics, prognosis and public health genomics, the AWS Diagnostic Development Initiative seeks to achieve a better understanding of this and other infectious diseases, which can lead to effective mitigation and prevention strategies.
Performance of red dot® during the COVID-19 pandemic
The ’instant triage’ of CXRs provided by Behold’s AI-based solution allowed to classify CXR exams as either Normal (meaning no visible signs of COVID-19 or other abnormalities) or Abnormal. In a period of 19 months (from May 2020 to December 2021), 82,093 CXR exams were analysed by red dot®.
An overview of the algorithm performance can be found below.
To evaluate algorithm performance, a subset of 1,289 CXRs were labelled by an FRCR Consultant Radiologist and compared with the classification generated by the algorithm. In the classification of Abnormal exams, red dot® showed an AUC of 0.91. In the classification of CXRs with visible signs of COVID-19 (based on the presence of pulmonary consolidation), red dot® showed an AUC of 0.92.
Below shows a CXR classified as Abnormal by the red dot® algorithm, along with a heatmap generated by the algorithm which shows the location of suspicious areas.
Below shows another CXR classified by the red dot® algorithm as Normal, which was subsequently confirmed by the FRCR Consultant Radiologist.
Conclusion
The work carried out by Behold.ai in partnership with Apollo Hospitals and with the support of the AWS Diagnostic Development Initiative showed that the red dot® platform can effectively support the triage and diagnosis of COVID-19. The work of Behold.ai contributed to tackling a health crisis, akin to the work the company is doing to tackle excess deaths from lung cancer, which have been exacerbated by the COVID-19 pandemic. The work also showed that the performance of the red dot® algorithm generalises well to populations outside of the NHS.
Future collaborations and partnerships between AI companies such as Behold.ai, hospital sites and technology initiatives will continue to pave the way for an accelerated diagnosis of diseases, including infectious diseases such as COVID-19.
References
- Worldwide Shortage Of COVID-19 Test Agents Plagues Health Systems – Even As Infections Surpass 200,000, 2020. Available in https://healthpolicy-watch.news/worldwide-shortage-of-covid-19-test-agents-plagues-health-systems-even-as-infections-surpass-200000/.
- Yang W, Sirajuddin A, Zhang X, Liu G, Teng Z, Zhao S. et al. The role of imaging in 2019 novel coronavirus pneumonia (COVID-19). Eur Radiol 2020; 30: 4874-82.
- Hopman J, Allegranzi B, Mehtar S. Managing COVID-19 in low- and middle-income countries. JAMA 2020;323:1549–50.
- Ghosh A, Nundy S, Mallick TK. How India is dealing with COVID-19 pandemic. Sens Int. 2020;1:100021.
- Behold.ai partners with Apollo Hospitals Group, India for rapid COVID-19 diagnosis using AI-based chest X-ray technology, 2020. Available in https://behold.ai/behold-ai-partners-with-apollo-hospitals-group-india-for-rapid-covid-19-diagnosis-using-ai-based-chest-x-ray-technology/.
- AWS Diagnostic Development Initiative (DDI). Available in https://aws.amazon.com/government-education/nonprofits/disaster-response/diagnostic-dev-initiative/.
Interested in talking to us about AI?
Get in touch to learn more about how red dot® can make a different to your patients and your organisation.