red dot® CT Head Day and Night reporting Case Study

How did our red dot® CT Head Day and Night reporting service perform?

390

non-contrast CT head examinations collected from the UK, USA and India

91.3%

of Haemorrhage cases identified that were missed by radiologists

69.1%

of Acute Infarct cases identified that were missed by radiologists

54.3%

ruled-out as Normal and removed from radiologist workload

Read the full case study:

The need

Non-contrast CT head examinations are routinely used as the first test to rule out brain abnormalities that are acute and time-sensitive, such as intracranial haemorrhage (ICH), skull fractures and early-onset stroke. Increases in the incidence of imaging as a result the introduction of NICE guidelines [2] have driven a growing use of and demand for CT head examinations, which has in turn increased workloads and pressures on radiology departments. To cope with these pressures, the NHS is spending £206M annually on outsourcing the reporting of radiological examinations [3].

In patients with head injuries, accurate and fast diagnosis of abnormalities from non-contrast CT head examinations is necessary for delivering timely treatments, and to ensure that patients will have the best chance of fully preserved neurological function [4]. Indeed, when patients with brain injury are able to start treatment earlier, they are more likely to be functionally independent and to have reduced disability 90 days after treatment [5].

With this in mind, and considering that the UK’s annual incidence of stroke is expected to rise by 60% by 2035 due to an ageing population, the National Health Service (NHS) has included timely brain imaging, faster diagnosis and better outcomes for patients with brain injury as key goals in its long-term plan [6].

At the same time, of the approximately 200,000 people admitted to hospital annually with a head injury in the UK, approximately 80% of them have CT head examinations that are reported as Normal, or as non-acute findings [7]. The need for processing these normal examinations on an urgent basis can slow down the evaluation of examinations that require more critical attention.

60%

increase in UK incidences of stroke by 2035

200k

people admitted to hospital annually with head injury

What is red dot® CTH?

In response to clinical challenges faced by radiology departments, red dot® CT head (CTH) has been developed by Behold.ai as a multi-functional solution to rule out, triage and manage non-contrast CT head examinations. This solution is a replacement to the out-of-hour reporting services conducted by teleradiology companies across the NHS, which have been set up to cope with the high volume of radiological examinations that need reporting by the NHS.

The red dot® CTH algorithm can rule out a subset of examinations that do not contain any significant abnormalities as ‘Normal’, and identify two of the most common and critical acute findings in CT head examinations: intracranial haemorrhage (ICH) and acute infarct (i.e. stroke), which can help prioritize patients and speed up treatment.

The red dot® CTH solution is the first implementation of an algorithm that identifies normal examinations in CT head examinations. The dual processing of acute and normal examinations also makes this a unique product among the offering of commercially available solutions that process CT head examinations and can be used to help reduce outsourcing costs. The workflow of the red dot® CT head algorithm and the benefits this tool can provide are presented in the figure below.

Diagram of the CTH pathway using red dot

The algorithm was trained on 2,059 CT head examinations, which were labelled and annotated by FRCR Consultant Radiologists and all examinations were arbitrated for inclusion into the algorithm by FRCR Consultant Neuroradiologists . The CTH solution also generates an image segmentation for each finding, which can aid clinicians in understanding the algorithm’s outputs.

Examples of examinations classified as acute infarct and segmented by the red dot® algorithm can be seen below.

A peer-reviewed validation study of the red dot® CTH algorithm is now published in Neuroradiology, the official journal of the European Society of Neuroradiology [1]. In this study, Behold.ai carried out a retrospective validation of its CT head algorithm, which allowed to analyse the performance of the algorithm for classifying ICH, acute infarct and normal examinations.

What did the study involve?

This study was performed on a dataset that consisted of 390 non-contrast CT head examinations, which were collected from the UK, USA and India. The dataset contained a wide range of clinical findings, which made it a challenging classification set.

As part of the study, images were ground-truthed by 3 FRCR consultant radiologists and 2 FRCR consultant neuroradiologists, who all had a minimum of 15 years’ experience at consultant level. Each CTH exam was independently read by two radiologists, who each assigned pre-defined disease labels to the exam and drew a segmentation map around identified pathologies. A diagram detailing the ground-truth process can be found below.

The performance of the AI solution was compared to 3 independent FRCR consultant radiologists, which represented the existing standard of care. These independent radiologists had 3, 5 and 11 years of experience at consultant level. The performance for triaging acute findings and for ruling out Normal examinations was thus compared between the algorithm and radiologists.

What were the results?

As we describe in more detail below, the algorithm showed excellent classification performance for all findings of interest (i.e., ICH, acute infarct and Normal).

AI triage of acute findings: In the triage of examinations that contained ICH and acute infarct, the red dot® algorithm showed a greater sensitivity than that of radiologists, as can be seen in the table below.

	Sensitivity		Specificity
Disease class	AI algorithm	Radiologists	AI algorithm	Radiologists
ICH	0.988	0.907	0.925	0.991
Acute infarct	0.833	0.618	0.927	0.997

The AI solution showed great potential for augmenting the diagnostic performance of radiologists, as the solution identified 91.3% of Haemorrhage cases missed by radiologists, and 69.1% of Acute Infarct cases missed by radiologists. An example of haemorrhages that were detected by the algorithm but missed by independent radiologists can be seen below.

As indicated in the table above, for the triage of ICH and acute infarct, the specificity of the algorithm was above 90%, but it was lower compared to radiologists. This indicates the need of using AI explainable methods such as the segmentation maps provided by this CTH solution, as these maps can allow radiologists to quickly spot and process false positives generated by AI algorithms.

Overall, the results on the AI triage of acute findings point to a great opportunity provided by the CTH solution for improving diagnostic accuracy and clinical safety in the processing of CT head examinations, and for the potential use of this solution as a clinical decision support system that includes explainable outputs.

AI classification of Normal as a rule‑out test: For the classification of Normal examinations, the red dot® algorithm was able to identify more than half of these examinations (54.3%) with a greater negative predictive value (NPV) than the average radiologist (0.93% NPV for the algorithm compared with 86.8% NPV for radiologists). Moreover, the red dot® algorithm did not miss any acute findings in the classification of Normal exams compared with radiologists, who had a 5.2% miss rate (i.e., abnormal exams incorrectly classified as Normal) that included 17 missed cases of haemorrhage or acute infarct.

This result indicates that the red dot® solution can successfully rule-out and remove a subset of Normal examinations; removing 54.3% of Normal examinations as shown in this study would in fact represent a 27.4% reduction in the total number of CT examinations that require immediate reporting. The rule-out functionality of the red dot® solution can thus play a relevant role in managing workloads for CT head examinations in radiology departments. This can in turn provide benefits in the form of improved clinical performance, as it allows radiologists to better focus their resources and attention in patients that need these resources and reduce costs

Conclusions and next steps

This published study demonstrated that Behold.ai’s AI red dot® CT Head Day and Night reporting solution can effectively detect and triage patients with acute CTH exam findings (i.e., ICH and acute infarct), and that it can rule out Normal examinations with a lower miss rate than expert clinicians and an overall superior performance. Integrating the red dot® CTH solution with existing radiology workflows can enable the prioritization of acutely unwell patients, with an associated improvement of clinical accuracy, and the provision of a safety net for critical findings.

Moreover, through its identification of Normal examinations, the implementation of the CTH solution can help manage radiological resources, deliver immediate cash savings in the reporting of CTH examinations and it can also enable the reduction of outsourcing costs.

As part of additional evidence generation plans, a real-world evaluation of the performance and the clinical impact associated to the implementation of the red dot® CTH algorithm is expected to take place in the next months, along with a further validation of the product on a larger dataset. This will also allow to test the robustness of the product across diverse patient populations in terms of age, sex and geographical location.

References

Dyer, T., Chawda, S., Alkilani, R., Morgan, T. N., Hughes, M., & Rasalingham, S. Validation of an artificial intelligence solution for acute triage and rule-out normal of non-contrast CT head examinations. Neuroradiology, 64(4), 735-743, 2022.
National Institute for Health and Care Excellence (Great Britain). Head injury: assessment and early management. National Institute for Health and Care Excellence (NICE), 2017.
Royal College of Radiologists, “Clinical Radiology UK Workforce Census 2020 Report,” R. Coll. Radiol., no. April, p. 72, 2021, [Online]. Available: https://www.rcr.ac.uk/system/files/publication/field_publication_files/clinical-radiology-uk-workforce-census-2020-report.pdf.
Saver JL. Time is brain – quantified. Stroke 37(1):263–266, 2006.
G. Jovin et al., “Thrombectomy within 8 Hours after Symptom Onset in Ischemic Stroke,” N. Engl. J. Med., vol. 372, no. 24,pp. 2296–2306, 2015.
Kapur N. The NHS long term plan SUSHRUTA. J. Heal. Policy Opin 12(1):10–11, 2020.
I. for H. and C. E. NICE, “Head injury triage, assessment, investigation and early management of head injury in children, young people and adults,” NICE Clin. Guidel., no. January, 2014.