Calderdale and Huddersfield NHS Trust Case Study

How did red dot® perform atCalderdale and Huddersfield NHS Trust?

3,810

images processed over 74 days

7.5

seconds on average per scan

28.8%

saving in radiologist workload

71%

reduction in waiting times for suspected lung cancer patients

Of the 3,810 images processed during this period, the model classified 119 (3.1%) as Suspected Lung Cancer (SLC), the majority of which were found to contain signs of tumour, and flagged 1,097 cases as High Confidence Normal (HCN) with a negative predictive value (NPV) of 0.94 and a discrepancy rate of 1.7%.

red dot® at Calderdale and Huddersfield NHS Trust featured on ITV News

The report focuses on the success of Artificial Intelligence at CHFT to help speed up reporting of chest x-rays for suspected lung cancer.

"... I find the high confidence normal assessment to be very impressive and already find myself trusting it. That it reliably takes away approximately a quarter of the workload is enough of a selling point in of itself."

Dr Adrian Hood, Consultant Chest and Cardiac Radiologist

Overview

The behold.ai red dot® service has been deployed at Calderdale and Huddersfield NHS Trust (CHT) since March 2023, and has processed 3,810 exams to date (30^th May 2023). The service uses an artificial intelligence model to rule-out normal chest x-ray exams and fast-track the diagnosis of those with suspected lung cancer. The autonomous AI algorithm can classify exams as High Confidence Normal (HCN), suggesting they can be removed from radiology worklist and auto-reported, and flag exams as containing SLC (Suspected Lung Cancer) to be prioritised for reporting and fast-track diagnosis.

The red dot® platform classified 1,097 (28.8%) of the 3,810 exams as HCN. This represents the removal of 28.8% of radiologists’ workload. The model performed with a negative predictive value (NPV) of 0.94 and a false negative rate of 1.7%. One of these HCN discrepancies was initially considered to be urgent by behold.ai radiologists, but was subsequently reported as normal by CHT radiologists. The platform also flagged 119 (3.1%) exams as SLC-positive, indicating that these patients can be prioritised for reporting and fast-track diagnosis.

Good levels of service have been maintained by the red dot® platform throughout this period. Results were returned within the expected timeframes, with a mean service time of 7.5 seconds. Overall 100% of exams had a <30 second delay in returning a result.

Of the 3,810 exams processed, the hospital indicated a disagreement with 32 results. This is a hospital disagreement rate of 0.8%.

This case study demonstrates that the red dot® platform is a valuable clinical tool for radiologists that provides time saving benefits in the care pathway.

CHT Workflow

What were the results?

HCN

Of the 3,810 chest images processed during this period, the red dot® platform classed 1097 (28.8%) as HCN. Removing 28.8% of total CXR cases from radiologists’ workload has allowed the Trust to prioritise resources more effectively.

Of the 1,097 images classed as HCN, there were 66 cases where hospital radiologists disagreed with the model’s classification. These discrepancies were identified during the behold.ai HCN audit process, in which an FRCR-trained radiologist reviews all exams classed as HCN within 24 hours of processing.

One of these discrepancies was initially considered to be urgent by auditing radiologists, at which point the hospital was notified immediately for follow-up. This exam was subsequently reported as normal by CHT radiologists.

The Trust requested further imaging for three discrepancy exams, all of which were ultimately reported as normal. Two were nipple shadows, and the third was suspected normal but required characterising with further rule-out imaging.

HCN Discrepancies: Calderdale

66 discrepancies out of 1097 tagged exams (NPV: 0.94)

Of these 66 HCN discrepancies, the most common factor was bony abnormalities. These accounted for just under a third of all discrepancies, and were typically instances of scoliosis, congenital bony abnormalities or old rib fractures.

The model performed with an NPV of 0.94 and a discrepancy rate of 1.7%, none of which were clinically significant. This real-world performance closely aligns with previously peer-reviewed and published results from behold.ai.

SLC

During this period the red dot® platform flagged 119 (3.1%) CXR exams as SLC, indicating that they contain signs potentially indicative of lung cancer. Of these 119 exams, behold.ai radiologists have audited a 71% subset (83 exams). Of these audited exams, 48 (57.8%) were found to contain signs of tumour including nodules, masses and hilar enlargement. A further 7 exams (8.4%) were found to contain nipple shadows, three of which were unilateral and could plausibly indicate tumours.

The vast majority (95.2%) of cases were found to contain some form of abnormality. The three most common factors in these abnormalities were nodules, masses and hilar enlargement, all three of which can be indicative of lung cancer.

Use of the red dot® platform significantly reduced average pathway times from CXR to CXR report. The Trust saw a 71% reduction in average time taken from conducting CXRs to being reported by the Trust, shrinking from 7 days to 2 days.

These results show that the red dot® platform is a valuable tool for prioritising patients for fast-track diagnosis, and for optimising the lung cancer pathway.

Conclusion

Performance at Calderdale and Huddersfield NHS Trust has demonstrated that the red dot® platform is a valuable clinical tool.

The model effectively rules out 28.8% of total scans, indicating that they can be removed from radiologists’ workload and auto-reported. It did so with a high degree of accuracy; the model performed with an NPV of 0.94 and a discrepancy rate of 1.7%.

This period also demonstrated the effectiveness of red dot® platform for flagging exams as SLC. Of the audited exams that were classed as SLC by the model, 95.2% were determined to be abnormal and the majority (57.8%) were determined to contain signs of tumour. Use of the red dot® platform reduced the average time from CXR to CXR report from 7 days to 2 days.

The red dot® service saves time and resources by easing the burden on radiologists and allowing them to prioritise their workload. It also reduces lung cancer pathway times, enabling patients with suspected lung cancer to be prioritised for reporting and fast-track diagnosis.