Technology plays an integral role in almost every aspect of our lives and its capabilities are constantly developing. Technological advances in medicine in particular have had a huge impact. There is no doubt that the precision, accuracy, and efficiency of medical tasks has been vastly improved by implementing these technologies, but are there any risks to becoming so heavily reliant on machines and software?

Author: Jo Sharpe

Second year PhD student Ethan Du-Crow, based in the Division of Informatics, Imaging and Data Sciences, is investigating the efficacy of computer-aided detection (CAD) software in assisting the detection of breast cancer in mammography.

What is CAD?

CAD software was designed to assist medical professionals in identifying cancers on an image and has been used routinely in breast cancer screening for years. Signs of breast cancer are variable and subtle and therefore can be occasionally missed. CAD software assists radiologists by highlighting areas on the mammogram image that could be abnormal. However, it is not without its problems.

A paper published in 2012 looked at how effectively can radiologists use computer-aided detection. They found that on average CAD increases sensitivity by 9.3% at the cost of a 12.4% increase in the recall rate. This means that although more cancers are being spotted, CAD is increasing  the frequency of false positives, leading to an increased number of unnecessary breast biopsies, additional tests and of course a great deal of stress and anxiety for the women and their families. This is a consequence of the high number of false prompts and is one of the biggest problems CAD faces. Achieving the appropriate balance between sensitivity and specificity is crucial for a successful detection system.

Ethan’s PhD

The interaction between radiologists and CAD systems is not well understood and it is unclear how different prompting methods may affect the sensitivity and specificity. A previous study found that the use of CAD affected the way the reader searched an image; whilst using CAD the reader’s attention was drawn to the markers, meaning they were less likely to spot something if the software failed to highlight it. This is problematic and suggests that in instances where the software misses a cancer, the reader is distracted and is therefore less likely to identify a cancer compared to when CAD software was not used at all. Ethan is dissecting the complex interactions between software and reader that may mean we overestimate its benefits.

‘Second reader’ study

CAD software for mammography is designed to be used as a ‘second reader’ – the image is first read unaided and then read again with the assistance of CAD. Ethan explained how the aforementioned study failed to take this into account, as it only looked at how CAD markers affect the way images are searched when participants read the image once. He is investigating how second reader CAD affects visual search, with the aim of discerning the interaction between readers and the software when used in a second reader mode.

The ‘safety net’ effect

It has been suggested that the initial viewing of an image may be rushed because the reader knows they have the ‘safety net’ of a CAD marked image to look at afterwards. This has not been observed experimentally but could be problematic because it would mean that readers are not searching the image as thoroughly as they would if CAD was not used. An important question Ethan wants to address is if the initial pre-CAD search is adversely affected by it being preliminary to an additional search with CAD, and therefore different to the unaided search when CAD is not used.

The study

Ethan’s study will assess the complex interplay between the software and how readers search an image. He has recruited non-expert participants to search mammogram- like images for groups of bright spots that could indicate cancer.  The participants will read half of the images without CAD and the other half with CAD in second reader mode. In order to see if there is a ‘safety net’ effect, or if the markers are a distraction, the participants’ eye-movements will be tracked during the experiment. The results are then compared between CAD and no CAD conditions.

Findings so far

To date, data have been collected from 50 participants. Ethan has found that observer sensitivity significantly increased in the CAD condition compared to the no-CAD condition. For targets unmarked by CAD, observer sensitivity was 5% lower in the CAD condition compared to the same targets in the no-CAD condition and eye-tracking data showed that coverage of the image before CAD was significantly lower than when CAD was not used at all. This confirms the idea that the initial search may be influenced by the subsequent availability of CAD.

Conclusions and implications

Ethan will finish data collection with a total of 52 participants by 7th November 2018 and from his analysis so far, it is clear that the results will have implications for the use of CAD and studies of its efficacy. Previous work that has shown significant improvement in sensitivity post-CAD compared to pre-CAD, but Ethan’s results suggest that they are overestimating the benefits of CAD because they fail to consider the safety net effect. Future CAD efficacy studies should account for this effect when estimating benefit. Following completion of this study, he will be working with expert medical readers and investigating whether the same effects that we observed with non-experts also occur with them.

He is presenting this work at the SPIE Medical Imaging in San Diego in February 2019, and is pleased with his progress so far:

“We have some very interesting results that has set up the rest of the project nicely. The intial study leads on to further work in which we will be creating a realistic clinical setup with dual-screen eye-tracking using actual mammograms and commercial CAD systems. Ultimately the goal is to find a way of improving reader sensitivity by using CAD with minimal increase in the number of false positive results.”

He goes on to say that there is potential to extend the work to other imaging modalities such as digital breast tomosynthesis (DBT) and automated breast ultrasound (ABUS), which literally adds a new dimension to the search task!