ORCID

Abstract

ObjectivesDeep learning models developed for the classification of radiological reports have lacked explainability. We aimed to validate and explain a pretrained classification model by applying it to the removal of confounding data from a radiological dataset.MethodsTwo radiologists categorized 2038 anonymized MRI head free-text radiology reports for abnormality and for small vessel disease presence. Of these reports, 80% (n = 1630) were used to fine-tune pretrained transformer models to classify scans. Five-fold cross-validation was used in model development. The models were tested on the remaining 20% of the reports (n = 408). SHapley Additive exPlanations (SHAP) were used to explain the results.ResultsThe models exhibited excellent classification performance, with a mean receiver operating characteristic (ROC) area under the curve (AUC) of 0.98 for abnormality classification and 0.99 for small vessel disease classification. SHAP highlighted relevant words in both cases.ConclusionsThis application validated the use of a pretrained transformer in detecting confounding data in research cohorts, and exhibited explainable results that allow the models’ decisions to be understood. By highlighting the specific report terms that drive each prediction, the explainable model output can be reviewed and critiqued by subject matter experts, supporting trust, error analysis, and iterative refinement of AI tools within clinical workflows.Advances in knowledgeThis application demonstrates the feasibility of explainable report classification, and the fine-tuned model could be used in future for automatic removal of confounding data from radiology datasets, while providing transparent, case-level justifications that support audit, governance, and clinician acceptance.

Publication Date

2026-01-16

Publication Title

BJR|Artificial Intelligence

Volume

3

Issue

1

Acceptance Date

2026-01-09

Deposit Date

2026-02-05

Download

Share

COinS