Clinical validation of an artificial intelligence algorithm for classifying tuberculosis and pulmonary findings in chest radiographs
BackgroundChest X-ray (CXR) interpretation is critical in diagnosing various lung diseases. However, physicians, not specialists, are often the first ones to read them, frequently facing challenges in accurate interpretation. Artificial Intelligence (AI) algorithms could be of great help, but using...
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2025-02-01
|
| Series: | Frontiers in Artificial Intelligence |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/frai.2025.1512910/full |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1825206252517457920 |
|---|---|
| author | Thiago Fellipe Ortiz de Camargo Thiago Fellipe Ortiz de Camargo Guilherme Alberto Sousa Ribeiro Guilherme Alberto Sousa Ribeiro Maria Carolina Bueno da Silva Luan Oliveira da Silva Pedro Paulo Teixeira e Silva Torres Denise do Socorro Rodrigues da Silva Mayler Olombrada Nunes de Santos William Salibe Filho Marcela Emer Egypto Rosa Magdala de Araujo Novaes Thiago Augusto Massarutto Osvaldo Landi Junior Elaine Yanata Marcio Rodrigues da Cunha Reis Gilberto Szarf Pedro Vieira Santana Netto Joselisa Peres Queiroz de Paiva |
| author_facet | Thiago Fellipe Ortiz de Camargo Thiago Fellipe Ortiz de Camargo Guilherme Alberto Sousa Ribeiro Guilherme Alberto Sousa Ribeiro Maria Carolina Bueno da Silva Luan Oliveira da Silva Pedro Paulo Teixeira e Silva Torres Denise do Socorro Rodrigues da Silva Mayler Olombrada Nunes de Santos William Salibe Filho Marcela Emer Egypto Rosa Magdala de Araujo Novaes Thiago Augusto Massarutto Osvaldo Landi Junior Elaine Yanata Marcio Rodrigues da Cunha Reis Gilberto Szarf Pedro Vieira Santana Netto Joselisa Peres Queiroz de Paiva |
| author_sort | Thiago Fellipe Ortiz de Camargo |
| collection | DOAJ |
| description | BackgroundChest X-ray (CXR) interpretation is critical in diagnosing various lung diseases. However, physicians, not specialists, are often the first ones to read them, frequently facing challenges in accurate interpretation. Artificial Intelligence (AI) algorithms could be of great help, but using real-world data is crucial to ensure their effectiveness in diverse healthcare settings. This study evaluates a deep learning algorithm designed for CXR interpretation, focusing on its utility for non-specialists in thoracic radiology physicians.PurposeTo assess the performance of a Convolutional Neural Networks (CNNs)-based AI algorithm in interpreting CXRs and compare it with a team of physicians, including thoracic radiologists, who served as the gold-standard.MethodsA retrospective study from January 2021 to July 2023 evaluated an algorithm with three independent models for Lung Abnormality, Radiological Findings, and Tuberculosis. The algorithm's performance was measured using accuracy, sensitivity, and specificity. Two groups of physicians validated the model: one with varying specialties and experience levels in interpreting chest radiographs (Group A) and another of board-certified thoracic radiologists (Group B). The study also assessed the agreement between the two groups on the algorithm's heatmap and its influence on their decisions.ResultsIn the internal validation, the Lung Abnormality and Tuberculosis models achieved an AUC of 0.94, while the Radiological Findings model yielded a mean AUC of 0.84. During the external validation, utilizing the ground truth generated by board-certified thoracic radiologists, the algorithm achieved better sensitivity in 6 out of 11 classes than physicians with varying experience levels. Furthermore, Group A physicians demonstrated higher agreement with the algorithm in identifying markings in specific lung regions than Group B (37.56% Group A vs. 21.75% Group B). Additionally, physicians declared that the algorithm did not influence their decisions in 93% of the cases.ConclusionThis retrospective clinical validation study assesses an AI algorithm's effectiveness in interpreting Chest X-rays (CXR). The results show the algorithm's performance is comparable to Group A physicians, using gold-standard analysis (Group B) as the reference. Notably, both Groups reported minimal influence of the algorithm on their decisions in most cases. |
| format | Article |
| id | doaj-art-9c8ab802833042b2bec07f3e9d61ea05 |
| institution | Kabale University |
| issn | 2624-8212 |
| language | English |
| publishDate | 2025-02-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Artificial Intelligence |
| spelling | doaj-art-9c8ab802833042b2bec07f3e9d61ea052025-02-07T11:13:11ZengFrontiers Media S.A.Frontiers in Artificial Intelligence2624-82122025-02-01810.3389/frai.2025.15129101512910Clinical validation of an artificial intelligence algorithm for classifying tuberculosis and pulmonary findings in chest radiographsThiago Fellipe Ortiz de Camargo0Thiago Fellipe Ortiz de Camargo1Guilherme Alberto Sousa Ribeiro2Guilherme Alberto Sousa Ribeiro3Maria Carolina Bueno da Silva4Luan Oliveira da Silva5Pedro Paulo Teixeira e Silva Torres6Denise do Socorro Rodrigues da Silva7Mayler Olombrada Nunes de Santos8William Salibe Filho9Marcela Emer Egypto Rosa10Magdala de Araujo Novaes11Thiago Augusto Massarutto12Osvaldo Landi Junior13Elaine Yanata14Marcio Rodrigues da Cunha Reis15Gilberto Szarf16Pedro Vieira Santana Netto17Joselisa Peres Queiroz de Paiva18Image Research Center, Hospital Israelita Albert Einstein, São Paulo, BrazilElectrical, Mechanical and Computer Engineering School, Federal University of Goias, Goias, BrazilImage Research Center, Hospital Israelita Albert Einstein, São Paulo, BrazilElectrical, Mechanical and Computer Engineering School, Federal University of Goias, Goias, BrazilImage Research Center, Hospital Israelita Albert Einstein, São Paulo, BrazilImage Research Center, Hospital Israelita Albert Einstein, São Paulo, BrazilImage Research Center, Hospital Israelita Albert Einstein, São Paulo, BrazilInfectology Division, Clemente Ferreira Institute, São Paulo, BrazilAparecida of Goiania Municipal Hospital, Hospital Israelita Albert Einstein, Goias, BrazilPulmonary Division, Heart Institute, São Paulo, BrazilImage Research Center, Hospital Israelita Albert Einstein, São Paulo, BrazilMedical Sciences Center, Federal University of Pernambuco, Pernambuco, BrazilDiagnostic Imaging Research and Study Institute Foundation, São Paulo, BrazilDiagnostic Imaging Research and Study Institute Foundation, São Paulo, BrazilImage Research Center, Hospital Israelita Albert Einstein, São Paulo, BrazilStudies and Researches in Science and Technology Group, Federal Institute of Goias, Goias, BrazilImage Research Center, Hospital Israelita Albert Einstein, São Paulo, BrazilImage Research Center, Hospital Israelita Albert Einstein, São Paulo, BrazilImage Research Center, Hospital Israelita Albert Einstein, São Paulo, BrazilBackgroundChest X-ray (CXR) interpretation is critical in diagnosing various lung diseases. However, physicians, not specialists, are often the first ones to read them, frequently facing challenges in accurate interpretation. Artificial Intelligence (AI) algorithms could be of great help, but using real-world data is crucial to ensure their effectiveness in diverse healthcare settings. This study evaluates a deep learning algorithm designed for CXR interpretation, focusing on its utility for non-specialists in thoracic radiology physicians.PurposeTo assess the performance of a Convolutional Neural Networks (CNNs)-based AI algorithm in interpreting CXRs and compare it with a team of physicians, including thoracic radiologists, who served as the gold-standard.MethodsA retrospective study from January 2021 to July 2023 evaluated an algorithm with three independent models for Lung Abnormality, Radiological Findings, and Tuberculosis. The algorithm's performance was measured using accuracy, sensitivity, and specificity. Two groups of physicians validated the model: one with varying specialties and experience levels in interpreting chest radiographs (Group A) and another of board-certified thoracic radiologists (Group B). The study also assessed the agreement between the two groups on the algorithm's heatmap and its influence on their decisions.ResultsIn the internal validation, the Lung Abnormality and Tuberculosis models achieved an AUC of 0.94, while the Radiological Findings model yielded a mean AUC of 0.84. During the external validation, utilizing the ground truth generated by board-certified thoracic radiologists, the algorithm achieved better sensitivity in 6 out of 11 classes than physicians with varying experience levels. Furthermore, Group A physicians demonstrated higher agreement with the algorithm in identifying markings in specific lung regions than Group B (37.56% Group A vs. 21.75% Group B). Additionally, physicians declared that the algorithm did not influence their decisions in 93% of the cases.ConclusionThis retrospective clinical validation study assesses an AI algorithm's effectiveness in interpreting Chest X-rays (CXR). The results show the algorithm's performance is comparable to Group A physicians, using gold-standard analysis (Group B) as the reference. Notably, both Groups reported minimal influence of the algorithm on their decisions in most cases.https://www.frontiersin.org/articles/10.3389/frai.2025.1512910/fullchest X-raysartificial intelligencedeep learningclinical validationconvolutional neural network |
| spellingShingle | Thiago Fellipe Ortiz de Camargo Thiago Fellipe Ortiz de Camargo Guilherme Alberto Sousa Ribeiro Guilherme Alberto Sousa Ribeiro Maria Carolina Bueno da Silva Luan Oliveira da Silva Pedro Paulo Teixeira e Silva Torres Denise do Socorro Rodrigues da Silva Mayler Olombrada Nunes de Santos William Salibe Filho Marcela Emer Egypto Rosa Magdala de Araujo Novaes Thiago Augusto Massarutto Osvaldo Landi Junior Elaine Yanata Marcio Rodrigues da Cunha Reis Gilberto Szarf Pedro Vieira Santana Netto Joselisa Peres Queiroz de Paiva Clinical validation of an artificial intelligence algorithm for classifying tuberculosis and pulmonary findings in chest radiographs Frontiers in Artificial Intelligence chest X-rays artificial intelligence deep learning clinical validation convolutional neural network |
| title | Clinical validation of an artificial intelligence algorithm for classifying tuberculosis and pulmonary findings in chest radiographs |
| title_full | Clinical validation of an artificial intelligence algorithm for classifying tuberculosis and pulmonary findings in chest radiographs |
| title_fullStr | Clinical validation of an artificial intelligence algorithm for classifying tuberculosis and pulmonary findings in chest radiographs |
| title_full_unstemmed | Clinical validation of an artificial intelligence algorithm for classifying tuberculosis and pulmonary findings in chest radiographs |
| title_short | Clinical validation of an artificial intelligence algorithm for classifying tuberculosis and pulmonary findings in chest radiographs |
| title_sort | clinical validation of an artificial intelligence algorithm for classifying tuberculosis and pulmonary findings in chest radiographs |
| topic | chest X-rays artificial intelligence deep learning clinical validation convolutional neural network |
| url | https://www.frontiersin.org/articles/10.3389/frai.2025.1512910/full |
| work_keys_str_mv | AT thiagofellipeortizdecamargo clinicalvalidationofanartificialintelligencealgorithmforclassifyingtuberculosisandpulmonaryfindingsinchestradiographs AT thiagofellipeortizdecamargo clinicalvalidationofanartificialintelligencealgorithmforclassifyingtuberculosisandpulmonaryfindingsinchestradiographs AT guilhermealbertosousaribeiro clinicalvalidationofanartificialintelligencealgorithmforclassifyingtuberculosisandpulmonaryfindingsinchestradiographs AT guilhermealbertosousaribeiro clinicalvalidationofanartificialintelligencealgorithmforclassifyingtuberculosisandpulmonaryfindingsinchestradiographs AT mariacarolinabuenodasilva clinicalvalidationofanartificialintelligencealgorithmforclassifyingtuberculosisandpulmonaryfindingsinchestradiographs AT luanoliveiradasilva clinicalvalidationofanartificialintelligencealgorithmforclassifyingtuberculosisandpulmonaryfindingsinchestradiographs AT pedropauloteixeiraesilvatorres clinicalvalidationofanartificialintelligencealgorithmforclassifyingtuberculosisandpulmonaryfindingsinchestradiographs AT denisedosocorrorodriguesdasilva clinicalvalidationofanartificialintelligencealgorithmforclassifyingtuberculosisandpulmonaryfindingsinchestradiographs AT maylerolombradanunesdesantos clinicalvalidationofanartificialintelligencealgorithmforclassifyingtuberculosisandpulmonaryfindingsinchestradiographs AT williamsalibefilho clinicalvalidationofanartificialintelligencealgorithmforclassifyingtuberculosisandpulmonaryfindingsinchestradiographs AT marcelaemeregyptorosa clinicalvalidationofanartificialintelligencealgorithmforclassifyingtuberculosisandpulmonaryfindingsinchestradiographs AT magdaladearaujonovaes clinicalvalidationofanartificialintelligencealgorithmforclassifyingtuberculosisandpulmonaryfindingsinchestradiographs AT thiagoaugustomassarutto clinicalvalidationofanartificialintelligencealgorithmforclassifyingtuberculosisandpulmonaryfindingsinchestradiographs AT osvaldolandijunior clinicalvalidationofanartificialintelligencealgorithmforclassifyingtuberculosisandpulmonaryfindingsinchestradiographs AT elaineyanata clinicalvalidationofanartificialintelligencealgorithmforclassifyingtuberculosisandpulmonaryfindingsinchestradiographs AT marciorodriguesdacunhareis clinicalvalidationofanartificialintelligencealgorithmforclassifyingtuberculosisandpulmonaryfindingsinchestradiographs AT gilbertoszarf clinicalvalidationofanartificialintelligencealgorithmforclassifyingtuberculosisandpulmonaryfindingsinchestradiographs AT pedrovieirasantananetto clinicalvalidationofanartificialintelligencealgorithmforclassifyingtuberculosisandpulmonaryfindingsinchestradiographs AT joselisaperesqueirozdepaiva clinicalvalidationofanartificialintelligencealgorithmforclassifyingtuberculosisandpulmonaryfindingsinchestradiographs |