Machine Learning for Predicting Diagnostic Test Discordance in Malaria Surveillance: A Gradient Boosting Approach with SHAP Interpretation


Authors : May Stow

Volume/Issue : Volume 11 - 2026, Issue 1 - January


Google Scholar : https://tinyurl.com/mpnksepw

Scribd : https://tinyurl.com/4cdmzfbm

DOI : https://doi.org/10.38124/ijisrt/26jan131

Note : A published paper may take 4-5 working days from the publication date to appear in PlumX Metrics, Semantic Scholar, and ResearchGate.


Abstract : Malaria remains a critical public health challenge in Nigeria, where accurate diagnosis is essential for effective disease management and resource allocation. Discordance between rapid diagnostic tests (RDTs) and microscopy poses significant challenges for malaria surveillance programs, potentially leading to misdiagnosis and inappropriate treatment decisions. This study aimed to develop a machine learning model for predicting diagnostic test discordance between RDT and microscopy in malaria surveillance data from Bayelsa State, Nigeria. A dataset comprising 2,100 monthly observations from eight Local Government Areas spanning January 2019 to December 2024 was analyzed. The methodology incorporated Bland Altman agreement analysis, feature engineering with climate and health system variables, and gradient boosting classification with class weight balancing to address data imbalance. Model interpretation was achieved through SHapley Additive exPlanations (SHAP) analysis. The Bland Altman analysis revealed a mean difference of negative 2.33 percentage points between RDT and microscopy, with limits of agreement spanning negative 19.28 to positive 14.62 percentage points. The LightGBM classifier achieved an area under the receiver operating characteristic curve of 0.901, with precision of 0.67, recall of 0.74, and F1 score of 0.703. SHAP analysis identified rainfall, climate index, geographic location, and humidity as the most influential predictors of diagnostic discordance. This study contributes an interpretable machine learning framework for identifying conditions associated with diagnostic disagreement, potentially informing quality assurance protocols and targeted interventions in malaria endemic regions.

Keywords : Malaria Diagnosis, Diagnostic Discordance, Machine Learning, SHAP, Rapid Diagnostic Test, Microscopy.

References :

  1. World Health Organization, "World malaria report 2023," Geneva, Switzerland, 2023. [Online]. Available: https://www.who.int/publications/i/item/9789240086173
  2. R. W. Snow, "Global malaria eradication and the importance of Plasmodium falciparum epidemiology in Africa," BMC Med., vol. 13, p. 23, 2015, doi: 10.1186/s12916-014-0254-7.
  3. A. M. Noor, D. K. Kinyoki, C. W. Mundia, C. W. Kabaria, J. W. Mutua, V. A. Alegana, I. S. Fall, and R. W. Snow, "The changing risk of Plasmodium falciparum malaria infection in Africa: 2000–10," Lancet, vol. 383, no. 9930, pp. 1739–1747, 2014, doi: 10.1016/S0140-6736(13)62566-0.
  4. J. L. Gallup and J. D. Sachs, "The economic burden of malaria," Am. J. Trop. Med. Hyg., vol. 64, no. 1–2, pp. 85–96, 2001, doi: 10.4269/ajtmh.2001.64.85.
  5. World Health Organization, "Guidelines for malaria," Geneva, Switzerland, 2023. [Online]. Available: https://www.who.int/publications/i/item/guidelines-for-malaria
  6. I. Bates, V. Bekoe, and A. Asamoa-Adu, "Improving the accuracy of malaria-related laboratory tests in Ghana," Malar. J., vol. 3, p. 38, 2004, doi: 10.1186/1475-2875-3-38.
  7. C. K. Murray, R. A. Gasser, A. J. Magill, and R. S. Miller, "Update on rapid diagnostic testing for malaria," Clin. Microbiol. Rev., vol. 21, no. 1, pp. 97–110, 2008, doi: 10.1128/CMR.00035-07.
  8. D. Bell, C. Wongsrichanalai, and J. W. Barnwell, "Ensuring quality and access for malaria diagnosis: How can it be achieved?," Nat. Rev. Microbiol., vol. 4, pp. S7–S20, 2006, doi: 10.1038/nrmicro1525.
  9. A. Endeshaw, T. Gebre, J. Ngondi, P. M. Graves, E. B. Shargie, Y. Ejigsemahu, B. Ayele, G. Yohannes, T. Ber, A. Byber, A. Lemon, and F. O. Richards, "Evaluation of light microscopy and rapid diagnostic test for the detection of malaria under operational field conditions," Malar. J., vol. 7, p. 118, 2008, doi: 10.1186/1475-2875-7-118.
  10. A. F. Fagbamigbe, "On the discriminatory and predictive accuracy of the RDT against the microscopy in the diagnosis of malaria among under-five children in Nigeria," Malar. J., vol. 18, no. 1, p. 46, 2019, doi: 10.1186/s12936-019-2678-1.
  11. O. O. Oladosu and W. A. Oyibo, "Performance evaluation of a popular malaria RDT in Nigeria compared with microscopy," J. Parasitol. Res., vol. 2020, p. 3650848, 2020, doi: 10.1155/2020/3650848.
  12. C. I. R. Chandler, C. Jones, G. Boniface, K. Juma, H. Reyburn, and C. J. M. Whitty, "Guidelines and mindlines: Why do clinical staff over-diagnose malaria in Tanzania?," Malar. J., vol. 7, p. 53, 2008, doi: 10.1186/1475-2875-7-53.
  13. D. J. Kyabayinze, C. Asiimwe, D. Nakanjako, J. Naber, H. Counihan, J. K. Tibenderana, and S. G. Staedke, "Use of RDTs to improve malaria diagnosis and fever case management at primary health care facilities in Uganda," Malar. J., vol. 9, p. 200, 2010, doi: 10.1186/1475-2875-9-200.
  14. E. Mbunge, S. G. Fashoto, J. Odun-Ayo, and C. Metfula, "Application of machine learning and deep learning for malaria diagnosis: A systematic literature review," J. King Saud Univ. Comput. Inf. Sci., vol. 35, p. 101680, 2023, doi: 10.1016/j.jksuci.2023.101680.
  15. O. Nkiruka, R. Prasad, and O. Clement, "Prediction of malaria incidence using climate variability and machine learning," Inform. Med. Unlocked, vol. 22, p. 100508, 2021, doi: 10.1016/j.imu.2020.100508.
  16. P. Martineau, L. Cardenas, D. Kappel, L. Lorenz, V. L. R. G. Fiaccone, K. V. Braga, and P. G. S. Florentino, "Prediction of malaria using deep learning models: A case study on city clusters in the state of Amazonas, Brazil," Sci. Rep., vol. 12, p. 12437, 2022, doi: 10.1038/s41598-022-16455-3.
  17. M. Stow and E. C. M. Obasi, "An interpretable early warning system for malaria outbreaks in Bayelsa State using deep learning and climate data," Int. J. Adv. Res. Comput. Commun. Eng., vol. 14, no. 8, pp. 38–46, 2025, doi: 10.17148/IJARCCE.2025.14806.
  18. A. Adadi and M. Berrada, "Peeking inside the black-box: A survey on explainable artificial intelligence (XAI)," IEEE Access, vol. 6, pp. 52138–52160, 2018, doi: 10.1109/ACCESS.2018.2870052.
  19. S. M. Lundberg and S. I. Lee, "A unified approach to interpreting model predictions," in Proc. Adv. Neural Inf. Process. Syst., vol. 30, pp. 4765–4774, 2017.
  20. M. Stow and A. A. Stewart, "Empirical analysis of SHAP stability under data corruption across datasets and model architectures," Int. Adv. Res. J. Sci. Eng. Technol., vol. 12, no. 8, pp. 92–110, 2025, doi: 10.17148/IARJSET.2025.12810.
  21. M. Stow and A. A. Stewart, "Interpreting machine learning predictions with SHAP and LIME for transparent decision making," Int. J. Comput. Sci. Math. Theory, vol. 11, no. 8, pp. 22–49, 2025, doi: 10.56201/ijcsmt.vol.11.no8.2025.pg22.49.
  22. Federal Ministry of Health Nigeria, "National malaria strategic plan 2021–2025," Abuja, Nigeria, 2020.
  23. A. Moody, "Rapid diagnostic tests for malaria parasites," Clin. Microbiol. Rev., vol. 15, no. 1, pp. 66–78, 2002, doi: 10.1128/CMR.15.1.66-78.2002.
  24. P. L. Alonso, G. Brown, M. Arevalo-Herrera, F. Binka, C. Chitnis, F. Collins, O. K. Doumbo, B. Greenwood, B. F. Hall, M. M. Levine, K. Mendis, R. D. Newman, C. V. Plowe, M. H. Rodriguez, R. Sinden, L. Slutsker, and M. Tanner, "A research agenda to underpin malaria eradication," PLoS Med., vol. 8, no. 1, p. e1000406, 2011, doi: 10.1371/journal.pmed.1000406.
  25. K. Wafula, C. J. Mwangi, G. S. Amwayi, and C. M. Mureithi, "Quality of malaria microscopy diagnosis by laboratory personnel in a clinical setting," Pan Afr. Med. J., vol. 32, p. 51, 2019, doi: 10.11604/pamj.2019.32.51.17576.
  26. O. A. Mokuolu, M. T. Ajayi, M. L. Ntadom, C. N. Ezeiru, O. T. Bakare, H. O. Musa, C. O. Falade, and S. K. Ojurongbe, "Malaria rapid diagnostic tests and malaria microscopy for guiding malaria treatment of uncomplicated fevers in Nigeria," Clin. Infect. Dis., vol. 63, pp. S290–S297, 2016, doi: 10.1093/cid/ciw631.
  27. O. B. Awosolu, Z. S. Yahaya, and M. T. Farah Haziqah, "Performance evaluation of nested PCR, light microscopy, and PfHRP2 RDT in the detection of falciparum malaria in Nigeria," Pathogens, vol. 11, no. 11, p. 1312, 2022, doi: 10.3390/pathogens11111312.
  28. O. T. Oyeyemi, A. F. Ogunlade, and I. O. Oyewole, "Comparative assessment of microscopy and rapid diagnostic test for malaria diagnosis in southwestern Nigeria," J. Parasitol. Vector Biol., vol. 7, no. 2, pp. 34–41, 2015.
  29. O. I. Ita, A. E. Udo, N. E. Usang, N. O. Adegunloye, E. I. Archibong, and J. U. Akpan, "A diagnostic performance evaluation of rapid diagnostic tests and microscopy for malaria diagnosis using nested PCR as reference standard," Niger. J. Clin. Pract., vol. 23, no. 3, pp. 355–361, 2020, doi: 10.4103/njcp.njcp_539_19.
  30. J. Zhu, K. Wang, S. Li, and Z. Chen, "Stacking ensemble method for malaria incidence prediction," PLoS ONE, vol. 16, no. 7, p. e0253545, 2021, doi: 10.1371/journal.pone.0253545.
  31. T. A. Ojurongbe, O. A. Mokuolu, O. Oyelami, S. S. Okekunle, O. O. Abioye-Kuteyi, T. A. Adeyemo, and O. A. Ojurongbe, "Prediction of malaria positivity using patients' demographic and environmental features," Malar. J., vol. 22, p. 372, 2023, doi: 10.1186/s12936-023-04805-x.
  32. P. U. Eze, C. I. Okagbue, E. A. Ogbonnia, and I. N. Okafor, "Application of machine learning models in predicting malaria prevalence in Nigeria," J. Parasit. Dis., vol. 48, pp. 1–12, 2024, doi: 10.1007/s12639-025-01880-6.
  33. M. Stow, "Explainable machine learning framework for income prediction with class imbalance optimization," Int. J. Adv. Res. Comput. Commun. Eng., vol. 14, no. 8, Art. no. 14801, 2025, doi: 10.17148/IJARCCE.2025.14801.
  34. S. M. Lundberg, G. Erion, H. Chen, A. DeGrave, J. M. Prutkin, B. Nair, R. Katz, J. Himmelfarb, N. Bansal, and S. I. Lee, "From local explanations to global understanding with explainable AI for trees," Nat. Mach. Intell., vol. 2, pp. 56–67, 2020, doi: 10.1038/s42256-019-0138-9.
  35. D. K. Muriithi, G. O. Odhiambo, and F. M. Mwangi, "Explainable artificial intelligence models for predicting malaria risk in Kenya," Int. J. Environ. Res. Public Health, vol. 20, no. 13, p. 6257, 2023, doi: 10.3390/ijerph20136257.
  36. M. Stow, "When data augmentation hurts: A systematic evaluation of SMOTE-based techniques on medical datasets," Int. J. Adv. Res. Comput. Sci., vol. 16, no. 4, pp. 14–33, 2025, doi: 10.26483/ijarcs.v16i4.7313.
  37. B. Shickel, P. J. Tighe, A. Bihorac, and P. Rashidi, "Deep EHR: A survey of recent advances in deep learning techniques for electronic health record analysis," IEEE J. Biomed. Health Inform., vol. 22, no. 5, pp. 1589–1604, 2018, doi: 10.1109/JBHI.2017.2767063.
  38. J. M. Bland and D. G. Altman, "Statistical methods for assessing agreement between two methods of clinical measurement," Lancet, vol. 327, no. 8476, pp. 307–310, 1986, doi: 10.1016/S0140-6736(86)90837-8.
  39. L. I. Lin, "A concordance correlation coefficient to evaluate reproducibility," Biometrics, vol. 45, no. 1, pp. 255–268, 1989, doi: 10.2307/2532051.
  40. G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T. Y. Liu, "LightGBM: A highly efficient gradient boosting decision tree," in Proc. Adv. Neural Inf. Process. Syst., vol. 30, pp. 3146–3154, 2017.

41. M. C. Thomson, S. J. Mason, T. Phindela, and S. J. Connor, "Use of rainfall and sea surface temperature monitoring for malaria early warning in Botswana," Am. J. Trop. Med. Hyg., vol. 73, no. 1, pp. 214–221, 2005, doi: 10.4269/ajtmh.2005.73.214.

Malaria remains a critical public health challenge in Nigeria, where accurate diagnosis is essential for effective disease management and resource allocation. Discordance between rapid diagnostic tests (RDTs) and microscopy poses significant challenges for malaria surveillance programs, potentially leading to misdiagnosis and inappropriate treatment decisions. This study aimed to develop a machine learning model for predicting diagnostic test discordance between RDT and microscopy in malaria surveillance data from Bayelsa State, Nigeria. A dataset comprising 2,100 monthly observations from eight Local Government Areas spanning January 2019 to December 2024 was analyzed. The methodology incorporated Bland Altman agreement analysis, feature engineering with climate and health system variables, and gradient boosting classification with class weight balancing to address data imbalance. Model interpretation was achieved through SHapley Additive exPlanations (SHAP) analysis. The Bland Altman analysis revealed a mean difference of negative 2.33 percentage points between RDT and microscopy, with limits of agreement spanning negative 19.28 to positive 14.62 percentage points. The LightGBM classifier achieved an area under the receiver operating characteristic curve of 0.901, with precision of 0.67, recall of 0.74, and F1 score of 0.703. SHAP analysis identified rainfall, climate index, geographic location, and humidity as the most influential predictors of diagnostic discordance. This study contributes an interpretable machine learning framework for identifying conditions associated with diagnostic disagreement, potentially informing quality assurance protocols and targeted interventions in malaria endemic regions.

Keywords : Malaria Diagnosis, Diagnostic Discordance, Machine Learning, SHAP, Rapid Diagnostic Test, Microscopy.

Paper Submission Last Date
28 - February - 2026

SUBMIT YOUR PAPER CALL FOR PAPERS
Video Explanation for Published paper

Never miss an update from Papermashup

Get notified about the latest tutorials and downloads.

Subscribe by Email

Get alerts directly into your inbox after each post and stay updated.
Subscribe
OR

Subscribe by RSS

Add our RSS to your feedreader to get regular updates from us.
Subscribe