Authors : Ajinkya Mhatre; Sandeep R. Warhade; Omkar Pawar; Sayali Kokate; Samyak Jain; Dr. Emmanuel M

Volume/Issue : Volume 9 - 2024, Issue 5 - May

Google Scholar : https://tinyurl.com/yshxnbm4

Scribd : https://tinyurl.com/65ub5wpx

DOI : https://doi.org/10.38124/ijisrt/IJISRT24MAY1964

Abstract : Using the application of Large Language Models (LLMs) in healthcare settings, mainly focusing on addressing general illness inquiries through chatbot interfaces. Leveraging the capabilities of LLMs, explore their potential to provide accurate and contextually relevant responses to users seeking information about common health concerns. LLM have the capacity continuously learn and improve from user interaction. Through benchmarking experiments, this paper evaluates the accuracy (61%) of LLM-based chatbots in understanding and responding to user queries related to general illnesses. The findings demonstrate the performance of LLMs against established benchmarks, shedding light on their efficacy in healthcare applications. By examining the intersection of LLM technology and healthcare, this research contributes to advancing the development of intelligent chatbot systems capable of providing reliable and informative support to individuals seeking medical guidance for general health issues.

Keywords : LLM, Healthcare, Chatbot, Benchmark, Accuracy.

References :

1. P. Rajpurkar, E. Chen, O. Banerjee, and E. J. Topol, “AI in health and medicine,” Nature Medicine, vol. 28, no. 1. Nature Research, pp. 31–38, Jan. 01, 2022. doi: 10.1038/s41591-021-01614-0.
2. B. Galitsky, “LLM-based Personalized Recommendations in Health,” 2024, doi: 10.20944/preprints202402.1709.v1.
3. R. Szilágyi and M. Tóth, “Use of LLM for SMEs, opportunities and challenges,” Journal of Agricultural Informatics, vol. 14, no. 2, Jan. 2024, doi: 10.17700/jai.2023.14.2.703.
4. J. G. Meyer et al., “ChatGPT and large language models in academia: opportunities and challenges,” BioData Mining, vol. 16, no. 1. BioMed Central Ltd, Dec. 01, 2023. doi: 10.1186/s13040-023-00339-9.
5. Y. Yao, J. Duan, K. Xu, Y. Cai, Z. Sun, and Y. Zhang, “A survey on Large Language Model (LLM) security and privacy: The Good, The Bad, and The Ugly,” High-Confidence Computing, p. 100211, Mar. 2024, doi: 10.1016/j.hcc.2024.100211.
6. F. Jiang et al., “Artificial intelligence in healthcare: Past, present and future,” Stroke and Vascular Neurology, vol. 2, no. 4. BMJ Publishing Group, pp. 230–243, Dec. 01, 2017. doi: 10.1136/svn-2017-000101.
7. Ankit Bhakkad, S.C. Dharmadhikari, M. Emmanuel, and Parag Kulkarni, E-VSM: Novel Text Representation Model to Capture Contex-Based Closeness between Two Text Documents. 2013.
8. Anjali Chauhan, A Review on Various Aspects of MongoDb Databases. 2019.
9. O. Topsakal and T. C. Akinci, “Creating Large Language Model Applications Utilizing LangChain: A Primer on Developing LLM Apps Fast,” International Conference on Applied Engineering and Natural Sciences, vol. 1, no. 1, pp. 1050–1056, Jul. 2023, doi: 10.59287/icaens.1127.
10. C. Luo, J. Zhan, L. Wang, and Q. Yang, “Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks,” Feb. 2017, [Online]. Available: http://arxiv.org/abs/1702.05870
11. M. Emmanuel, B. D. R. Ramesh, and S. M. Khatri, “A novel scheme for term weighting in text categorization : Positive impact factor,” in Proceedings - 2013 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2013, 2013, pp. 2292–2297. doi: 10.1109/SMC.2013.392.
12. R. Hasan MBA and J. Ferdous, “Dominance of AI and Machine Learning Techniques in Hybrid Movie Recommendation System Applying Text-to-number Conversion and Cosine Similarity Approaches,” 2024, doi: 10.32996/jcsts.
13. P. Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” May 2020, [Online]. Available: http://arxiv.org/abs/2005.11401
14. A. Pal, L. K. Umapathi, and M. Sankarasubbu, “MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering,” 2022.
15. M. R. Parvez, W. U. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Retrieval Augmented Code Generation and Summarization,” Aug. 2021, [Online]. Available: http://arxiv.org/abs/2108.11601
16. W. Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention,” in SOSP 2023 - Proceedings of the 29th ACM Symposium on Operating Systems Principles, Association for Computing Machinery, Inc, Oct. 2023, pp. 611–626. doi: 10.1145/3600006.3613165.
17. Y. Li, Z. Song, and W. Li, “Benchmarking Large Language Models in Adolescent Growth and Development: A Comparative Analysis of Claude2, ChatGPT-3.5, and Google Bard,” 2024, doi: 10.21203/rs.3.rs-3858549/v1.
18. B. Niu and G. F. N. Mvondo, “I Am ChatGPT, the ultimate AI Chatbot! Investigating the determinants of users’ loyalty and ethical usage concerns of ChatGPT,” Journal of Retailing and Consumer Services, vol. 76, Jan. 2024, doi: 10.1016/j.jretconser.2023.103562.
19. L. Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” Jun. 2023, [Online]. Available: http://arxiv.org/abs/2306.05685
20. S. Schwartz, A. Yaeli, and S. Shlomov, “Enhancing Trust in LLM-Based AI Automation Agents: New Considerations and Future Challenges,” Aug. 2023, [Online]. Available: http://arxiv.org/abs/2308.05391
21. B. D. Lund, D. Khan, and M. Yuvaraj, “ChatGPT in medical libraries, possibilities and future directions: An integrative review,” Health Info Libr J, Mar. 2024, doi: 10.1111/hir.12518.