Authors : Aishwary Bodhale; Akanksha Meshram

Volume/Issue : Volume 11 - 2026, Issue 3 - March

Google Scholar : https://tinyurl.com/35z5969u

Scribd : https://tinyurl.com/589j4y5m

DOI : https://doi.org/10.38124/ijisrt/26mar527

Note : A published paper may take 4-5 working days from the publication date to appear in PlumX Metrics, Semantic Scholar, and ResearchGate.

Abstract : Human-in-the-Loop (HiL) systems integration into customer experience (CX) schemes is a disruptive strategy in terms of expanding service delivery without sacrificing quality. This is a review article that investigates the paradigm shift of orchestrated human intelligence and agentic AI in the context of CX with particular attention given to the metrics that are needed to assess, optimize, and scale such systems of hybrid forms. By conducting a topical review of literature published in 2015-2024, we realize the fact that the current CX measures do not sufficiently reflect the dynamic interaction between humans and artificial agents. According to the traditional measures of endpoint (CSAT, NPS, CES), it is not possible to measure the quality of orchestration, the effectiveness of handoff and the effectiveness of human-AI cooperation. Our framework of metrics includes five dimensions that are structured as (1) Orchestration Efficiency Metrics, (2) CX Journey Analytics, (3) Agent Performance Indicators, (4) Business Impact Measures, and (5) Learning and Adaptation Metrics. The review provides a synthesis of 87 relevant studies, which uncovered three new trends: the continuous versus discrete approach to measurement, the necessity to track the emotional trajectory, and the necessity to align multi-stakeholder metrics. This is, to the best of our knowledge, one of the first reviews to explicitly conceptualize human, AI orchestration as a quantifiable construct and to systematize measurements in specific Human-in-the-Loop customer experience system metrics. We end with research agenda that focuses on creation of real time orchestration scoring systems, considerations of ethical metrics and industry specific implementation. This publication has a contribution to the academic literature and practice as it offers a systematic method of measuring what is important in human-AI collaborative CX systems.

Keywords : Human-in-the-Loop; Agentic AI; Customer Experience Metrics; Service Orchestration; Human-AI Collaboration; CX Measurement.

References :

1. T. H. Davenport, A. Guha, D. Grewal, and T. Bressgott, “How artificial intelligence will change the future of marketing,” Journal of the Academy of Marketing Science, vol. 48, no. 1, pp. 24-42, 2020.
2. K. Holstein, J. W. Vaughan, H. Daumé III, M. Dudík, and H. Wallach, “Improving fairness in machine learning systems: What do industry practitioners need?” in Proc. 2019 CHI Conf. Human Factors in Computing Systems, pp. 1-16, 2019.
3. A. Rai, P. Constantinides, and S. Sarker, “Editorial case for integrative digital ensembles: Adopting complementary multi-entity perspectives,” MIS Quarterly, vol. 43, no. 4, pp. iii–xviii, 2019.
4. F. F. Reichheld, “The one number you need to grow,” Harvard Business Review, vol. 81, no. 12, pp. 46-55, 2003.
5. M. Dixon, K. Freeman, and N. Toman, “Stop trying to delight your customers,” Harvard Business Review, vol. 88, no. 7/8, pp. 116–122, 2010.
6. B. Larivière, D. Bowen, T. W. Andreassen, W. Kunz, N. J. Sirianni, C. Voss, N. V. Wünderlich, and A. De Keyser, “Service encounter 2.0: An investigation into the roles of technology, employees and customers,” Journal of Business Research, vol. 79, pp. 238–246, 2017.
7. R. Knote, A. Janson, M. Söllner, and J. M. Leimeister, “Value co-creation in smart services: A functional affordances perspective on smart personal assistants,” Journal of the Association for Information Systems, vol. 22, no. 2, pp. 418-458, 2021.
8. M. J. Page, J. E. McKenzie, P. M. Bossuyt, I. Boutron, T. C. Hoffmann, C. D. Mulrow, and D. Moher, “The PRISMA 2020 statement: An updated guideline for reporting systematic reviews,” BMJ, vol. 372, p. n71, 2021.
9. V. Braun and V. Clarke, “Using thematic analysis in psychology,” Qualitative Research in Psychology, vol. 3, no. 2, pp. 77–101, 2006.
10. K. N. Lemon and P. C. Verhoef, “Understanding customer experience throughout the customer journey,” Journal of Marketing, vol. 80, no. 6, pp. 69-96, 2016.
11. A.De Keyser, K. Verleye, K. N. Lemon, T. L. Keiningham, and P. Klaus, “Moving the customer experience field forward: Introducing the touchpoints, context, qualities (TCQ) nomenclature,” Journal of Service Research, vol. 23, no. 4, pp. 433-455, 2020.
12. V. Kumar, B. Rajan, R. Venkatesan, and J. Lecinski, “Understanding the role of artificial intelligence in personalized engagement marketing,” California Management Review, vol. 63, no. 4, pp. 135-155, 2021.
13. Gartner, “Market guide for conversational AI platforms,” Gartner Research, 2023.
14. Forrester Research, “The state of AI in customer service,” Forrester Research Report, 2022.
15. A. Baird and L. M. Maruping, “The next generation of research on IS use: A theoretical framework of delegation to and from agentic IS artifacts,” MIS Quarterly, vol. 45, no. 1, pp. 315-342, 2021.
16. S. L. Vargo and R. F. Lusch, “Institutions and axioms: An extension and update of service-dominant logic,” Journal of the Academy of Marketing Science, vol. 44, no. 1, pp. 5-23, 2016.
17. E. Hutchins, Cognition in the wild. Cambridge, MA: MIT Press, 1995.
18. J. Zhang and D. A. Norman, “Representations in distributed cognitive tasks,” Cognitive Science, vol. 18, no. 1, pp. 87-122, 1994.
19. P. Wright, B. Fields, and M. Harrison, “Analyzing human-computer interaction as distributed cognition: The resources model,” Human-Computer Interaction, vol. 15, no. 1, pp. 1-41, 2000.
20. R. W. Picard, Affective computing. Cambridge, MA: MIT Press, 1997.
21. R. W. Picard, “Affective computing: Challenges,” International Journal of Human-Computer Studies, vol. 59, no. 1-2, pp. 55-64, 2003.
22. Seeber, E. Bittner, R. O. Briggs, T. de Vreede, G. J. de Vreede, A. Elkins, and M. Söllner, “Machines as teammates: A research agenda on AI in team collaboration,” Information & Management, vol. 57, no. 2, p. 103174, 2020.
23. N. J. McNeese, M. Demir, N. J. Cooke, and C. Myers, “Teaming with a synthetic teammate: Insights into human-autonomy teaming,” Human Factors, vol. 63, no. 5, pp. 690-706, 2021.
24. E. Bittner, S. Oeste-Reiß, and J. M. Leimeister, “Where is the bot in our team? Toward a taxonomy of design option for hybrid collaboration in teams,” in Proc. 52nd Hawaii Int. Conf. System Sciences, 2019.
25. D. Dellermann, A. Calma, N. Lipusch, T. Weber, S. Weigel, and P. Ebel, “The future of human-AI collaboration: A taxonomy of design knowledge for hybrid intelligence systems,” in Proc. 52nd Hawaii Int. Conf. System Sciences, 2019.
26. B. Green and Y. Chen, “The principles and limits of algorithm-in-the-loop decision making,” Proceedings of the ACM on Human-Computer Interaction, vol. 3, CSCW, pp. 1-24, 2019.
27. A. Maedche, C. Legner, A. Benlian, B. Berger, H. Gimpel, T. Hess, and M. Söllner, “AI-based digital assistants,” Business & Information Systems Engineering, vol. 61, no. 4, pp. 535-544, 2019.
28. J. Sweller, “Cognitive load theory,” in Psychology of Learning and Motivation, vol. 55. New York: Academic Press, 2011, pp. 37-76.
29. R. M. Ryan and E. L. Deci, “Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being,” American Psychologist, vol. 55, no. 1, pp. 68-78, 2000.
30. C. R. Berger and R. J. Calabrese, “Some explorations in initial interaction and beyond: Toward a developmental theory of interpersonal communication,” Human Communication Research, vol. 1, no. 2, pp. 99-112, 1975.
31. R. T. Rust and M. H. Huang, “The service revolution and the transformation of marketing science,” Marketing Science, vol. 33, no. 2, pp. 206-221, 2014.
32. B. D. Mittelstadt, P. Allo, M. Taddeo, S. Wachter, and L. Floridi, “The ethics of algorithms: Mapping the debate,” Big Data & Society, vol. 3, no. 2, pp. 1-21, 2016.
33. C. Argyris and D. A. Schön, Organizational learning: A theory of action perspective. Reading, MA: Addison-Wesley, 1978.