Authors :
Nitin Gupta
Volume/Issue :
Volume 10 - 2025, Issue 5 - May
Google Scholar :
https://tinyurl.com/4d4dhhe3
DOI :
https://doi.org/10.38124/ijisrt/25may2034
Note : A published paper may take 4-5 working days from the publication date to appear in PlumX Metrics, Semantic Scholar, and ResearchGate.
Abstract :
In modern distributed systems, particularly those leveraging asynchronous message processing, the ability to
introduce controlled delays for messages is crucial for various functionalities, including retry mechanisms, scheduled
deliveries, and rate throttling. This white paper presents the design and operational principles of a novel delay subsystem
built entirely on Apache Kafka. By strategically utilizing a set of Kafka topics that represent discrete delay durations, this
design eliminates the need for external schedulers, databases, or additional services, thereby minimizing architectural
complexity and coupling. It also solves the problem associated with data residency due to various legal concerns in the
banking and fintech industry. The paper details the "denomination" approach to accumulating arbitrary delays, analogous
to dispensing currency change. It elucidates the inherent advantages of this Kafka-native approach, such as natural sorting,
Kafka's robust write performance, and preventing data loss by retaining data temporarily. Furthermore, it provides a
comprehensive walkthrough of the message flow, worker behavior, and the critical role of message headers in maintaining
logical record integrity. This subsystem offers a highly scalable, resilient, and cost-effective solution for managing delayed
messages within a Kafka-centric architecture.
Keywords :
Kafka Delay System; Innovative System;Data Loss Prevention; Data Residency; Banking; Fintech; Data Complaint; Distributed Architecture.
References :
- Ravi Kiran Mallidi, Manmohan Sharma, Sreenivas Rao Vangala, “Streaming Platform Implementation in Banking and Financial Systems”, 2022,IEEE
- Djerf, Angela, “ A Comparitive Study between EU-GDPR and the US-CCPA” Department of Business Law, 2023, HARN63 20231
- Abhishek Mhalle, Jianming Yong, Xiaohui Tao, Jun Shen, “Data Privacy and System Security for Banking and Financial Service Industry based on Cloud Computing Infrastructure”,2018,IEEE
- Theofanis P. Rapts, Andrea Passarella, “On Efficieny Partitioning a Topic in Apache Kafka”, 2022, arXiv:2205.09415
- Shreya Gupta, Boyang Huang, Russell Impagliazzo, “The Greedy Coin Change Problem”, 2024, arXiv.2411.18137
- Jiaxin Li,Qian Li, “Analysis of quue management in theme parks introducing the fast pass system”, 2023, Elsevier Ltd
- Dylan Scott, Viktor Gamov, Dave Klein, “Kafka in action”, 2022, Manning
In modern distributed systems, particularly those leveraging asynchronous message processing, the ability to
introduce controlled delays for messages is crucial for various functionalities, including retry mechanisms, scheduled
deliveries, and rate throttling. This white paper presents the design and operational principles of a novel delay subsystem
built entirely on Apache Kafka. By strategically utilizing a set of Kafka topics that represent discrete delay durations, this
design eliminates the need for external schedulers, databases, or additional services, thereby minimizing architectural
complexity and coupling. It also solves the problem associated with data residency due to various legal concerns in the
banking and fintech industry. The paper details the "denomination" approach to accumulating arbitrary delays, analogous
to dispensing currency change. It elucidates the inherent advantages of this Kafka-native approach, such as natural sorting,
Kafka's robust write performance, and preventing data loss by retaining data temporarily. Furthermore, it provides a
comprehensive walkthrough of the message flow, worker behavior, and the critical role of message headers in maintaining
logical record integrity. This subsystem offers a highly scalable, resilient, and cost-effective solution for managing delayed
messages within a Kafka-centric architecture.
Keywords :
Kafka Delay System; Innovative System;Data Loss Prevention; Data Residency; Banking; Fintech; Data Complaint; Distributed Architecture.
