Authors : Sheyasree Ghosh; Akanksha Srivastava; Soumen Bhowmik; Subhajyoti Maity

Volume/Issue : Volume 11 - 2026, Issue 9 - September

Google Scholar : https://tinyurl.com/dtsan4md

DOI : https://doi.org/10.38124/ijisrt/26sep333

Note : A published paper may take 4-5 working days from the publication date to appear in PlumX Metrics, Semantic Scholar, and ResearchGate.

Abstract : Large Language Models (LLMs) find increasing usage in applications that involve reasoning over large spans of text such as document comprehension, question-answering across multiple documents, and large scale code understanding. However, even current state-of-the-art designs struggle to efficiently capture long-range dependencies. Despite Transformerbased designs being highly effective in handling tasks with short contexts, they suffer from scalability issues as well as degradation of contextual effects caused by the high complexity of the underlying attention computations. In this paper, we conduct a comparative study of Transformer-based and State-Space Model designs when it comes to long-context reasoning. A selection of representative models from both classes is systematically studied through carefully conducted experiments involving various types of long-sequence generation and comprehension tasks. In order to ensure comparability, a new metric called Context Retention Score (CRS) is proposed which quantifies the effect of distant tokens on model decisions. As evidenced in our results, Transformer-based models have a pronounced tendency to forget distant information while StateSpace Model-based designs retain information more stably and yield better results for tasks that involve long context.

Keywords : LLM,GPT-4, Data Hiding, Rotary Position Embedding, State-Space Model

References :

1. Vaswani, A., et al. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems (NeurIPS).
2. Brown, T., et al. (2020). Language Models Are Few-Shot Learners. NeurIPS.
3. Touvron, H., et al. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302.13971.
4. Gu, A., & Dao, T. (2023). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv preprint arXiv:2312.00752.
5. Peng, B., et al. (2023). RWKV: Reinventing RNNs for the Transformer Era. arXiv preprint arXiv:2305.13048.
6. Beltagy, I., Peters, M. E., & Cohan, A. (2020). Longformer: The Long-Document Transformer. arXiv preprint arXiv:2004.05150.
7. Zaheer, M., et al. (2020). BigBird: Transformers for Longer Sequences. NeurIPS.
8. Press, O., Smith, N. A., & Lewis, M. (2021). Train Short, Test Long: Attention with Linear Biases. arXiv preprint arXiv:2108.12409.
9. Fu, J., et al. (2023). LongBench: Benchmarking Long-Context Understanding. arXiv preprint arXiv:2308.14508.
10. OpenAI (2023). GPT-4 Technical Report. arXiv preprint arXiv:2303.08774.