Authors : Dhiraj Manoj Shribate

Volume/Issue : Volume 10 - 2025, Issue 3 - March

---

Google Scholar : https://tinyurl.com/4ukkncsn

Scribd : https://tinyurl.com/yttekmr4

DOI : https://doi.org/10.38124/ijisrt/25mar147

Google Scholar

Note : A published paper may take 4-5 working days from the publication date to appear in PlumX Metrics, Semantic Scholar, and ResearchGate.

Note : Google Scholar may take 15 to 20 days to display the article.

---

Abstract : Optimization plays a crucial role in the development and performance of machine learning models. Various optimization techniques have been developed to enhance model efficiency, accuracy, and generalization. This paper provides a comprehensive review of optimization algorithms used in machine learning, categorized into first-order, second-order, and heuristic-based methods. We discuss their advantages, limitations, and applications, highlighting recent advancements and future research directions.

References :

1. Kingma, D. P., & Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980.
2. Nesterov, Y. (1983). A Method for Solving the Convex Programming Problem with Convergence Rate O(1/k²). Soviet Mathematics Doklady, 27(2), 372-376.
3. Robbins, H., & Monro, S. (1951). A Stochastic Approximation Method. The Annals of Mathematical Statistics, 22(3), 400-407.
4. Broyden, C. G. (1970). The Convergence of a Class of Double-rank Minimization Algorithms 1. General Considerations. IMA Journal of Applied Mathematics, 6(1), 76-90.
5. Hansen, N., & Ostermeier, A. (2001). Completely Derandomized Self-Adaptation in Evolution Strategies. Evolutionary Computation, 9(2), 159-195.
6. Kennedy, J., & Eberhart, R. (1995). Particle Swarm Optimization. Proceedings of ICNN'95 - International Conference on Neural Networks, 4, 1942-1948.
7. Kirkpatrick, S., Gelatt, C. D., & Vecchi, M. P. (1983). Optimization by Simulated Annealing. Science, 220(4598), 671-680.
8. Dorigo, M., & Gambardella, L. M. (1997). Ant Colony System: A Cooperative Learning Approach to the Traveling Salesman Problem. IEEE Transactions on Evolutionary Computation, 1(1), 53-66.
9. Storn, R., & Price, K. (1997). Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces. Journal of Global Optimization, 11(4), 341-359.
10. LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep Learning. Nature, 521(7553), 436-444.
11. Bottou, L. (2018). Stochastic Gradient Descent Tricks. In Neural Networks: Tricks of the Trade, 421-436. Springer.
12. Sutskever, I., Martens, J., Dahl, G. E., & Hinton, G. E. (2013). On the Importance of Initialization and Momentum in Deep Learning. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), 1139-1147.
13. Nocedal, J., & Wright, S. J. (2006). Numerical Optimization (2nd ed.). Springer.
14. Shewchuk, J. R. (1994). An Introduction to the Conjugate Gradient Method Without the Agonizing Pain. Technical Report, CMU-CS-94-125.
15. Yang, X. S., et al. (2014). A New Metaheuristic Bat-Inspired Algorithm. In Nature-Inspired Computation and Applications (pp. 65-74). Springer.