Mathematical justification on the origin of the sigmoid in logistic regression

Abstract

Logistic regression is a commonly used classification algorithm in machine learning. It allows categorizing data into discrete classes by learning the relationship from a given set of labeled data. It learns a linear relationship from the given data set and then introduces nonlinearity through an activation function to determine a hyperplane that separates the learning points into two subclasses. In the case of logistic regression, the sigmoid is the most used activation function to perform binary classification. The choice of sigmoid for binary classifications is justified by its ability to transform any real number into a probability between 0 and 1. This study provides, through two different approaches, a rigorous mathematical answer to the crucial question that torments us, namely where does this logistic function on which most neural network algorithms are based come from? References Stoltzfus, "Logistic regression: A brief primer," Academic Emergency Medicine, vol. 18, no. 10, pp. 1099-1104, 2011. Peng et al, "An introduction to logistic regression analysis and reporting," The Journal of Educational Research, vol. 96, no. 1, pp. . Costa et al, "A logistic regression model for consumer default risk," Journal of Applied Statistics, vol. 47, no. 13-15, pp. 2879-2894, 2020. Gatzert and M. Schubert, "Cyber risk management in the us banking and insurance industry: A textual and empirical analysis of determinants and value," Journal of Risk and Insurance, vol. 89, no. 3, pp. 725-763, 2022. Premsmith and H. Ketmaneechairat, "A predictive model for heart disease detection using data mining techniques," Journal of Advances in Information Technology, vol. 12, pp. 14-20, 2021. Almotiri et al, "Using api with logistic regression model to predict hotel reservation cancellation by detecting the cancellation factors," International Journal of Advanced Computer Science and Applications, vol. 12 , no. 6 , pp. . Kudupudi et al, "Spam message detection using logistic regression," International Journal of Advanced Computer Science and Applications, vol. 9, no. 9, pp. 815-818, 2021. Keren and . Cuiwei, "Housing price analysis using linear regression and logistic regression: A comprehensive explanation using melbourne real estate data," in 2021 IEEE International Conference on Computing (ICOCO), 2021, pp. 241-246. Dheeru and G. Casey, "UCI machine learning repository," 2017. [Online]. Available: http://archive.ics. uci.edu/ml Goldbloom, "Kaggle dataset," 2010. [Online]. Available: https: //www.kaggle.com/ Febrianti et al, "The parameter estimation of logistic regression with maximum likelihood method and score function modification," Journal of Physics: Conference Series, vol. 1725, no. 1, p. 012014, 2021. Lucena, "Loss functions for classification using structured entropy," 2022. [Online]. Available: https://arxiv.org/abs/2206.07122 Hikmat and A. Abdulazeez, "Comparison of optimization techniques based on gradient descent algorithm: a review," PalArch?s Journal of Archaeology of Egypt/Egyptology, vol. 18, no. 4, pp. 2715-2743, 2021.E. Norton et al, “Odds ratios-current best practice and use,” Journal of the American Medical Association, vol. 320, no. 1, pp. 84–85, 2018 Tong, Fundamental Properties and Sampling Distributions of the Multivariate Normal Distribution. Springer New York, 1990, pp. 23– 61.