Abstract
Most of the non-asymptotic theoretical work in regression is carried out for the square loss, where estimators can be obtained through closed-form expressions. In this paper, we use and extend tools from the convex optimization literature, namely self-concordant functions, to provide simple extensions of theoretical results for the square loss to the logistic loss. We apply the extension techniques to logistic regression with regularization by the ℓ2-norm and regularization by the ℓ1-norm, showing that new results for binary classification through logistic regression can be easily derived from corresponding results for least-squares regression.
Highlights
The theoretical analysis of statistical methods is usually greatly simplified when the estimators have closed-form expressions
We have provided an extension of self-concordant functions that allows the simple extensions of theoretical results for the square loss to the logistic loss
We have applied the extension techniques to regularization by the l2-norm and regularization by the l1-norm, showing that new results for logistic regression can be derived from corresponding results for leastsquares regression, without added complex assumptions
Summary
The theoretical analysis of statistical methods is usually greatly simplified when the estimators have closed-form expressions. Its classical analysis requires cumbersome notations and assumptions regarding second and third-order derivatives (see, e.g., [6, 7]) This situation was greatly enhanced with the introduction of the notion of self-concordant functions, i.e., functions whose third derivatives are controlled by their second derivatives. This essentially shows that the analysis of logistic regression can be done non-asymptotically using the local quadratic approximation of the logistic loss, without complex additional assumptions. In order to consider such extensions and make sure that the new results closely match the corresponding ones for least-squares regression, we derive in Appendix G new Bernstein-like concentration inequalities for quadratic forms of bounded random variables, obtained from general results on U-statistics [11]. We let denote P and E general probability measures and expectations
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
