Improved Hoeffding’s Lemma and Hoeffding’s Tail Bounds

David Hertz

doi:10.5539/ijsp.v10n5p27

Abstract

The purpose of this article is to improve Hoeffding&#39;s lemma and consequently Hoeffding&#39;s tail bounds. The improvement pertains to left skewed zero mean random variables X\in[a,b], where a&lt;0 and -a&gt;b. The proof of Hoeffding&#39;s improved lemma uses Taylor&#39;s expansion, the convexity of \exp(sx), s\in \RR, and an unnoticed observation since Hoeffding&#39;s publication in 1963 that for -a&gt;b the maximum of&nbsp; the intermediate function \tau(1-\tau) appearing in Hoeffding&#39;s proof is attained at an endpoint rather than at \tau=0.5 as in the case b&gt;-a. Using Hoeffding&#39;s&nbsp; improved lemma we obtain one sided and two sided&nbsp; tail bounds&nbsp; for \PP(S_n\ge t) and \PP(|S_n|\ge t), respectively, where S_n=\sum_{i=1}^nX_i and the X_i\in[a_i,b_i],i=1,...,n are independent zero mean&nbsp; random variables (not necessarily identically distributed). It is interesting to note that we&nbsp; could&nbsp; also improve Hoeffding&#39;s two sided bound for all \{X_i:&nbsp; -a_i\ne b_i,i=1,...,n\}. This is&nbsp; so because here the one sided bound should be&nbsp; increased by \PP(-S_n\ge t),&nbsp; wherein the left skewed intervals become right skewed and vice versa.

Highlights

By googling Hoeffding bounds we obtain many applications and theoretical uses of Hoeffding’s bounds to signal processing to machine learning, to information theory, communication, and coding (Raginsky an Sason, 2018), to randomized algorithms (Mitzenmacher and Upfal, 2017) to name but a few
When the underlying distribution is skewed to the left we show how the one sided bound in Hoeffding’s Lemma can be improved
The organization of the remaining Sections is as follows. in Section 2 we present the proof of Hoeffding’s improved lemma

Summary

Introduction

By googling Hoeffding bounds we obtain many applications and theoretical uses of Hoeffding’s bounds to signal processing (e.g., to time series analysis, compressed sensing, sensor networks, financial signal processing, to name but a few) to machine learning, to information theory, communication, and coding (Raginsky an Sason, 2018), to randomized algorithms (Mitzenmacher and Upfal, 2017) to name but a few. Communication networks are treated in (Mitzenmacher and Upfal, 2017) where the authors rederive Hoeffding’s results and present applications to packet routing in sparse networks. When the underlying distribution is skewed to the left we show how the one sided bound in Hoeffding’s Lemma can be improved. The two sided Hoffding’s bound though can always be improved for skewed distributions either to the left or to the right.

Proof of Hoeffding’s Improved Lemma

Hoeffding’s Improved One Sided Tail Bound

Hoeffding’s Improved Two Sided Bound

Conclusion