In this paper, a family of infinite dimensional Lie algebras L˜ is introduced and investigated, called the extended Heisenberg-Virasoro algebra, denoted by L˜. These Lie algebras are related to the N=2 superconformal algebra and the Bershadsky-Polyakov algebra. We study restricted modules and associated vertex algebras of the Lie algebra L˜. More precisely, we construct its associated vertex (operator) algebras VL˜(ℓ123,0), and show that the category of vertex algebra VL˜(ℓ123,0)-modules is equivalent to the category of restricted L˜-modules of level ℓ123. Then we give uniform constructions of simple restricted L˜-modules. Also, we present several equivalent characterizations of simple restricted modules over L˜.