Ambient energy harvesting is touted as a low cost solution to prolong the life of low-powered devices, reduce the carbon footprint, and make the system self-sustainable. Most research to date has focused either on the physical aspects of the energy conversion process or on the optimal consumption policies of the harvested energy at the system level. However, although intuitively understood, to the best of our knowledge, the idea of self-sustainability is yet to be systematically studied as a performance metric. In this paper, we provide a mathematical definition of the concept of self-sustainability of an energy harvesting system, based on the complementary idea of eventual energy outage, rather than the usual energy outage. In particular, we analyze a harvest-store-consume system with infinite battery capacity, stochastic energy arrivals, and fixed energy consumption rate. Using the random walk theory, we identify the necessary condition for the system to be self-sustainable. General formulas are given for the self-sustainability probability in the form of integral equations. Since these integral equations are difficult to solve analytically, an exponential upper bound for eventual energy outage is given using martingales. This bound guarantees that the eventual energy outage can be made arbitrarily small simply by increasing the initial battery energy. We then give an asymptotic formula; and for the special case when the energy arrival follows a Poisson process, we are also able to find an exact formula for the eventual energy outage probability. We also show that the harvest-store-consume system is mathematically equivalent to a GI / ${G}$ /1 queueing system, which allows us to easily find the energy outage probability, in case the necessary condition for self-sustainability is violated. Monte-Carlo simulations verify our analysis.