This paper presents a nonlinear stochastic optimal control algorithm based on an enhanced stochastic averaging method of energy envelope to mitigate the response of nonlinear hysteretic systems. The enhanced stochastic averaging strategy for non-integrable hysteretic systems addresses some of the shortcomings of conventional methods including consideration of a collective contribution of damping terms and the assumption of the independence of stochastic excitations. Hamilton-Jacobi-Bellman equation is used to formulate the optimization problem with a cost function that incorporates high order terms of the energy of the system in order to optimize the dynamic response. Unlike other techniques, this strategy considers the stochastic nonlinear behavior of the system in the optimization process of the control design. The performance of the proposed controller is evaluated for a one-story building on a nonlinear foundation under white noise and historic ground motions. The results are compared with those for the uncontrolled structure and the structure controlled using a linear quadratic regulator algorithm designed using stochastic linearization (SLQR). The proposed method is shown to provide considerable improvements over other control cases. In particular under Kobe ground motion, 14% and 21% reductions are observed for the maximum relative displacement and foundation displacement, respectively compared to the SLQR with similar maximum control force levels.