The analysis of non-Gaussian time series has been studied extensively and has many applications. Many successful models can be viewed as special cases or variations of the generalized autoregressive moving average (GARMA) models of Benjamin et al. (2003), where a link function similar to that used in generalized linear models is introduced and the conditional mean, under the link function, assumes an ARMA structure. Under such a model, the ‘transformed’ time series, under the same link function, assumes an ARMA form as well. Unfortunately, unless the link function is an identity function, the error sequence defined in the transformed ARMA model is usually not a martingale difference sequence. In this paper we extend the GARMA model in such a way that the resulting ARMA model in the transformed space has a martingale difference sequence as its error sequence. The benefit of such an extension are four-folds. It has easily verifiable conditions for stationarity and ergodicity; its Gaussian pseudo-likelihood estimator is consistent; standard time series model building tools are ready to use; and its MLE’s asymptotic distribution can be established. We also proposes two new classes of non-Gaussian time series models under the new framework. The performance of the proposed models is demonstrated with simulated and real examples.