This paper concerns the analysis of discrete-valued time series using a class of categorical ARMA models recently proposed by Biswas and Song (2009). Such ARMA processes are flexible to model discrete-valued time series, allowing a wide range of marginal distributions such as binomial, multinomial, Poisson and nominal/ordinal categorical probability mass functions. To apply these models in the data analysis this paper focuses on the development of a needed statistical toolbox, which includes maximum likelihood estimation and inference, model selection, and goodness-of-fit test. Particularly in AR models a bias-corrected AIC statistic is derived for the order selection, while a randomized conditional moment (RCM) test is furnished to examine the goodness-of-fit. Finite-sample performances of the proposed methods are examined through simulation studies, in which the bias-corrected AIC is shown to outperform the traditional AIC and BIC statistics and the RCM test achieves desirable power. As part of the numeric illustration, a data analysis of categorical time series on infant sleep quality is provided by the application of this new toolbox.