In this paper, a gain-adapted speech recognition method in unknown noise is developed in the time domain. Noise is assumed to be colored. To cope with the notable nonstationary nature of speech signals such as fricative, glides, liquids, and transition region between phones, the nonstationary autoregressive (NAR) hidden Markov model (HMM) is used for clean speech. The nonstationary AR process is represented by using polynomial functions with a linear combination of M known basis functions. When only noisy signals are available, the estimation problem of unknown noise inevitably arises. By using multiple Kalman filters, the estimation of noise model and gain contour of speech is performed.