AbstractHydrological models are widely used for flood forecasting, and proper initialization of hydrological models is essential. Although the unscented Kalman filter (UKF) has shown promise in the context of operational forecasting, it is limited by its intrinsic instability caused by the loss of positive semidefiniteness of the covariance matrix. Effective methods for tackling this problem have not been seen in the literature. The cubature Kalman filter (CKF) is a powerful nonlinear filter for state estimation, which has received little attention in the hydrologic context. In this paper, to overcome the instability problem of the UKF, a new modified UKF (UKF‐M) is proposed by modifying the default UKF tuning strategy, and a novel nonlinear filter (CKF) is introduced to update the states of hydrological models. The methods are tested using a lumped hydrological model. The results of the stability experiment suggest that both the UKF‐M and CKF can overcome the instability problem, while the original UKF can fail due to the loss of positive semidefiniteness of the covariance matrix. The results of a reforecast experiment in an operational context show that both filters can improve the forecast performance by updating the model state based on the observed discharge, with the UKF‐M achieving better forecast performances. The UKF‐M and CKF can be used to overcome the instability problem of the original UKF and are valuable methods for improving forecast performances of hydrological models by state updating.