In the theory of quantum dynamical filtering, one of the biggest issues is that the underlying system dynamics represented by a quantum stochastic differential equation must be known exactly in order that the corresponding filter provides an optimal performance; however, this assumption is generally unrealistic. Therefore, in this paper, we consider a class of linear quantum systems subjected to time-varying norm-bounded parametric uncertainties and then propose a robust observer such that the variance of the estimation error is guaranteed to be within a certain bound. Although in the linear case much of classical control theory can be applied to quantum systems, the quantum robust observer obtained in this paper does not have a classical analogue due to the system's specific structure with respect to the uncertainties. Moreover, by considering a typical quantum control problem, we show that the proposed robust observer is fairly robust against a parametric uncertainty of the system even when the other estimators--the optimal Kalman filter and risk-sensitive observer--fail in the estimation.