We study nonstationary frames of matrix-valued Gabor systems and wavelet systems in the matrix-valued function space [Formula: see text]. First, we show that a diagonal matrix-valued window function constitutes a frame for [Formula: see text] whenever each diagonal entry constitutes a frame for the space [Formula: see text]. This is not true for arbitrary nonzero matrix-valued function. Using this, we prove the existence of nonstationary matrix-valued Gabor frames associated with the Weyl–Heisenberg group in terms of density of real numbers. We give a representation of the frame operator of matrix-valued nonstationary Gabor system. A necessary condition with explicit frame bounds for nonstationary matrix-valued Gabor frames associated with the Weyl–Heisenberg group is given. We discuss matrix-valued frame preserving maps in terms of adjointablity, with respect to the matrix-valued inner product, of bounded linear operators acting on [Formula: see text]. It is shown that the image of a matrix-valued Gabor frame under bounded, linear and invertible operator on [Formula: see text] may not be a frame for [Formula: see text]. In this direction, we give sufficient conditions on bounded linear operators which can preserve frame conditions. Finally, we give necessary and sufficient condition for the existence of nonstationary matrix-valued wavelet frames associated with the extended affine group.