In this note we consider continuous-time systems x'(t) = A(t) x(t) + B(t) u(t), y(t) = C(t) x(t) + D(t) u(t), as well as discrete-time systems x(t+1) = A(t) x(t) + B(t) u(t), y(t) = C(t) x(t) + D(t) u(t) whose coefficient matrices A, B, C and D are not exactly known. More precisely, all that is known about the systems is their nonzero pattern, i.e., the locations of the nonzero entries in the coefficient matrices. We characterize the patterns that guarantee controllability and observability, respectively, for all choices of nonzero time functions at the matrix positions defined by the pattern, which extends a result by Mayeda and Yamada for time-invariant systems. As it turns out, the conditions on the patterns for time-invariant and for time-varying discrete-time systems coincide, provided that the underlying time interval is sufficiently long. In contrast, the conditions for time-varying continuous-time systems are more restrictive than in the time-invariant case.