This paper presents a comprehensive investigation of the problem of a harmonic oscillator with time-depending frequencies in the framework of the Vlasov theory and the Wigner function apparatus for quantum systems in the phase space. A new method is proposed to find an exact solution of this problem using a relation of the Vlasov equation chain with the Schrödinger equation and with the Moyal equation for the Wigner function. A method of averaging the energy function over the Wigner function in the phase space can be used to obtain time-dependent energy spectrum for a quantum system. The Vlasov equation solution can be represented in the form of characteristics satisfying the Hill equation. A particular case of the Hill equation, namely the Mathieu equation with unstable solutions, has been considered in details. An analysis of the dynamics of an unstable quantum system shows that the phase space square bounded with the Wigner function level line conserves in time, but the phase space square bounded with the energy function line increases. In this case the Vlasov equation characteristic is situated on the crosspoint of the Wigner function level line and the energy function line. This crosspoint moves in time with a trajectory that represents the unstable system dynamics. Each such trajectory has its own energy, and the averaging of these energies by the Wigner function results in time-dependent discreet energy spectrum for the whole system. An explicit expression has been obtained for the Wigner function of the 4th rank in the generalized phase space