The probability density function for transient response of non-linear stochastic system is investigated through the stochastic averaging and Mellin transform. The stochastic averaging based on the generalized harmonic functions is adopted to reduce the system dimension and derive the one-dimensional Itô stochastic differential equation with respect to amplitude response. To solve the Fokker–Plank–Kolmogorov equation governing the amplitude response probability density, the Mellin transform is first implemented to obtain the differential relation of complex fractional moments. Combining the expansion form of transient probability density with respect to complex fractional moments and the differential relations at different transform parameters yields a set of closed-form first-order ordinary differential equations. The complex fractional moments which are determined by the solution of the above equations can be used to directly construct the probability density function of system response. Numerical results for a van der Pol oscillator subject to stochastically external and parametric excitations are given to illustrate the application, the convergence and the precision of the proposed procedure.