In this paper, we carried out phase-field simulations for investigating self-organized composite morphologies formation and interface instability of two phase immiscible binary alloy thin-film growth during physical vapor deposition. Predicted results show that for lower the deposition rate the solid-gas interface keeps planar and the lateral composition modulation (LCM) configuration occurs. As the deposition rate increases, the self-organized composite morphology gradually transfers to vertical composition modulations (VCM) configuration and the random composition modulations (RCM) configuration with the decrease of the boundary layer of the incident vapor and the unsteady solid-gas interface. Results show that the nanoprecipitate concentration modulation (NCPM) depends strongly on the nominally phase fraction. Moreover, we found that the historical dependence of self-organized composite morphologies and solid-gas interface stability, i.e., the initial condition can strongly influence the microstructure of the thin film during physical vapor deposition. Finally, we provide an analysis method for interface instability based on fast Fourier transform (FFT), and the frequency characteristics of cellular and bump structures are identified.