This article proposes a vision-based virtual impedance control (VBVIC) scheme for the robotic system without pre-specified task trajectory, where a neural network (NN) is used to compensate for the unmodeled dynamics and parameter uncertainties. In the proposed scheme, a novel control framework combined virtual potential field (VPF) with the impedance control is proposed to achieve non-contact impedance control, where the task trajectory does not need to be pre-specified but is generated by integrating the bio-inspired Tau-J into the virtual impedance model. Moreover, the virtual force generated by VPF is utilized to achieve the shaping and tracking of the task trajectory in the virtual impedance control scheme, and the information of target and obstacle is obtained by the visual sensing. The proposed VPF-based VBVIC scheme is analyzed by the Lyapunov stability theory and validated by a carrying task in simulations and experiments, respectively.