To control the relative position and relative velocity of spacecraft formation, a time-varying nonlinear relative motion model under J2 perturbation in an elliptical orbit is established in the port-Hamiltonian (PH) framework, and a distributed control law for spacecraft formation is developed using the consensus algorithm for parameter estimation and the passivity-based control (PBC) method based on the state-error interconnection and damping assignment (IDA) technique. First, the influence of J2 perturbation on the potential energy and orbit parameters in the model is considered when the relative motion model is built in the PH frame, and the expression of the relative motion acceleration under J2 perturbation is given. Second, to solve the problem that the state of the chief spacecraft cannot be directly obtained from the deputy spacecraft under distributed communication, a consensus-based parameter estimation method is introduced, the estimated parameter of the chief spacecraft is applied to the relative motion model in the PH frame, and a state error model with the estimated parameters derived from the consistency algorithm is established. Then, after the stability analysis is conducted on the desired PH system containing the estimated parameter values, the desired Hamiltonian energy function with the estimated parameters is designed according to the time-varying errors of the relative equilibrium states of the deputy spacecraft and the errors between deputy spacecraft, and the distributed control law of spacecraft formation is derived based on the state-error IDA-PBC method. Finally, the expected relative motion trajectory designed based on the TH equation is used to simulate a spacecraft formation under J2 perturbation, and the results indicate that the spacecraft can quickly converge to the expected formation under the influence of the control law.