Interfaces between structural multi-materials generally exhibit asymmetric resistance to tension and compression. Given this interface behavior, this work suggests an energy-based approach to improve the interface configuration for multi-material topology optimization. Based on the strain spectral decomposition, we decompose the structural elastic strain energy into tensile and compressive portions. In the density-based topology optimization framework, we use the gradient-based method to track the interface between multiple materials. Then, we construct an interface-associated scalar field to penalize the tensile portion of the strain energy, causing a pseudo-degradation of the strain energy at the interface region. Finally, within limited material usages and by minimizing the linear weighted structural strain energy and its pseudo-degradation, multi-material topology optimization with improved interface configuration is achieved. Several 2D and 3D numerical examples are investigated, by which the effectiveness and robustness of the suggested approach are fairly validated.