The building mass damper (BMD) is a design concept in which a structure is substructured in such a way that the upper substructure behaves as a large-mass tuned mass damper for the lower one. Its correct tuning usually requires softening (partial isolating) the upper substructure, which limits its application for retrofitting. The recently proposed inerter-based building mass damper (IBMD) solves, reasonably, the softening dynamically through the use of an inerter. This implies an in parallel intervention, which drastically simplifies the practicability of the BMD for retrofitting. The present paper involves comprehensive numerical simulations for multi-objective optimization of an IBMD, accounting for inherent structural damping. In particular, the deformation of the upper substructure is an additional objective function, besides the deformation of the lower substructure as considered in a previous work. Results demonstrate that the IBMD outperforms various benchmark interventions, including dampers, stiffeners, softening devices, and inerters without dampers. Additionally, it was found that the inherent structural damping partially replaces the need for additional damping. Finally, it was also shown that emphasizing the optimization of the lower substructure deformation enhances the overall structural safety in most cases. Nevertheless, the multi-objective optimization increases the versatility of the design concept in the case of substructures with different allowable deformations.