Squeal is a noise phenomenon occurring in the last stage of automobile braking with a high-frequency sound. It is very difficult to express the phenomenon using a mathematical model, since the origin of squeal noise is physically complex. However, the possibility of squeal generation can be predicted by solving the vibration equation of the self-excited system using the complex eigenvalue analysis method. The results of the method are expressed as the magnitude of the unstable mode, and the generation of squeal noise can be prevented by reducing the magnitude of the unstable mode of the brake system. The objective of this research is to determine the optimal design process focused on the calliper housing shape to suppress squeal noise generation by reducing the system instability. The objective function is set to minimize the real part of the complex eigenvalue, i.e. the instability index. In the optimization design process, the design variable for topology optimization is established by focusing on the finger part of the calliper housing, which transmits the braking pressure to the pad lining. To supplement the complex shape generated by the topology optimization process, parametric design variables are selected for the subsequent process. Parameters are set to adjust the housing finger stiffness and are defined by considering the topology optimization result. Finally, the asymmetric shape of the calliper housing is obtained to reduce squeal noise generation.