It is quite important, in industrial communities, to design devices in order to let them have required abilities. The design process is, however, time-consuming since the classical design process requires a lot of trial and error process. To overcome this difficulty, we sometimes use topology optimization, which is considered as the most flexible computer-aided design method. In the previous study, we have proposed a topology optimization with the fast multipole boundary element method (FMBEM) for acoustic devices in 3D. The proposed method is, however, insufficient for practical use since the method deals only with rigid scatterer. In order to enhance the applicability of our method, in this paper, we present a topology optimization method with the FMBEM for 3D-acoustics with impedance boundary condition. A detailed derivation of the topological derivative for the problem and algorithm of the topology optimization, along with some numerical examples which verify the correctness of the proposed method, are presented.