In this Letter, we report an approach for the inverse design of binary apertures to generate desired three-dimensional (3D) diffraction patterns in free space. The approach relies on an optimal accumulation algorithm, aiming to determine the distribution of the binary aperture for 3D target patterns in the regime of Fresnel diffraction. This algorithm features high fidelity for complex inverse design compared with conventional iterative algorithms. To demonstrate the validity of our method, various 2D and 3D patterns are chosen and generated using a digital micromirror device that serves as a reconfigurable binary aperture. Experimentally, the generated diffraction patterns exhibit high fidelity with respect to the target ones, achieving an averaged Pearson correlation coefficient of 0.90 for 2D patterns and 0.87 for 3D patterns, respectively. Our work may find applications in laser beam shaping, structured light illumination, and diffractive optical elements.