Bound states in the continuum (BICs) provide, what we believe to be, a novel and efficient way for light trapping. However, using BICs to confine the light into a three-dimensional compact volume remains a challenging task, since the energy leakage at the lateral boundaries dominates the cavity loss when its footprint shrinks to considerably small, and hence, sophisticated boundary designs turn out to be inevitable. Conventional design methods fail in solving the lateral boundary problem because a large number of degree-of-freedoms (DOFs) are involved. Here, we propose a fully automatic optimization method to promote the performance of lateral confinement for a miniaturized BIC cavity. Briefly, we combine a random parameter adjustment process with a convolutional neural network (CNN), to automatically predict the optimal boundary design in the parameter space that contains a number of DOFs. As a result, the quality factor that is accounted for lateral leakage increases from 4.32 × 104 in the baseline design to 6.32 × 105 in the optimized design. This work confirms the effectiveness of using CNNs for photonic optimization and will motivate the development of compact optical cavities for on-chip lasers, OLEDs, and sensor arrays.