Purpose: The intermediate goal of this work is to develop a novel 3D image boundary detection algorithm in order to reliably detect the soft tissue boundaries in patient CT images. This is the very critical first step towards a potential solution for the currently unsolved problem ‐ automatic verification of deformable image registration (DIR) on patient images. Our hypothetic procedure of DIR verification is 1) tissue boundary detection, 2) boundary matching, and 3) verification of DIR results using the matched tissue boundaries. Methods: A novel 3D image boundary detection algorithm was developed. Multiple steps were used in the tissue boundary procedure, including 3D gradient computation supporting anisotropic voxels, hysteresis thresholding, non‐maxima suppression directly based on the boundary normal directions, boundary thinning, tissue boundary type selection based on the image intensity values on both sides of the boundaries, boundary separation and voxel reconnection based on boundary probability tests. These steps are innovative and critical to enable reliable detection of separated tissue boundaries that would be used in the following boundary matching step. Preprocessing includes automatic couch table removal and non‐local‐mean noise reduction. Post‐processing includes small piece removal and boundary voxels to point coordinate conversion. Results: The algorithm was implemented in MATLAB and tested using patients' head‐neck and pelvis CT images. Results were visually assessed. Compared to the results of the 3D extension of the standard Canny edge detection algorithm, results of the proposed algorithm are significantly more accurate and reliable. Conclusion: The proposed algorithm is a new approach of medical image processing. Combined with image feature matching methods from computer vision fields, it could lead to a potential solution for DIR verification, which is the current roadblock for many clinical applications including treatment adaptation, automatic segmentation and treatment response evaluation.