Nowadays, the reliability of data analysis and decision-making based on deep learning (DL) remains a primary concern in promoting DL technology for industrial non-destructive testing and evaluation (NDT&E). This study focuses on the quantitative assessment of the reliability of various automated data analysis techniques in NDT&E. To achieve this, optical pulsed thermography was employed to inspect three non-planar carbon-fiber-reinforced polymer (CFRP) samples, each containing embedded Teflon to simulate debonding defects. After applying thermographic signal reconstruction and first-order derivation processing to the raw thermal data, automated analysis was performed using three image-processing-based segmentation methods and three sequential-signal-based DL algorithms. Additionally, an experienced inspector manually analyzed the data for comparison purposes. Subsequently, the probability of detection and false positive analyses were conducted to quantitatively evaluate and compare the reliability of the automated and human-based evaluations. The comparison results demonstrated that optimal DL classification and advanced image-processing-based segmentation techniques could achieve performance levels close to that of human inspectors in defect detection, even for challenging non-planar CFRP samples.