Delamination defects, as common damages occurring in fiber reinforced composite laminates, are often difficult to be detected due to their invisibility and concealment. Terahertz (THz) technique, as a promising alternative to conventional nondestructive testing (NDT) approaches, has emerged great potentials in quantitative characterization of delamination defects in composites. However, since THz signals are susceptible to confounding interference during THz inspection, such as the noise, interlayer reflection, dispersion and overlap, the accuracy and speed of defect localization and identification are usually contradictory. Generally, to improve the accuracy, many complex manual signal processing methods can be applied, which however will inevitably decrease the inference speed for delamination inspection and cannot fulfill the automatic and real-time analysis requirements in practical THz applications. Here, an intelligent THz 3D characterization system based on the lightweight deep learning (DL) model is proposed to realize the automatic and real-time characterization of hidden delamination defects in composites with high accuracy and low CPU latency. The core aims to specifically design a lightweight and end-to-end CPU adaptive network to obtain the optimal balance between the accuracy and speed. A series of experiments are implemented to validate the superior comprehensive performance of the system in terms of classification accuracy and inference speed on low-cost CPU-based platform. Compared with other methods, the proposed system can be not only able to obtain the superior accuracy, but also to obtain low CPU latency. Therefore, our work provides a novel and general paradigm for DL-based THz intelligent 3D characterization, which will promote the deployment of the system for the automatic and real-time THz characterization in industrial applications.