The threaded sleeve connection (TSC) is widely used in the connection of reinforcing bars and the looseness is difficult to observe directly. The guided wave (GW) can be used to detect TSC’s looseness due to its ability to propagate to invisible regions, which is difficult to identify with traditional signal processing methods if the looseness is slight. In this study, we employ the convolutional neural network (CNN) to process the magnetostrictive GW signals for the detection of TSC’s slight looseness. Experiments are carried out on the thread-sleeve-connected reinforcing bars to obtain the passing GW signals as the dataset to train the CNNs and the states of the TSC contain tighten and slight looseness in which loosening angle is less than 5°. In order to improve the time–frequency resolution of CNN, a CNN with multi-scale kernel size is built. For comparison, another three CNNs with one kernel size are built. Using the GW signals, we train these four CNNs and then these four trained CNNs are employed to analyze the testing data. Only the CNN with multi-scale kernel size can achieve 100% slight looseness detection accuracy. The results show that the CNN can be used to detect the slight looseness of reinforcing bar’s TSC and the CNN trained with multi-scale kernel size performs better.