Many vision-based bolt looseness detection methods that directly observe the bolts have been developed. However, these methods have many limitations in terms of the conditions and processes of their implementation. To address these problems, this paper proposed a fully automated vision-based bolt looseness detection method for a rigid multi-bolt connection. The proposed method combines digital shearing speckle pattern interferometry (DSSPI) and recurrent neural network (RNN) and involves capturing speckle fringe patterns under various looseness cases using the DSSPI system and classifying these patterns with an RNN model to detect the loose bolts. The proposed method can detect all the bolts within the measured surface at one time, which is efficient. On the other hand, it eliminates the need for prior information such as the initial angle and position of each bolt. It can even detect unseen bolts in multi-bolt connections, making it applicable for connections in complex structures in which occlusion often occurs. Additionally, the method eliminates the complex process of distortion rectification. These features make the method achieve a single-judgment time (four bolts at one detection) of only 4.70 millisecond with a detection accuracy over 99%, which has potential for the real-time detection of loose bolts in multi-bolt connections.