A pattern recognition method for intrusion signals is introduced, leveraging distributed optical fiber sensors and convolutional neural networks. To enhance the model's generalization capabilities, two strategies are employed. Firstly, the sensor structure is refined, incorporating the strengths of both the Sagnac and Michelson interferometers to create a hybrid design with a wide frequency response. Comparative experiments confirm that this exceptional frequency response significantly boosts the model's generalization power. Secondly, to address the limitations of distributed fiber optic sensors, a multimodal approach is proposed, integrating video frame difference and fiber optic pattern recognition. The trained convolutional neural network model exhibits low computational overhead, robust generalization, and practical applicability. When applied to a dataset collected in diverse scenarios not included in the model training, the model achieved a CNR of 0% and a FPR of only 0.26% for the intrusion behavior of climbing fences.