Precisely sensing the light field direction information plays the essential role in the fields of three-dimensional (3D) imaging, light field sensing, target positioning and tracking, remote sensing, etc. It is thrilling to find that the optical fiber can be used as a sensing component due to its high sensitivity, compact size, and strong resistance to electromagnetic interference. According to the core principle that the few-mode fiber output speckle pattern is sensitive to the change of incident light field direction, the variation characteristics is further investigated in this research study. Based on the simulation and analysis of the fiber transmission characteristics, the output speckle corresponding to the incident light field with the direction in the range of ±6° horizontally and vertically are calculated. Furthermore, a deep convolutional neural network (CNN): fiber speckle demodulation network (FSDNET) is proposed and constructed to establish what we believe to be a novel way to reveal and identify the mapping relationship between the light field direction and the output speckle. The theoretical simulation shows that the mean absolute error (MAE) between the perceived light field directions and the true directions is 0.01°. Then, a light field direction sensing system based on the few-mode fiber is developed. Regarding to the performance of the sensing system, the MAE of the FSDNET for the light field directions that have appeared in the training set is 0.0389°, and for testing set of the unknown directions that have not appeared in the training set, the MAE is 0.0570°. Therefore, the simulation and experimental results prove that high performance sensing of light field direction can be achieved by the proposed few-mode fiber sensing system and the FSDNET.