Optical fiber Bragg sensors (FBGs) have great potential in the field of flexible wearable devices for tracking human gestures. Due to different human sizes, different wearable tensions inevitably cause errors when tracking arm joint movements. We have designed a flexible wearable smart sleeve with four heads of FBG and spandex polyurethane fibers (SPFs). SPFs sewn with flexible fabric sleeves convert elbow yawing and wrist pitching and roll into an axial strain of FBG. The measuring system has been developed to deduce personalized sensitivity using a dynamic calibration method. For males and females, dynamic calibration, verification and tracking tests were carried out. From the male’s experimental data, the relative errors between the verification sensitivity and the personalized sensitivity are 1.93%, 5.85% and 7.16%, and the average relative errors between the tracking sensitivity and the personalized sensitivity are 7.09%, 5.58% and 2.52%, respectively. And from the data of the female’s experiment, the relative errors between the verification sensitivity and the personalized sensitivity are 0.25%, 5.0% and 6.75%, and the average relative errors between the tracking sensitivity and the personalized sensitivity are 0.99%, 5.56% and 6.95%, respectively. The experimental data have shown that this wearable smart sleeve and the measuring system work well. The research results can be used to develop FBG sensing systems for monitoring joint movements for different human sizes on-line.