In recent years, fiber-optic shape sensing, which means reconstructing the deformation state of structures from strain information measured by fiber-optic sensors, has attracted interest from many fields. Most of the existing shape-sensing research for flexible three-dimensional (3D) slender structures is based on simple strain-curvature transformation, relying on standardized substrates with a circular cross section and particular forms of fiber layouts. To develop a more general shape-sensing method that can be applied to a wider range of practical engineering conditions, in this paper, the strain–displacement relationship of 3D slender structures under the effect of multiple deformation modes coupling is described by introducing absolute nodal coordinate formulation element, and the deformation reconstruction problem is reorganized into a nonlinear optimization problem that can be applied to large deformation and accommodates different cross-sectional shapes and optical fiber layouts. Due to the rapid increase in the calculation difficulties of nonlinear optimization problems with the number of variables, an element-by-element solving strategy is adopted, and nodal degrees of freedom that have less influence on the overall shape of the structure are merged. In addition, in order to address the issue of local strain anomalies caused by unmodeled factors such as section warping, the pointwise matching between the theoretical strains and measured strains is relaxed to an average matching in subregions to capture the overall deformation, improving the robustness and computational efficiency of the solution process. The accuracy and computational performance of the proposed method are verified through numerical simulation and experiment.