Tractions exerted by cells on their surroundings play an important role in many biological processes including stem cell differentiation, tumorigenesis, cell migration, cancer metastasis, and angiogenesis. The ability to quantify these tractions is important in understanding and manipulating these processes. Three-dimensional traction force microscopy (3DTFM) provides reliable means of evaluating cellular tractions by first measuring the displacement of fluorescent beads in response to these tractions in the surrounding matrix, and then using this measurement to compute the tractions. However, most applications of 3DTFM assume that the surrounding extra-cellular matrix (ECM) is non-fibrous, despite the fact that in many natural and synthetic environments the ECM contains a significant proportion of fibrous components. Motivated by this, we develop a computational approach for determining tractions, while accounting for the fibrous nature of the ECM. In particular, we make use of a fiber-based constitutive model in which the stress contains contributions from a distribution of nonlinear elastic fibers and a hyperelastic matrix. We solve an inverse problem with the nodal values of the traction vector as unknowns, and minimize the difference between a predicted displacement field, obtained by solving the equations of equilibrium in conjunction with the fiber-based constitutive model, and the measured displacement field at the bead locations. We employ a gradient-based minimization method to solve this problem and determine the gradient efficiently by solving for the appropriate adjoint field. We apply this algorithm to problems with experimentally observed cell geometries and synthetic, albeit realistic, traction fields to gauge its sensitivity to noise, and quantify the impact of using an incorrect constitutive model: the so-called model error. We conclude that the approach is robust to noise, yielding about 10% error in tractions for 5% displacement noise. We also conclude that the impact of model error is significant, where using a nonlinear exponential hyperelastic model instead of the fiber-based model, can lead to more than 100% error in the traction field. These results underline the importance of using appropriate constitutive models in 3DTFM, especially in fibrous ECM constructs.