Cell-generated tractions play an important role in various physiological and pathological processes such as stem-cell differentiation, cell migration, wound healing, and cancer metastasis. Traction force microscopy (TFM) is a technique for quantifying cellular tractions during cell-matrix interactions. Most applications of this technique have heretofore assumed that the matrix surrounding the cells is linear elastic and undergoes infinitesimal strains, but recent experiments have shown that the traction-induced strains can be large (e.g., more than 50%). In this paper, we propose a novel three-dimensional (3D) TFM approach that consistently accounts for both the geometric nonlinearity introduced by large strains in the matrix, and the material nonlinearity due to strain-stiffening of the matrix. In particular, we pose the TFM problem as a nonlinear inverse hyperelasticity problem in the stressed configuration of the matrix, with the objective of determining the cellular tractions that are consistent with the measured displacement field in the matrix. We formulate the inverse problem as a constrained minimization problem and develop an efficient adjoint-based minimization procedure to solve it. We first validate our approach using simulated data, and quantify its sensitivity to noise. We then employ the new approach to recover tractions exerted by NIH 3T3 cells fully encapsulated in hydrogel matrices of varying stiffness. We find that neglecting nonlinear effects can induce significant errors in traction reconstructions. We also find that cellular tractions roughly increase with gel stiffness, while the strain energy appears to saturate.