Performing soil-structure interaction (SSI) analysis under strong shaking scenarios using the direct method is becoming more prevalent in engineering practice. However, given the pronounced nonlinearity in soil, structure, and foundation response, the robustness of current methodologies is open to question, and benchmarking these methodologies against instrumented case histories is needed to identify the strengths and weaknesses in the approach. In this study, a three-dimensional (3D) Nonlinear (NL) Soil-Structure Interaction (SSI) analysis methodology in the time-domain code LS-DYNA is benchmarked against the actual recordings for a mid-rise building. The well-instrumented Building Research Institute (BRI) Annex Building (ANX) in Tsukuba City northeast of Tokyo, Japan, which features an adjacent free-field geotechnical downhole array, is used for this case study. A Finite Element (FE) three-dimensional 3D SSI model of the site, including the BRI ANX building, that considers both material and geometric nonlinearities is first developed. Two events are applied to the model: the 2011 Tohoku-Oki earthquake main shock (Mw9.1), referred to as the Great East Japan Earthquake (GEJE), and the 2002 Southern Ibaraki Prefecture earthquake (Mw4.9), selected from the BRI strong motion network due to the relatively wide amplitudes recorded at the site. The former event induced maximum shear strains in the soil of 1.6%, while the latter event induced maximum shear strains of 0.08%. For both horizontal and vertical components of shakings, the predicted response of the structure and adjacent free-field downhole array are compared with the corresponding recordings. The results show that the structural response is significantly affected by the accuracy of the predicted free-field ground surface response, especially at the structural fundamental period as opposed to the site period. In addition, results indicate that the fixed-base and flexible-base structural models’ responses are substantially different, mainly attributed to the SSI effects that cannot be captured in the fixed-base model. Finally, a comprehensive discussion on the minimum amount of data required to adequately capture nonlinear structural response using the proposed SSI methodology is provided in this case study.