In many systems of interest, most of the structure is well approximated as linear but some parts must be treated as nonlinear to get accurate response predictions: significant nonlinear effects are due to the connections between coupled subsystems, such as in automotive or aerospace structures. The present work aims at predicting the nonlinear behavior of coupled systems using a substructuring technique in the modal domain. This study focuses on the effects of nonlinear connections on the dynamics of an assembly in which the coupled subsystems can be considered as linear. Each connection is instead considered as a quasi-linear substructure with stiffness that is function of amplitude or energy. The iterative procedure used here is enhanced with respect to previous works by enforcing a better control of the total energy at each iteration allowing to obtain the solution for a prescribed set of energy levels. Also, the initial guess and the convergence criterion have been modified to speed up the procedure. This technique is applied to a system made of two continuous linear subsystems coupled by nonlinear connections. The numerical results of the coupling are first compared to the ones obtained by using the Harmonic Balance technique on the model of the complete assembly to evaluate its effectiveness and understand the effects of modal truncation. Besides, a nonlinear connecting element, specifically designed in order to have a nearly cubic hardening behavior, is used in an experimental setup. Substructuring results are compared to experimental results measured on the assembled system, in order to evaluate the correlation between mode shapes and the accuracy in the resonance frequency at several excitation levels.