Vibration isolators with both stiffness and damping nonlinearities show promise for exhibiting compound advantages for broadband vibration isolation. A nonlinear isolator with a NiTiNOL wire rope is proposed with cubic stiffness, hysteretic damping, and pinching effects induced by geometric constraints, inner frictions, and phase transitions, respectively. A combined method of a beam constraint model and a Bouc-Wen model is presented to characterize the restoring force of the NiTiNOL wire rope. The frequency responses of the nonlinear isolator were analyzed through a harmonic balance method with an alternating frequency/time domain technique. The generalized equivalent stiffness and the generalized equivalent damping ratio were defined for a comprehensive understanding of the nonlinear characteristics. The isolator exhibited a stiffness-softening-hardening characteristic. The pinching effect, the Bouc-Wen hysteresis, and the cubic stiffness mainly influenced the equivalent stiffness at the initial value, the small displacements, and the large displacements, respectively. The rate-independent damping ratio increased and then decreased with increasing displacement, and the parameters influenced the damping ratio change in different ways. Compared to an isolator with a steel wire rope, the isolator with a NiTiNOL wire rope exhibited less initial stiffness and a stronger damping effect, and thus, better vibration isolation performance. The relationships of the peak displacement transmissibility and the resonant frequency with the excitation amplitude were both non-monotonic due to the non-monotonic changes of the stiffness and the damping ratio. The minimum peak transmissibility, the lowest resonant frequency, and their corresponding excitation amplitudes depended on the isolator parameters. The isolator demonstrated stiffness–softening and stiffness–hardening types of jump phenomena with different parameters.