The analysis and modeling of high-resolution spectra of nonrigid molecules require a specific Hamiltonian and group-theoretical formulation that differs significantly from that of more familiar rigid systems. Within the framework of Hougen-Bunker-Johns (HBJ) theory, this paper is devoted to the construction of a nonrigid Hamiltonian based on a suitable combination of numerical calculations for the nonrigid part in conjunction with the irreducible tensor operator method for the rigid part. For the first time, a variational calculation from abinitio potential energy surfaces is performed using the HBJ kinetic energy operator built from vibrational, large-amplitude motion, and rotational tensor operators expressed in terms of curvilinear and normal coordinates. Group theory for nonrigid molecules plays a central role in the characterization of the overall tunneling splittings and is discussed in the present approach. The construction of the dipole moment operator is also examined. Validation tests consisting of a careful convergence study of the energy levels as well as a comparison of results obtained from independent computer codes are given for the nonrigid molecules CH2, CH3, NH3, and H2O2. This work paves the way for the modeling of high-resolution spectra of larger nonrigid systems.