In this paper, we report a series of transformations for the construction of a Hamiltonian model for nonrigid polyatomic molecules in the framework of the Hougen-Bunker-Johns formalism (HBJ). This model is expressed in normal mode coordinates for small vibrations and in a specific coordinate ρ to describe the large amplitude motion. For the first time, a general procedure linking the "true" curvilinear coordinates to ρ is proposed, allowing the expression of the potential energy part in the same coordinate representation as the kinetic energy operator, whatever the number of atoms. A Lie group-based method is also proposed for the derivation of the reference configuration in the internal axis system. This work opens new perspectives for future high-resolution spectroscopy studies of nonrigid, medium-sized molecules using HBJ-type Hamiltonians. Illustrative examples and computation of vibrational energy levels on semirigid and nonrigid molecules are given to validate this method.