We analyze the phase diffusion, quantum fluctuations and their spectral features of a one-dimensional Bose-Josephson junction (BJJ) nonlinearly coupled to a bosonic heat bath. The phase diffusion is considered by taking into account of random modulations of the BJJ modes causing a phase loss of initial coherence between the ground and excited states, whereby the frequency modulation is incorporated in the system-reservoir Hamiltonian by an interaction term linear in bath operators but nonlinear in system (BJJ) operators. We examine the dependence of the phase diffusion coefficient on the on-site interaction and temperature in the zero- and π-phase modes and demonstrate its phase transition-like behavior between the Josephson oscillation and the macroscopic quantum self-trapping (MQST) regimes in the π-phase mode. Based on the thermal canonical Wigner distribution, which is the equilibrium solution of the associated quantum Langevin equationfor phase, coherence factor is calculated to study phase diffusion for the zero- and π-phase modes. We investigate the quantum fluctuations of the relative phase and population imbalance in terms of fluctuation spectra which capture an interesting shift in Josephson frequency induced by frequency fluctuation due to nonlinear system-reservoir coupling, as well as the on-site interaction-induced splitting in the weak dissipative regime.