We investigate carrier and collective mode dynamics in 2H-NbSe$_2$ using time-resolved optical pump-probe spectroscopy and compare the results with first-principle calculations. Broadband ultrafast reflectivity studies of 2H-NbSe$_2$ in a wide temperature interval covering the normal, charge density wave (CDW) and superconducting phase were performed. Spectral features observed in the transient reflectivity experiment were associated with specific optical transitions obtained from band structure calculations. Displacive excitation of coherent phonons showed CDW-associated coherent oscillations of the soft phonon mode across the whole spectral range. Temperature evolution of this coherent phonon mode in the low-excitation linear regime shows softening of the mode down to the CDW transition temperature T$_{CDW}$ with subsequent hardening below T$_{CDW}$. The global fit of the broadband probe data reveals four different relaxation times associated with characteristic electron-electron, electron-phonon and phonon-phonon relaxation processes. From first principle calculations of electron-phonon coupling we associate the few picosecond electron-phonon relaxation time $\tau_2$ with a specific group of phonons with frequencies around 20 meV. On the other hand, the anomalously long relaxation time of $\tau_3$~100 ps is associated with anharmonicity-driven phonon-phonon scattering. All relaxation processes result from anomalies near the second order CDW phase transition that are reflected in the temperature dependencies of the characteristic relaxation times and amplitudes of optical densities. At highest fluences we observe electronic melting of the CDW and disappearance of the mode hardening below T$_{CDW}$.