We report how the complex intra-pulse polarization dynamics of coherent optical wavebreaking and incoherent Raman amplification processes in all-normal dispersion (ANDi) fibers vary for femto and picosecond pump pulses. Using high temporal resolution vector supercontinuum simulations, we identify deterministic polarization dynamics caused by wavebreaking and self-phase modulation for femtosecond pulses and quasi-chaotic polarization evolution driven by Raman amplification of quantum noise for picosecond pulses. In contrast to cross-phase modulation instability, the Raman-based polarization noise has no power threshold and is reduced by aligning the higher energy polarization component with the lower index axis of the fiber. The degree of polarization stability is quantified using new time domain parameters that build on the spectrally averaged degree of coherence used in supercontinuum research to quantify the output spectral stability. We show that the spectral coherence is intrinsically linked to polarization noise, and that the noise will occur in both polarization maintaining (PM) and non-PM fibers, spanning a broad range of pulse energies, durations, and fiber birefringence values. This analysis provides an in-depth understanding of the nonlinear polarization dynamics associated with coherent and incoherent propagation in ANDi fibers.