It is shown that polarization sensitive (optically heterodyned) detection combined with π/2 optical phase biasing between the optical Kerr gate signal and the local oscillator signal provides an effective method for the investigation of dynamics of third-order nonlinear optical processes. The method permits selective enhancement of either the real or the imaginary component of the nonlinear optical response of the medium under study. Both the magnitudes and the signs of the real and the imaginary components of the complex third-order optical susceptibility, contributing to the Kerr gate phenomenon, can thus be determined independently. The method is used to investigate the third-order nonlinear optical response of solutions of canthaxanthin (4,4′-dioxo-β-carotene) in tetrahydrofuran. With the use of ultrafast 60 fs laser pulses at 620 nm, the instantaneous, coherent part of the response is separated from an incoherent, time-delayed response. The real and the imaginary components of the instantaneous complex second hyperpolarizability γ are derived. The analysis of the delayed part of the signal suggests that it originates in an effective χ(5) nonlinear process due to the change of linear susceptibility after population of a two-photon excited state. However, an unusual feature observed is that the two-photon pumped excited state does not produce a significant change of linear susceptibility, but it is a lower state, most probably the 2 1Ag state, subsequently populated by nonradiative relaxation which produces a dominant change in linear susceptibility responsible for the incoherent nonlinear optical response. From a theoretical fit of the temporal behavior of the delayed nonlinear response, the rates of nonradiative population and subsequent decay of the 2 1Ag state are derived. These rates are in qualitative agreement with published spectroscopic results for this class of compounds.