Ultrafast all-optical effects in quadratically nonlinear media are studied, with particular emphasis being given to the effect of group-velocity mismatch of the optical pulses at different carrier frequencies. The basic consequences of the resulting pulse walk-off for fundamental all-optical concepts based on quadratic nonlinearities, such as phase-insensitive amplification without and with cascading, intensity- and phase-controlled polarization rotation, and switching relying on phase modulation, are studied in detail. We derive optimal parameters for the application of these concepts in all-optical devices.