Difference frequency generation between broadband visible noncollinear optical parametric amplifier (NOPA) pulses and the fundamental pump laser pulses allows the generation of ultrashort infrared pulses with passively stabilized carrier-envelope phase. A simple prism compressor for the visible NOPA pulses is sufficient to generate few-cycle pulses in the infrared and no additional compression is needed. We theoretically investigate the concept, explain the principles, and demonstrate it for high repetition rate, long pulse durations, and various wavelengths by applying it to a Ti:sapphire and an Yb:KYW-based laser systems. For the latter sub-15 fs phase stable pulses around 1.8 μm with an energy of 100 nJ are obtained at 100 kHz repetition rate.