The Pancharatnam phase for displaced number states of the harmonic oscillator is discussed. In particular, it is examined how a single quantum oscillator, driven by a suitable transient external force, evolves form the initial eigenstate um(x) to the final, displaced number state modified with a suitable phase factor. The significance of this, usually neglected, phase factor for the solution of the relevant time-dependent Schrodinger equation and for the geometric phase accumulated in the wavefunction during the time evolution of the system is examined. The general expression for the geometric phase for a noncyclic evolution from an initial displaced number state at time t1 to the final state at time t2 is derived and two applications, that of delta and harmonic forcing, are worked out. The special case of cyclic evolution is subsequently discussed and, in particular, the conditions leading to such an evolution are derived. The relationship to the classical notion of cyclic evolution is also examined in some detail and it is demonstrated that the general expression for the Pancharatnam phase reduces to the corresponding Berry phase.