The presence of young stars in the immediate vicinity and strong tidal field of SgrA* remains unexplained. One currently popular idea for their origin posits that the stars were bused in by an Intermediate-Mass Black Hole (IMBH) which has inspiraled into the Galactic Center a few million years ago. Yu and Tremaine (2003) have argued that in this case some of the old stars in the SgrA* cusp would be ejected by hard gravitational collisions with the IMBH. Here we derive a general expression for the phase-space distribution of the ejected high-velocity stars, given the distribution function of the stars in the cusp. We compute it explicitly for the Peebles-Young distribution function of the cusp, and make a detailed model for the time-dependent ejection of stars during the IMBH inspiral. We find that (1) the stars are ejected in a burst lasting a few dynamical friction timescales; if the ejected stars are detected by Gaia they are likely to be produced by a single inspiral event, (2) if the inspiral is circular than in the beginning of the burst the velocity vectors of the ejected stars cluster around the inspiral plane, but rapidly isotropise as the burst proceeds, (3) if the inspiral is eccentric, then the stars are ejected in a broad jet roughly perpendicular to the Runge-Lenz vector of the IMBH orbit. In a typical cusp the orbit will precess with a period of \sim 10^5 years, and the rate of ejection into our part of the Galaxy (as defined by e.g. the Gaia visibility domain) will be modulated periodically. Gaia, together with the ground-based follow-up observations, will be able to clock many high-velocity stars back to their ejection from the Galactic Center, thus measuring some of the above phenomena. This would provide a clear signature of the IMBH inspiral in the past 10--20 Myr.