The central framework of a filtered lattice Boltzmann collision operator formulation is to remove hydrodynamic moments that are not supported by the order of isotropy of a given lattice velocity set. Due to the natural moment orthogonality of the Hermite polynomials, the form of a filtered collision operator is obtained directly via truncation of the Hermite expansion. In this paper, we present an extension of the filtered collision operator formulation to enforce Galilean invariance. This is accomplished by representing hydrodynamic moments in the relative reference frame with respect to local fluid velocity. The resulting collision operator has a compact and fully Galilean invariant form, and it can then be exactly expressed in terms of an infinite Hermite expansion. Giving a lattice velocity set of specific order of isotropy, a proper truncation of this expansion can be directly determined. Higher order terms are retained in the truncation if a higher order lattice velocity set is used, so that Galilean invariance is attained asymptotically. The previously known filtered collision operator forms can be seen as a limit of zero fluid velocity.