We study in three dimensions the problem of a spatially homogeneous zero-temperature ideal Fermi gas of spin-polarized particles of mass $m$ perturbed by the presence of a single distinguishable impurity of mass $M$. The interaction between the impurity and the fermions involves only the partial $s$ wave through the scattering length $a$ and has negligible range $b$ compared to the inverse Fermi wave number $1/{k}_{\mathrm{F}}$ of the gas. Through the interactions with the Fermi gas the impurity gives birth to a quasiparticle, which will be here a Fermi polaron (or more precisely a monomeron). We consider the general case of an impurity moving with wave vector $\mathbf{K}\ensuremath{\ne}\mathbf{0}$: Then the quasiparticle acquires a finite lifetime in its initial momentum channel because it can radiate particle-hole pairs in the Fermi sea. A description of the system using a variational approach, based on a finite number of particle-hole excitations of the Fermi sea, then becomes inappropriate around $\mathbf{K}=\mathbf{0}$. We rely thus upon perturbation theory, where the small and negative parameter ${k}_{\mathrm{F}}a\ensuremath{\rightarrow}{0}^{\ensuremath{-}}$ excludes any branches other than the monomeronic one in the ground state (as, e.g., the dimeronic one), and allows us a systematic study of the system. We calculate the impurity self-energy ${\ensuremath{\Sigma}}^{(2)}(\mathbf{K},\ensuremath{\omega})$ up to second order included in $a$. Remarkably, we obtain an analytical explicit expression for ${\ensuremath{\Sigma}}^{(2)}(\mathbf{K},\ensuremath{\omega})$, allowing us to study its derivatives in the plane $(K,\ensuremath{\omega})$. These present interesting singularities, which in general appear in the third-order derivatives ${\ensuremath{\partial}}^{3}{\ensuremath{\Sigma}}^{(2)}(\mathbf{K},\ensuremath{\omega})$. In the special case of equal masses, $M=m$, singularities appear already in the physically more accessible second-order derivatives ${\ensuremath{\partial}}^{2}{\ensuremath{\Sigma}}^{(2)}(\mathbf{K},\ensuremath{\omega})$; using a self-consistent heuristic approach based on ${\ensuremath{\Sigma}}^{(2)}$ we then regularize the divergence of the second-order derivative ${\ensuremath{\partial}}_{K}^{2}\ensuremath{\Delta}E(\mathbf{K})$ of the complex energy of the quasiparticle found in Trefzger and Castin [Europhys. Lett. 104, 50005 (2013)] at $K={k}_{\mathrm{F}}$, and we predict an interesting scaling law in the neighborhood of $K={k}_{\mathrm{F}}$. As a by product of our theory we have access to all moments of the momentum of the particle-hole pair emitted by the impurity while damping its motion in the Fermi sea at the level of Fermi's golden rule.