Many self-propelled objects are large enough to exhibit inertial effects but still suffer from environmental fluctuations. The corresponding basic equations of motion are governed by active Langevin dynamics, which involve inertia, friction, and stochastic noise for both the translational and orientational degrees of freedom coupled via the self-propulsion along the particle orientation. In this paper, we generalize the active Langevin model to time-dependent parameters and explicitly discuss the effect of time-dependent inertia for achiral and chiral particles. Realizations of this situation are manifold, ranging from minirockets (which are self-propelled by burning their own mass), to dust particles in plasma (which lose mass by evaporating material), to walkers with expiring activity. Here we present analytical solutions for several dynamical correlation functions, such as mean-square displacement and orientational and velocity autocorrelation functions. If the parameters exhibit a slow power law in time, we obtain anomalous superdiffusion with a nontrivial dynamical exponent. Finally, we constitute the "Langevin rocket" model by including orientational fluctuations in the traditional Tsiolkovsky rocket equation. We calculate the mean reach of the Langevin rocket and discuss different mass ejection strategies to maximize it. Our results can be tested in experiments on macroscopic robotic or living particles or in self-propelled mesoscopic objects moving in media of low viscosity, such as complex plasma.