The energy transport in aero-acoustics is investigated in the Lagrangian frame. First, based on finite-time Lyapunov exponent (FTLE) and momentum potential theory, a Lagrangian approach is proposed to identify transport barriers of acoustic energy. Specifically, the method, named relative flux gradient (RFG), is presented in detail. Then, to verify the method, it is applied to analytical fields, showing that it could reveal the wavefronts and energy transport barriers depending on the time interval of computation. Moreover, RFG is applied to analyze a simulated flow field of an open cavity flow, and the results are compared with the Lagrangian coherent structures identified by FTLE, demonstrating great similarity. Importantly, the differences between the structures are further analyzed, illustrating several transport channels that correspond to the Rossiter mode and showing a complex interaction between acoustic and vorticity modes. Finally, the relationship between the identified transport barriers and the acoustic behaviors in Eulerian frame is studied in detail. The results show that the transport barriers identified by RFG significantly impact the orbits in phase space, and in particular, RFG has the potential to illustrate and analyze the transport of acoustic energy in complex flow fields in a quantitative way: one method for direct analysis of acoustic phenomena in complex flow regions.