As the number of antennas increases, the massive multiple-input multiple-output (MIMO) system can precisely point to user equipments (UEs) with narrow beams. Accurate and timely channel state information (CSI) feedback is crucial to keep UEs in service. Mobile UEs, however, may suffer from the narrow beam nature of the massive MIMO system since UEs can move out of the beam coverage. When intelligent reflecting surface (IRS) is applied to the massive MIMO system, the adjustment of the IRS cannot be frequent as the IRS is controlled remotely by the base station (BS). Limiting the number of CSI feedback and the number of both BS and IRS adjustments significantly reduces the overhead of transmission and computation to the system. In this paper, we consider the UEs’ mobility adaptation problem in an IRS assisted multiuser massive MIMO downlink system with infrequent CSI feedback. We propose a beam control method that adapts to UEs’ mobility. The problem is constructed as a sum rate problem where both the BS and IRS are taken into account to jointly optimize the beamforming matrices. Simulation results show that our proposed algorithm can converge quickly and provide satisfactory performance for mobile UEs. At the same time, our proposed algorithm reduces the frequency of updating the beamforming matrices effectively both at the BS and at the IRS.