This letter studies the massive multiple-input multiple-output (MIMO) analog uncoded over-the-air computation (AirComp), which is a promising solution in Internet-of-Things (IoT) networks for fusion centers (FCs) to compute functions of data distributed in a large number of wireless devices (WDs). To avoid the heavy overhead of the perfect real-time channel state information (CSI) estimation, we develop a two-timescale hybrid beamforming (THB) scheme in AirComp. Under this setup, we aim to minimize the transmit power over time, while ensuring a minimum required computation distortion in terms of the mean-squared error (MSE). We propose a constrained stochastic successive convex approximation (CSSCA) based algorithm to solve the formulated stochastic non-convex optimization problem for the THB design. Specifically, the short-timescale receive digital beamforming matrix at the FC is optimized at each time slot based on the real-time effective low-dimensional CSI matrices by a sum-minimization-mean-squared-error (sum-MMSE) receiver, which can better deal with the signal misalignment. By contrast, the long-timescale transmit beamforming matrix at WDs and the receive analog beamforming matrix at the FC are designed based on the available channel statistics and updated in a frame-based manner, where a frame contains a fixed number of time slots. Numerical results show that the proposed THB design algorithm achieves similar transmit power performance as that achieved by the design when the perfect real-time CSI is available.