AbstractThis paper focuses on the multi‐pair amplify‐and‐forward massive multiple‐input multiple‐output (MIMO) relay enabled two‐tier heterogeneous cellular networks in terms of aggregate spectral efficiency (SE), where the macro cells are overlaid with dense small cells. Specially, the maximum ratio combining/maximal ratio transmission and zero‐forcing processing schemes are employed at massive MIMO relay, respectively. The macro base stations located in the center of the cells are equipped with massive MIMO antenna array, while the randomly distributed macro users (MUs), small cell base stations, and small cell users are equipped with single antenna. The positions of MUs, small cell users and small cell base stations are modelled as independent Poisson point processes. For such two‐tier heterogeneous cellular networks, to obtain the aggregate SE in target cell, we first formulate the end‐to‐end signal‐to‐interference noise ratios for a communication pair as well as the expression of power amplification factor at target macro base stations so that the total spectrum efficiency per cell is formulated. Then, by employing stochastic geometry, the closed‐form expressions of the power amplification factor and the expectation of the reciprocal of the equivalent end‐to‐end signal‐to‐interference noise ratios are achieved. With these results, we derive the lower bounds of SEs for maximum ratio combining/maximal ratio transmission and zero‐forcing schemes. The derivations display a clear perspective about the impact of network parameters on the system‐level performance, which includes macro cell radius, small cell radius, macro cell guard radius, the mean number of MUs per cell and so on. To achieve optimal (or suboptimal) system performance, we find that the radius of small cell should be less than the guard radius. For the guard radius, there exists an upper bound, only under which the increase of the guard radius would contribute to the improvement of SE. At the same time, our results show that with given network parameters, especially the guard radius and the small cell radius, there exists a unique greatest lower bound in terms of macro cell radius. When the macro cell radius is less than the greatest lower bound, the aggregate SE is increasing with the macro cell radius. On the contrary, the achievable aggregate SE saturates to the corresponding constant, approximately. In addition, the greatest lower bound of the macro cell radius is the function of the guard radius and the small cell radius. Copyright © 2016 John Wiley & Sons, Ltd.