Location fingerprinting using Received Signal Strength Indicators (RSSIs) has become a popular technique for indoor localization due to its use of existing Wi-Fi infrastructure and Wi-Fi-enabled devices. Artificial intelligence/machine learning techniques such as Deep Neural Networks (DNNs) have been adopted to make location fingerprinting more accurate and reliable for large-scale indoor localization applications. However, the success of DNNs for indoor localization depends on the availability of a large amount of pre-processed and labeled data for training, the collection of which could be time-consuming in large-scale indoor environments and even challenging during a pandemic situation like COVID-19. To address these issues in data collection, we investigate multi-dimensional RSSI data augmentation based on the Multi-Output Gaussian Process (MOGP), which, unlike the Single-Output Gaussian Process (SOGP), can exploit the correlation among the RSSIs from multiple access points in a single floor, neighboring floors, or a single building by collectively processing them. The feasibility of MOGP-based multi-dimensional RSSI data augmentation is demonstrated through experiments using the hierarchical indoor localization model based on a Recurrent Neural Network (RNN)—i.e., one of the state-of-the-art multi-building and multi-floor localization models—and the publicly available UJIIndoorLoc multi-building and multi-floor indoor localization database. The RNN model trained with the UJIIndoorLoc database augmented with the augmentation mode of “by a single building”, where an MOGP model is fitted based on the entire RSSI data of a building, outperforms the other two augmentation modes and results in the three-dimensional localization error of 8.42 m.