Protection systems are one of the most critical components in the transmission system and are becoming more digital with ongoing automation. These digital systems are prone to vulnerabilities/attacks, and exploitation of these vulnerabilities may cause major impacts on the electric grid performance. Multiple alarms reported in the control center could be a result of the faults (expected operations) or failures in the protection system (anomalies/ unexpected operation). Situational awareness gained through sensors such as a phasor measurement unit (PMU) and data acquired through the cyber system provide an opportunity to develop continuous cyber-physical monitoring of the system. Note that relay data are not reported in the control center continuously. This paper presents a cyber-physical data analytics based technique to monitor transmission protection system and detect malicious activity. Initially, continuous monitoring of PMU data is utilized for data anomaly detection, which includes bad or missing data using long short-term memory (LSTM). Then, PMU data of interest are utilized for failure diagnosis, using a semisupervised deep autoencoder model. In this research, cyber anomalies are modeled by manipulating the setting/logic design of protective devices, and a ridge regression based classifier with a feature engineering pipeline is used to detect cyber anomalies. The results from the deep autoencoder model and ridge regression based classifier are then utilized for detailed investigation to find the root causes of the observed events assisted by the cyber log data from the protection devices. The algorithm is validated using a real-time simulation of the IEEE test system with industrial hardware relays and PMUs in the loop. Data analytics algorithm running on server utilizes these real-time data continuously for anomaly detection and classification for the developed use cases.