Scan chain-based attacks are side-channel attacks focusing on one of the most significant features of hardware test circuitry. A technique called Design for Testability (DfT) involves integrating certain testability components into a hardware design. However, this creates a side channel for cryptanalysis, providing crypto devices vulnerable to scan-based attacks. Advanced Encryption Standard (AES) has been proven as the most powerful and secure symmetric encryption algorithm announced by USA Government and it outperforms all other existing cryptographic algorithms. Furthermore, the on-chip implementation of private key algorithms like AES has faced scan-based side-channel attacks. With the aim of protecting the data for secure communication, a new hybrid pipelined AES algorithm with enhanced security features is implemented. This paper proposes testing an AES core with unpredictable response compaction and bit level-masking throughout the scan chain process. A bit-level scan flipflop focused on masking as a scan protection solution for secure testing. The experimental results show that the best security is provided by the randomized addition of masked scan flipflop through the scan chain and also provides minimal design difficulty and power expansion overhead with some negligible delay measures. Thus, the proposed technique outperforms the state-of-the-art LUT-based S-box and the composite sub-byte transformation model regarding throughput rate 2 times and 15 times respectively. And security measured in the avalanche effect for the sub-pipelined model has been increased up to 95 per cent with reduced computational complexity. Also, the proposed sub-pipelined S-box utilizing a composite field arithmetic scheme achieves 7 per cent area effectiveness and 2.5 times the hardware complexity compared to the LUT-based model.