Zeroing neural network (ZNN), an effective method for tracking solutions of dynamic equations, has been developed and improved by various strategies, typically the application of nonlinear activation functions (AFs) and varying parameters (VPs). Unlike VPs, AFs applied in ZNN models act directly on real-time error. The processing unit of v needs to obtain neural state in real time. In the implementation process, highly nonlinear AFs become an important cause of time delays, which eventually leads to instability and oscillation. However, most studies focus on exploring new theoretically valid AFs to improve performance of ZNNs, while ignoring the adverse effects of highly nonlinear AFs. The nonlinearity of AFs requires us fully consider time-delay tolerance of ZNNs using nonlinear AFs, so as to ensure that the model is not unstable even when disturbed by time delays. In this work, delay-perturbed generalized ZNN (DP-GZNN) is proposed to investigate time-delay tolerance of generalized ZNN (G-ZNN) in solving dynamic Lyapunov equation. Considering the nonlinearity of AFs, two delay terms are elegantly added to G-ZNN and DP-GZNN is then derived. After rigorous mathematical derivations, sufficient conditions in a linear matrix inequality (LMI) manner are presented for global convergence of DP-GZNN. Through rich numerical experiments, hyperparameters involved in the analysis process are discussed in detail. Comparative simulations are also conducted to compare the ability of different ZNN models to resist time delays. It is worth to mention that this is the first time to consider the ability of G-ZNN to resist discrete and distributed time delays.