This article is concerned with passivity analysis of neural networks with a time-varying delay. Several techniques in the domain are improved to establish the new passivity criterion with less conservatism. First, a Lyapunov-Krasovskii functional (LKF) is constructed with two general delay-product-type terms which contain any chosen degree of polynomials in time-varying delay. Second, a general convexity lemma without conservatism is developed to address the positive-definiteness of the LKF and the negative-definiteness of its time-derivative. Then, with these improved results, a hierarchical passivity criterion of less conservatism is obtained for neural networks with a time-varying delay, whose size and conservatism vary with the maximal degree of the time-varying delay polynomial in the LKF. It is shown that the conservatism of the passivity criterion does not always reduce as the degree of the time-varying delay polynomial increases. Finally, a numerical example is given to illustrate the proposed criterion and benchmark against the existing results.