Nesting control is one of the most prevalent quantity-based controls for revenue management (RM) problem. This type of control sets nested booking limits to avoid the situation in which high-fare bookings are rejected in favor of low-fare class. There are two nesting policies that are commonly adopted in practical application, namely standard nesting policy (SNP) and theft nesting policy (TNP). Existing research usually chooses one of them and then studies how to optimize the booking limits under the selected policy. In this paper, we newly introduce a generalized nesting policy (GNP) that can enrich the family of nesting policies. It is certified that the traditional SNP and TNP are special cases of GNP. A mathematical model for the nesting control under GNP is proposed, in which the nesting policy and booking limits are both taken as the decision variables. Followed, a simulation-based optimization algorithm integrated of simulated annealing (SA) algorithm and finite difference (FD) algorithm is designed to explore an improved solution to the proposed model. In the integrated algorithm, SA is used to search a better nesting policy and FD is to further optimize the booking limits under the current policy. Numerical experiments are conducted on a three-leg airline network with customer choice behavior and they mainly show three aspects of findings. First, the solutions obtained by simulation-based algorithms make significant improvements of revenues compared to the popular EMSR heuristics. Second, on average, GNP outperforms traditional nesting policies SNP and TNP. Third, in most of arrival patterns, GNP has no clear advantage over SNP and TNP, but in about 18% of arrival patterns, some new nesting policies embedded in GNP make significant improvements.