A machine learning (ML) method was used to optimize the trap distribution of the charge trap nitride (CTN) to simultaneously improve its performance/reliability (P/R) characteristics, which are tradeoffs in 3-D NAND flash memories. Using an artificial neural network (ANN), we modeled the relationship between trap distributions and P/R characteristics. The ANN was trained using a large experimentally-calibrated technology computer-aided design (TCAD) simulation dataset. The gradient descent method was adapted to optimize the trap distribution, achieving the best P/R characteristics based on the well-trained ANN. Eventually, we found the best trap profile distributed in both space and energy. In particular, the energetic trap distribution had a larger impact on the P/R characteristics than that of the spatial trap distribution. Furthermore, in terms of the P/R characteristics, it was generally preferable to increase all inputs of the energetic trap distribution. However, the acceptor-like trap energy level ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$E_{TA}$ </tex-math></inline-formula> ) and its standard deviation ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\sigma _{EA}$ </tex-math></inline-formula> ) caused a tradeoff between P/R characteristics; therefore, ML was used to determine their optimal points. The proposed ML method allows the optimization of trap distribution to obtain the best P/R characteristics rapidly and quantitatively. Our findings could be used as a guideline for determining the physical properties of CTN in 3-D NAND flash cells.