We consider $N$-soliton solutions of the KP equation, (-4u_t+u_{xxx}+6uu_x)_x+3u_{yy}=0 . An $N$-soliton solution is a solution $u(x,y,t)$ which has the same set of $N$ line soliton solutions in both asymptotics $y\to\infty$ and $y\to -\infty$. The $N$-soliton solutions include all possible resonant interactions among those line solitons. We then classify those $N$-soliton solutions by defining a pair of $N$-numbers $({\bf n}^+,{\bf n}^-)$ with ${\bf n}^{\pm}=(n_1^{\pm},...,n_N^{\pm}), n_j^{\pm}\in\{1,...,2N\}$, which labels $N$ line solitons in the solution. The classification is related to the Schubert decomposition of the Grassmann manifolds Gr$(N,2N)$, where the solution of the KP equation is defined as a torus orbit. Then the interaction pattern of $N$-soliton solution can be described by the pair of Young diagrams associated with $({\bf n}^+,{\bf n}^-)$. We also show that $N$-soliton solutions of the KdV equation obtained by the constraint $\partial u/\partial y=0$ cannot have resonant interaction.