The Level-Set Method (LSM) with the smooth and distinct description of structural boundaries offers more superior benefits for topology optimization. However, the classic Finite Element Method (FEM) and ersatz material method in the LSM cause numerical deficiencies in the optimization, such as iterative instability and the crispness loss of boundaries. This work focuses on developing an Isogeometric Topology and Shape Optimization (ITSO) method for the design of composite structures, in which a Non-Uniform Rational B-splines (NURBS)-based Multi-phase Level-Sets (NM-LS) model is constructed to present the distribution of multiple materials. The NM-LS model consists of an immersed representation using level-sets, NURBS parameterized level-sets and its development. Isogeometric Analysis (IGA) with adaptive Gauss quadrature method are applied to solve unknown structural responses, which can effectively remove several numerical artifacts resulting from the linear interpolation of ersatz material model in both design and analysis. Three models of structural geometry, numerical analysis and topology description can be integrated by the same NURBS basis functions, which can effectively improve numerical precision and iterative stability. Finally, several 2D and 3D numerical examples are addressed to demonstrate the effectiveness and efficiency of the proposed ITSO method on the design of composite structures.