Conducting Compact-Tension-Shear (CTS) tests emerges as a promising research methodology to determine the fracture toughness of orthotropic materials in mixed-mode I&II. However, the absence of pertinent fracture parameter solutions incorporating material orthotropy impedes the further application and standardization of CTS testing. Therefore, this study performed a systematic two-dimensional finite element analysis (FEA) on orthotropic CTS specimens to evaluate critical fracture parameters, including the normalized stress intensity factors (YI, YII) and compliance (u). First, the impacts of three commonly cited boundary conditions (BCs) of CTS models were investigated, revealing that the simplified or equivalent methods developed for isotropic CTS models are no longer applicable. Subsequently, the suitable CTS models were used to analysis the coupling effects of material orthotropy (λ, ρ), crack length ratios (a/W), and loading angles (β). Findings indicated that YI or u no longer exhibit a monotonic relationship with respect to a/W or β under strong orthotropy. The impact of λ and ρ on fracture parameters is equally significant across different geometries and loading conditions. Further, the impact of orthotropy on YII is relatively small, while the more affected YI may have a difference compared to isotropic results of over 80%. Finally, CTS tests were carried out on the unidirectional carbon fiber-reinforced epoxy resin composite measuring the fracture toughness in mixed-modes I&II. The obtained fracture parameter solutions were applied and validated during the process. The acquired results are poised to contribute to the advancement of fracture toughness testing standardization for orthotropic materials under mixed-mode loading.