We study mode stability in photonic-crystal surface-emitting lasers (PCSELs) with large coupling-coefficient-length product κ1DL(>6). We observe that mode competition occurs at high current levels above threshold. Our combined experimental and theoretical study provides the first evidence of the mode competition originating from the high-order band-edge modes. The decreased threshold margin between these competing high-order modes and the main lasing mode with increasing cavity length as well as the spatial hole burning effect may deteriorate the single-mode stability. Our finding is essential for designing single-mode high-power PCSELs for which the strategy to suppress the high-order modes must be considered.