Abstract The ever-increasing demand for bandwidth in optical networks necessitates efficient spectrum utilization. To address this challenge, this paper proposes a novel hybrid spectrum sensing algorithm tailored explicitly for 256-QAM (Quadrature Amplitude Modulation) optical communication waveforms. The proposed algorithm combines the strengths of energy detection and cyclostationary feature detection to overcome the limitations of individual methods. Energy detection (ED) provides fast and low-complexity sensing, while cyclostationary feature detection offers higher accuracy and sensitivity. First, ED is employed for rapid initial spectrum assessment. Subsequently, matched filter (MF) detection is selectively applied only to frequency bands identified as potentially occupied by primary users based on the energy detection results. This selective approach significantly reduces computational complexity while maintaining high detection accuracy. The results demonstrate significant improvements in detection accuracy, sensitivity, and computational efficiency compared to existing methods. In particular, the hybrid algorithm performs better in scenarios where weak 256-QAM signals coexist with strong primary users, showcasing its effectiveness in dynamic spectrum-sharing applications. This work contributes significantly to optical spectrum sensing by offering an efficient and accurate solution for advanced radio systems. The proposed hybrid algorithm paves the way for improved spectrum utilization and facilitates the development of high-performance, next-generation optical networks. The projected method obtained a gain of −200 as compared with the existing methods.