A basic heterogeneous parallel Red–Black successive over-relaxation (SOR) implement, the mono-color floating-point scheme, was developed on graphics processing units (GPU) with OpenCL platform. Designed in fine granularity, compact data structure, and stencil function, a concise mapping relationship was created to implicitly describe the complex rules for searching neighbor elements, which could avoid low utilization of GPU in the traditional scheme of Red–Black SOR. The new mono-color floating-point scheme was applied to build fast Semi-Implicit Method for Pressure Linked Equations (SIMPLE) solver with OpenCL and OpenMP on the heterogeneous parallel computing device. Compared with SIMPLE solver in the traditional Red–Black SOR scheme, the new scheme can achieve 1.7 to 1.8 faster accelerative performance on the same GPU. And this scheme can eliminate the complex searching module in mono-color logical scheme and behave better than the mono-color logical scheme by 20–30% acceleration. Numerical cases in double precision showed that SIMPLE solver on GPU with new scheme of Red–Black SOR could save up to 92% computing time compared with the serial solver on CPU.