In fringe projection profilometry based on temporal phase unwrapping, determining a fringe order map commonly requires a large number of fringes. To reduce the fringe number, this paper proposes a concise absolute phase retrieval algorithm just by projecting four fringes. The first two orthogonal fringes with relatively large frequency can collect reliable height information. The second two fringes are designed the same as the first two, but the only difference is that each 2π-phase of them is shifted by a unique amount, which can robustly label a large number of fringe orders. For decoding the fringes, we develop an average intensity one-time extraction algorithm, which allows for the rapid acquisition of the two pairs of alternating current components. From this, the wrapped phase containing height information and the stair-coded phase providing fringe orders can be directly extracted by arctangent operation in a point-to-point manner. Furthermore, we also develop a universal fringe order correction algorithm that can simultaneously correct the common errors and the misalignment between the wrapped phase and fringe orders. Experiment results demonstrate that this method achieves comparable accuracy and adaptability to the phase-coding method, while utilizing two fewer fringes.