Spinning triangle is a critical region in the spinning process of staple yarn, which geometry influences the mechanical performances of fibers in the spinning triangle and determines the physical performance of spun yarns directly. Taking appropriate measures to reduce the spinning triangle and improve the qualities of yarn has attracted great interesting recently. In these modified ring spinning systems, the fibers in the spinning triangle are not arranged in the same plane, but arranged in concentric circular cones at front nip line, such as compact spinning, in which a fiber-converging device is installed on ring spinning frame, and the fiber bundle can be condensed before twisting in order to reduce the spinning triangle. Therefore, in this paper, a new theoretical model of fiber tension distribution in the spinning triangle with fiber concentric circular cones arrangement at front nip line is proposed based on the principle of minimum potential energy firstly. Then, corresponding residual torque within a yarn caused by the fiber tension is given. Finally, as an application of the proposed method, taking the spinning triangle in Ne40 cotton yarn 3-line rollers compact spinning as an example, the fiber tension distributions in the spinning triangle both with and without fiber buckling and corresponding residual torque within a yarn are numerically simulated, and the results are compared with those from earlier model. Furthermore, the properties of spun yarns produced by compact spinning system and ring spinning system are evaluated and analyzed.