The rise of shared bicycles has increased the demand for group riding, integrating bicycles into social groups. Additionally, retrograde riding, where cyclists travel against the designated direction, is a common behavior observed in bicycle flows. The interaction and self-organization phenomenon of group and retrograde behaviors are complex, significantly impacting traffic efficiency. This paper develops a two-dimensional Extended Moore Neighborhood and constructs state-updating rules for regular riding, group riding and retrograde riding. Each rule comprises a psychological decision layer and a physical execution layer, forming a cellular automaton model for group and retrograde bicycles. Field experiments are conducted to calibrate the model parameters and verify the behavioral characteristics. Finally, we execute numerical simulations at a signalized intersection to explore the coupling effects of group and retrograde behaviors on self-organization within the bicycle flow and the traffic capacity. The results indicate that group behavior increases queue length while reducing start wave speed and expansion degree. Retrograde behavior intensifies the negative effects on bicycle flow. These findings provide insights for managing both forward and retrograde bicycle flows.