One goal of neurobiology is to explain how decision-making in neuromuscular circuits produces behaviors. However, two obstacles complicate such efforts: individual behavioral variability and the challenge of simultaneously assessing multiple neuronal activities during behavior. Here, we circumvent these obstacles by analyzing whole animal behavior from a library of Caenorhabditis elegans male mating recordings. The copulating males express the GCaMP calcium sensor in the muscles, allowing simultaneous recording of posture and muscle activities. Our library contains wild type and males with selective neuronal desensitization in serotonergic neurons, which include male-specific posterior cord motor/interneurons and sensory ray neurons that modulate mating behavior. Incorporating deep learning-enabled computer vision, we developed a software to automatically quantify posture and muscle activities. By modeling, the posture and muscle activity data are classified into stereotyped modules, with the behaviors represented by serial executions and transitions among the modules. Detailed analysis of the modules reveals previously unidentified subtypes of the male's copulatory spicule prodding behavior. We find that wild-type and serotonergic neurons-suppressed males had different usage preferences for those module subtypes, highlighting the requirement of serotonergic neurons in the coordinated function of some muscles. In the structure of the behavior, bi-module repeats coincide with most of the previously described copulation steps, suggesting a recursive "repeat until success/give up" program is used for each step during mating. On the other hand, the transition orders of the bi-module repeats reveal the sub-behavioral hierarchy males employ to locate and inseminate hermaphrodites.