Long-term excessive forces loading from muscles of mastication during mandibular motions may result in disorders of temporomandibular joint (TMJ), myofascial pain, and restriction of jaw opening and closing. Current analysis of mandibular movements is generally conducted with a single opening, protrusive and lateral movements rather than composite motions that the three can be combined arbitrarily. The objective of this study was to construct theoretical equations reflecting the correlation between composite motions and muscle forces, and consequently to analyze the mandibular composite motions and the tensions of muscles of mastication in multiple dimensions. The muscle performances such as strength, power, and endurance of mandibular motions were analyzed and the effective motion range of each muscle was derived. The mandibular composite motion model was simplified by calculating muscle forces. An orthogonal rotation matrix based on muscle forces was established. A 3D printed mandible was used for in vitro simulation of mandibular motions on a robot and measurements of force were conducted. The theoretical model and forces were verified through a trajectory tracing experiment of mandibular motions driven by a 6-axis robot with force/torque sensors. Through the analysis of the mandibular composite motion model, the motion form was obtained and transferred to guide the motions of the robot. The error between the experimental data obtained by the 6-axis force/torque sensors and the theoretical data was within 0.6 N. Our system provides excellent visualization for analyzing the changes of muscle forces and locations during various mandibular movements. It is useful for clinicians to diagnose and formulate treatment for patients who suffer from (temporomandibular joint disorders) TMDs and restrict jaw movements. The system can potentially offer the comparison before and after treatment of TMDs or jaw surgery.