Beam frames can be found in many application scenarios, ranging from big-scale structure engineering of buildings, bridges to small systems of micromachines, sensors, robotics and metamaterials. However, several challenging issues, e.g. serial-parallel configurations, beam and rigid body coupling along with inconsistent dimension of state variables, often lead to a complicated dynamic modeling. In this paper, we show that the free vibration analysis of a class of beam structures with serial-parallel branches and rigid bodies can be significantly streamlined by modifying the transfer matrix method in a graphic manner. Treating general beam structures as a topological graph with standard building blocks enables a modular modeling process and hence offers a new way to divide the complex modeling issue into much easier steps. The transfer matrix of typical building blocks, such as a beam connected to rigid bodies, tree branches and parallel clamped sub-chains, are derived and standardized providing new perspectives and distinct forms in contrast to previous studies. The transfer matrix method exhibits several advantages of easy programming, accurate solution and small degrees-of-freedom for serial structures. The presented approach extends these characteristics to a range of multi-body systems consisting of rigid bodies and beams. Comparative validation with four application case studies confirms these advantages.