The machine layout problem (MLP) is to determine the optimal location arrangement of machines in a production system, usually in order to minimize the total material transmission distance. This article considers a type of MLP in an automated manufacturing system with unequal-area machines, machine replicas, and taking multi-row multi-column layout structure. The aim is to determine the near-optimal machine layout and orbit layout simultaneously. We model the problem as a mixed-integer linear program by introducing the concept of orbit-based relative locations and using auxiliary binary variables, including relative location assignment variables, in-block relationship variables, and material routing variables. We further exploit the optimal layout patterns, based on which, when the relative locations of the machines and orbits are given, we can quickly derive the near-optimal absolute machine and orbit positions without solving a mixed-integer linear program. A two-layer hybrid heuristic method is proposed to solve the problem: the outer layer is a variable neighborhood search metaheuristic used to explore the solution space of the relative locations, while the inner layer is a recursive heuristic based on the optimal layout patterns. Experiments on instances of different sizes are conducted, and the effectiveness and efficiency of the proposed algorithms are verified. Note to Practitioners—Many manufacturing systems, for example, those in the semiconductor industry, use automated guided vehicles running on orbits to transfer materials between machines. The design of such a manufacturing system consists of determining the numbers of vertical and horizontal orbits, their positions, and the positions of all machines. It is not unusual for there to be hundreds of machines. Furthermore, machines of the same type can be deployed non-adjacently in an automated transfer context since materials can easily be transmitted to an arbitrary machine. The flexibility and complexity of the layout make this layout problem extremely difficult to solve. To the best of our knowledge, no dedicated study has been conducted on this issue. Additionally, state-of-the-art methodologies for related machine layout problems cannot be extended to this problem, partly because one needs to consider the material routing problem explicitly on the orbit network. In this study, we reveal the optimal layout patterns given the relative locations of machines and orbits, and we develop a recursive heuristic that is thousands of times faster than solving the corresponding mixed-integer linear program. Our proposed two-layer hybrid heuristics can solve problems with hundreds of machines, while the inner-layer problem’s precision is guaranteed to within 1%. We believe our algorithm can be used for machine and orbit layout design in flexible manufacturing systems where machine-to-machine material shipment is enables in a rectangular trasmission network.