With the development of wave energy conversion technology, more and more wave power generation devices begin to appear. However, the wave energy capture efficiencies of proposed wave energy converters (WECs) are generally low, especially in some areas where wave energy resource is not rich. Instead, a solution of array-type WECs bears practical potential to increase the wave energy capturing capacity. In this paper, a novel floating-array-buoys WEC (FABWEC) system is presented as the basic model and the structure of its energy capture mechanism, oscillating-array-buoys, is optimized to improve the wave energy capture efficiency. A mathematical model is developed to calculate the efficiency of buoy capturing wave energy. The simulated FABWEC models of different buoys spacing, buoys placement modes and buoys actuating arm lengths are established and analyzed by the hydrodynamic software to assess the efficiency performances. The optimization model is ultimately obtained by comparing the performances of different models. Furthermore, the physical model experiments are carried out under laboratory conditions to verify the optimization results of simulation. The results show that for the cylindrical oscillating-array-buoys, the wave energy capture efficiency could be effectively improved by increasing the buoys spacing appropriately, placing the buoys vertically and arranging the lengths of buoy actuating arms staggered; The average efficiency of buoys capturing wave energy in a wave period can reach 0.36 according to simulated investigation, which is about 38% higher than that of the basic model; The experimental performances are consistent with the simulation results, and the experimental power output of the optimized model can reach up to 1.64 W. Compared with the basic model, the average power output of the experimental optimized model is improved about 50% and the output curve is more stable. It is significant that the optimization model of array-buoys structure verified by simulation and experimental investigation has been practically applied to the next generation FABWEC system, which provides a beneficial experience for the practical application and efficient operation of wave power generation devices.