Fish fin possesses large deformations in its motion cycle assisting fish's swimming, in which its geometric parameters such as surface area, aspect ratio change greatly, and the complex deformations and motions result in complicated hydrodynamic response. In this paper, the dynamic change of surface area is concentrated to improve the propulsion performance of underwater propeller. A novel variable area robotic fin is developed and the effect of surface area change on the hydrodynamic forces is investigated quantitatively. The robotic fin composes two parts: a base fin with hand shaped holes and a cover fin that fits the shape of the holes. The change of the surface area of the robotic fin is realized by rotating the cover fin to shield the holes in the base fin. A crank-rocker-cam composite mechanism is designed to realize the fin pitching motion and surface change motion synchronously with one driving motor. Four control modes of surface area change in a motion circle are investigated, namely, complete traditional invariable fin, traditional invariable fin with smaller surface, fin with larger surface during in-strokes and fin with larger surface during out-strokes. The thrust force and efficiency of the four control modes with various swimming speeds are detailed experimented and discussed. It is found that the variable area fin achieves a remarkable different hydrodynamic response and the corresponding control modes affect much. For the variable surface area fin, they generate average thrust force between the complete invariable fin and invariable fin with smaller surface, in which the fin with larger surface area during in-strokes follows closely the complete traditional invariable fin, while the fin with larger surface area during out-strokes performs more like the traditional invariable fin with smaller surface. It is interesting that fin with larger surface during in-strokes can generate much larger average thrust force than the fin with large surface during out-strokes. For the efficiency, the fin with larger surface during in-strokes behaves the best. And the effect of the surface area change ratio and time is closely connected with the control modes. Besides, the influences of pitching frequency and amplitude are also studied. The results demonstrate that the propulsive performance can be indeed improved by proper surface area change in a motion cycle, which will be an inspiration to the design of novel underwater robot propulsive system.