The study of flapping-wing aerodynamics faces a large control space with different wing kinematics and deformation. The adjoint-based approach, by solving an inverse problem to obtain simultaneously the sensitivity with respect to all control parameters, has a computational cost independent of the number of control parameters and becomes an efficient tool for the study of problems with a large control space. However, the adjoint equation is typically formulated in a fixed fluid domain. In a continuous formulation, a moving boundary or morphing domain results in inconsistency in the definition of an arbitrary perturbation at the boundary, which leads to ambiguousness and difficulty in the adjoint formulation if control parameters are related to boundary changes (e.g. the control of wing kinematics and dynamic deformation). The unsteady mapping function, as a traditional way to deal with moving boundaries, can in principle be a remedy for this situation. However, the derivation is often too complex to be feasible, even for simple problems. Part of the complexity comes from the unnecessary mapping of the interior mesh, while only mapping of the boundary is needed here. Non-cylindrical calculus, on the other hand, provides a boundary mapping and considers the rest of domain as an arbitrary extension from the boundary. Using non-cylindrical calculus to handle moving boundaries makes the derivation of the adjoint formulation much easier and also provides a simpler final formulation. The new adjoint-based optimization approach is validated for accuracy and efficiency by a well-defined case where a rigid plate plunges normally to an incoming flow. Then, the approach is applied for the optimization of drag reduction and propulsive efficiency of first a rigid plate and then a flexible plate which both flap with plunging and pitching motions against an incoming flow. For the rigid plate, the phase delay between pitching and plunging is the control and considered as both a constant (i.e. a single parameter) and a time-varying function (i.e. multiple parameters). The comparison between its arbitrary initial status and the two optimal solutions (with a single parameter or multiple parameters) reveals the mechanism and control strategy to reach the maximum thrust performance or propulsive efficiency. Essentially, the control is trying to benefit from both lift-induced thrust and viscous drag (by reducing it), and the viscous drag plays a dominant role in the optimization of efficiency. For the flexible plate, the control includes the amplitude and phase delay of the pitching motion and the leading eigenmodes to characterize the deformation. It is clear that flexibility brings about substantial improvement in both thrust performance and propulsive efficiency. Finally, the adjoint-based approach is extended to a three-dimensional study of a rectangular plate in hovering motion for lift performance. Both rigid and flexible cases are considered. The adjoint-based algorithm finds an optimal hovering motion with advanced rotation which has a large leading-edge vortex and strong downwash for lift benefit, and the introduction of flexibility enhances the wake capturing mechanism and generates a stronger downwash to push the lift coefficient higher.