We propose a generalization of Yang-Mills theory for which the symmetry algebra does not have to be factorized as mutually commuting algebras of a finite-dimensional Lie algebra and the algebra of functions on base space. The algebra of diffeomorphism can be constructed as an example, and a class of gravity theories can be interpreted as generalized Yang-Mills theories. These theories in general include a graviton, a dilaton and a rank-2 antisymmetric field, although Einstein gravity is also included as a special case. We present calculations suggesting that the connection in scattering amplitudes between Yang-Mills theory and gravity via BCJ duality can be made more manifest in this formulation.