We study a new class of infinite-dimensional Lie algebras W ∞ ( N + , N − ) generalizing the standard W ∞ algebra, viewed as a tensor operator algebra of SU ( 1 , 1 ) in a group-theoretic framework. Here we interpret W ∞ ( N + , N − ) either as an infinite continuation of the pseudo-unitary symmetry U ( N + , N − ) , or as a “higher- U ( N + , N − ) -spin extension” of the diffeomorphism algebra diff ( N + , N − ) of the N = N + + N − torus U ( 1 ) N . We highlight this higher-spin structure of W ∞ ( N + , N − ) by developing the representation theory of U ( N + , N − ) (discrete series), calculating higher-spin representations, coherent states and deriving Kähler structures on flag manifolds. They are essential ingredients to define operator symbols and to infer a geometric pathway between these generalized W ∞ symmetries and algebras of symbols of U ( N + , N − ) -tensor operators. Classical limits (Poisson brackets on flag manifolds) and quantum (Moyal) deformations are also discussed. As potential applications, we comment on the formulation of diffeomorphism-invariant gauge field theories, like gauge theories of higher-extended objects, and non-linear sigma models on flag manifolds.