The connection between the commutativity of a family of n×n matrices and the generalized joint numerical ranges is studied. For instance, it is shown that F is a family of mutually commuting normal matrices if and only if the joint numerical range Wk(A1,…,Am) is a polyhedral set for some k satisfying |n/2−k|≤1, where {A1,…,Am} is a basis for the linear span of the family; equivalently, Wk(X,Y) is polyhedral for any two X,Y∈F. More generally, characterization is given for the c-numerical range Wc(A1,…,Am) to be polyhedral for any n×n matrices A1,…,Am. Other results connecting the geometrical properties of the joint numerical ranges and the algebraic properties of the matrices are obtained. Implications of the results to representation theory, and quantum information science are discussed.