The extent of relative tilt angle, crystalline quality, and relaxation of GaxIn1−xSb layers grown on (001) GaSb substrates by organometallic vapor phase epitaxy have been investigated, using double-crystal x-ray diffraction and transmission electron microscopy. An unexpectedly large tilt is formed between Ga0.8In0.2Sb epitaxial layers and oriented (001) GaSb substrates which has not been previously reported. Double-crystal x-ray diffraction measurements revealed that the tilt angle between the Ga0.8In0.2Sb epilayers and the substrates increases as the layer thickness increases. A strong correlation has been established between the variation of the tilt angle and the residual strain in the layers. Transmission electron micrographs of Ga0.8In0.2Sb layers revealed that irregular dislocation activities occur in the layer at different distances from the interface which could be related to tilt formation and relaxation. The structural characteristics of the layers as a function of the compositional variation showed that the amount of tilt angle is small when the indium concentration was in the range from 0 to 12%, but increases at higher indium concentrations. Ga1−xInxSb layers with poor crystalline quality and small tilt angle are obtained when the indium concentration was more than 25%. The tilt angle between Ga0.8In0.2Sb epilayers and GaSb substrates grown at temperatures ranging from 560 to 620 °C was essentially identical, indicating that the tilt formation is not a kinetic effect.