To investigate the crystal electric field effect and antiferromagnetic transitions at 2.4 and 1.4 K, we carried out ultrasonic measurements on the Yb-based heavy-fermion antiferromagnet YbIrGe. We found characteristic softening of the elastic moduli originating from a quadrupole interaction. By theoretical analyses of the transverse elastic moduli, the magnetic susceptibility, and the Schottky specific heat between 10 and 300 K, we determined the crystal electric field: the ground doublet and an excited doublet separated by 138 K. The moduli exhibit elastic hardening below both 2.4 and 1.4 K, suggesting a strong coupling between a strain and magnetic order parameters. We estimated the strain dependence of the transition temperature at 2.4 K along the a- and c-axes from the Ehrenfest relation for the first time.