The determination of elastic constants of a transversely isotropic carbon fiber reinforced plastic (CFRP) composite is the prerequisite of mechanical strength calculation, material degradation evaluation, ultrasonic non-destructive testing for main load-bearing CFRP structures both after manufacturing and in service. Conventional ultrasonic methods request extra sample preparation, water immersion condition or special designed goniometric devices to rotate the test structures, making the elastic constants measurement expensive, inconvenient and not in situ. In this paper, a novel ultrasonic method that enables in situ determination of elastic constants via backwall reflection method (BRM) using a linear array probe is proposed. The BRM can achieve single sided measurement of ultrasonic travel times in various directions including the fiber direction using different pairs of transmitter and receiver array elements. Both quasi longitudinal and quasi shear waves are captured via the mode conversions from the multiple surface reflections. All elastic constants are determined through the particle swarm optimization by minimizing the sum of the squared deviations between the BRM measured and theoretically calculated multi-mode bulk wave travel times in the fiber orthogonal and parallel planes. This method is experimentally verified on a 4.45 mm thick unidirectional T700 carbon fiber/epoxy CFRP composite. The BRM measured Young's modulus in the fiber direction agrees well with that measured by tensile test, with a small deviation of −4.58%. This work proves that the proposed method is single sided, easy to operate, without the necessity of sample preparation, water immersion and extra rotation device, and can determine all elastic constants with high precision, which is therefore promising for in situ applications.