Addressing the nature of an unexpected smectic-A' phase in liquid crystal 54COOBC films, we perform large scale Monte Carlo simulations of a coupled hexatic-nematic XY model. The resulting finite-temperature phase diagram reveals a small region with composite Potts $\mathbb{Z}_3$ order above the vortex binding transition; this phase is characterized by relative hexatic-nematic ordering though both variables are disordered. The system develops algebraic hexatic and nematic order only at a lower temperature. This multi-step melting scenario agrees well with the experimental observations of a sharp specific heat anomaly that emerges above the onset of hexatic positional order. We therefore propose that the smectic-A' phase is characterized by composite Potts order and bound-states of fractional vortices.