Drift velocity of electrons is an important parameter when signal formation is considered in detectors. In micro-pattern gas detectors such as the gas electron multiplier (GEM), the drift velocity is affected by non-uniform electric field configurations. In the context of x-ray polarimetric application of GEM, where the time projection chamber (TPC) configuration is used, the drift velocity plays an important role in forming time binned images. The accuracy of these images governs the information on polarization of incident photons. The work presented here proposes an experimental method to determine the drift velocity of electrons in such a gas detector. The gas under study is Ne/DME (50/50). The experimental setup comprising a single GEM is similar to the TPC polarimeter configuration. The effect of gas pressure and electric field on drift velocity is presented in the work. It is encouraging to find that the experimental values match well with the simulated values. We briefly discuss the effect of variation in the drift velocity on the performance parameters of the polarimeter. The implementation of this method in any future TPC polarimeter is also explored.