In real decision making, decision makers tend to express their opinions with uncertainty when facing complicated decision problem and environment. This paper develops a novel consensus reaching process for multiple attribute group decision making (MAGDM) with hesitant fuzzy linguistic term sets (HFLTSs). Firstly, we define a new distance measure for two HFLTSs and propose a distance-based consensus measure for the MAGDM with HFLTSs. Then, based on this consensus measure, we develop a minimum adjustment distance consensus rule for the MAGDM with HFLTSs, which can minimize the adjustment distance between the original and adjusted opinions in the process of reaching consensus. Moreover, to obtain the collective opinion with maximum consensus, we develop a minimum distance aggregation model, which is to minimize the maximum of the distance between each decision maker's individual opinion and the collective opinion. Furthermore, based on the proposed consensus rule and aggregation model, we present a consensus reaching process for MAGDM with HFLTSs. Finally, we provide the convergence proof of the consensus reaching process, and a numerical example is used to demonstrate the validity of the consensus reaching process.