The efficacy of non-invasive mechanical ventilation (NIMV) on the postoperative ARF is conflicting and the failure rate of NIMV in this patient population is high. In our study, we hypothesized that the use of dexmedetomidine during NIMV in major abdominal surgical patients can reduce NIMV failure without significant side affect. Medical records of patients who underwent major abdominal surgery, admitted to our general surgery intensive care unit (ICU), developed postoperative ARF, received NIMV (with oro-nasal mask) and dexmedetomidine infusion were enrolled in this study. The infusion rate was adjusted to maintain a target sedation level of a Richmond Agitation-Sedation Scale (RASS) (-2)-(-3). The sedation was stopped when NIMV was discontinued. A total of 60 patients, 42 (70.0%) male, and 18 (30.0%) female, with a mean age of 68 ± 11 years were included in the study. The mean APACHE II score was 20 ± 6. Dexmedetomidine was infused for a median of 25 h (loading dose of 0.2 mcg/kg for 10 min, maintained at 0.2-0.7 mcg/kg/h, titrated every 30 min). RASS score of all study group significantly improved at the 2 h of dexmedetomidine initiation (+3 vs. -2, p = 0.01). A targeted sedation level was achieved in 92.5% of patients. Six (10.0%) patients developed bradycardia and 5 (8.3%) patients had hypotension. The mean NIMV application time was 23.4 ± 6.1 h. Seven (11.6%) patients experienced NIMV failure, all due to worsening pulmonary conditions, and required intubation and invasive ventilation. Fifty-three (88.3%) patients were successfully weaned from NIMV with dexmedetomidine sedation and discharged from ICU. The duration of NIMV application and ICU stay was shorter in NIMV succeded group (21.4 ± 3.2 vs. 29.9 ± 6.4; p = 0.012). Our study suggests that dexmedetomidine demonstrates effective sedation in patients with postoperative ARF during NIMV application after abdominal surgery. Dexmedetomidine can be considered safe and capable of improving NIMV success.