Type 2 diabetes is a common condition that causes the level of glucose in the blood to become too high and α-glucosidase inhibitors are therapeutic agents in managing type 2 diabetes. In the present study, we report a new series of 3-aceto(benzo)hydrazide-1,2,4-triazine analogs as potential therapeutic agents against type 2 diabetes. In the in vitro evaluations, most compounds showed much stronger α-glucosidase inhibitory activity than the standard drug acarbose. Especially, compound 2A, (N'-(5,6-diphenyl-1,2,4-triazin-3-yl)-4-methoxybenzohydrazide) as the most active compound showed IC50 = 12.0 μM which is 60 folds more potent than acarbose. In addition, the MTT test using the three cell lines HCT-116, MDA-MB-231, and A549 showed that the target compounds have low cytotoxic effects with IC50 values in the range of 60–280 μM, therefore they can be considered as safe compounds. Molecular docking studies predicted that the strong inhibitory activity of 2A is related to the interactions generated by the three residues Asp282, Trp481, and Asp616 with the triazine core and hydrazide motif in the active site of the enzyme. The significant inhibitory effect of 2A against α-glucosidase was also confirmed in vivo, where the compound showed equivalent activity to the standard drug acarbose on reducing blood glucose in the tested mice. In conclusion, this study introduces compound 2A as a new lead compound with favorable cytotoxicity, strong α-glucosidase inhibitory activity and in vivo hypoglycemic effect, for future investigation in the treatment of diabetes mellitus.