Purpose: Violent subjects were demonstrated to exhibit abnormal brain structures; however, the brain changes may be different between criminals committing affective (VA), predatory violence (VP), and non-violence (NV). Therefore, the purpose of this study was to compare the differences in brain structures and psychological characteristics between VA, VP, and NV offenders. Methods: Twenty male criminal subjects (7 VP; 6 VA; and 7 NV) offenders; and twenty age-matched male healthy non-criminals were enrolled in this study. All subjects received psychological assessments as well as magnetic resonance imaging scans of the brain. Analysis of variance (ANOVA) was performed to understand the differences among four groups with Bonferroni correction. The voxel-based morphometry and voxel-wise diffusion tensor imaging analyses were performed to compare the gray matter (GM) volume and white matter (WM) integrity between the groups. In significant regions, a Spearman correlation analysis was performed to understand the relationship between the brain changes and psychological scores. Results: The ANOVA analysis showed that AUDIT scores were significantly different among four groups, but no significant group difference was noted after Bonferroni correction. The imaging comparisons further demonstrated that the VP and NV offenders exhibited significant alterations of WM and GM tissues in the rectus and superior temporal gyrus, respectively. In addition, the VP offenders exhibited greater GM volumes than VA offenders in the right middle frontal gyrus, and NV offenders had greater GM volumes than VP offenders in the bilateral thalamus. Conclusion: We concluded that the VA, VP, and NV groups exhibited different degrees of alterations in GM and WM tissues in regions involved in emotion and cognition.