With the growth of access to internal system resources, the insider threat problem is emerging and can bring immeasurable losses to enterprises. In order to detect the hidden threats and guide the formulation of enterprise management strategy, it is important to analyze and understand the employee behavior. Thus, we suggest a user behavior analysis system framework, which mainly includes data processing, user behavior modeling and results analysis. Data Adjusting (DA) strategy and optimized eXtreme Gradient Boosting (XGBoost) model are utilized with the aim of full analysis small amount of feature information. The strategy for detecting suspicious behavior can be the following. Firstly, select initial suspicious data, misclassification data retention and combination sampling. Secondly, for further behavior model construction, an improved Particle Swarm Optimization algorithm based on ethnic randomized particles (ERPSO), which introduces Gaussian white noise with adjustable intensity into acceleration coefficients is given for searching the optimal XGBoost parameters. In addition, based on the designed DA strategy and the proposed ERPSO algorithm, we have also compared the results of the proposed methods with the current state-of-the-art methods. Experimental results show that the XGBoost optimized by the ERPSO (ERPSO-XGBoost) model has comprehensive performance, which proves the rationality and effectiveness of the insider behavior analysis system framework. Through a comprehensive understanding of insider behavior, the obvious characteristic behavior is found to adjust the management strategy and guide the behavior modeling, so as to prevent more losses in time.