In order to solve the difficulties in the utilization of aluminum dross resources, non-fired bricks with aluminum dross with high nitrogen concentration as the main raw material were prepared. Three process parameters, including forming pressure, mixing-water amount, and aluminum dross particle size, were subjected to single-factor experiments. Based on the response surface method, a mathematical model was established between the process parameters and the non-fired bricks’ compressive properties, which were subjected to ANOVA. The process parameters were optimized and then verified experimentally. According to the results, the established regression model is able to accurately predict the compressive properties of non-fired bricks. The difference between the experimental value and the model’s predicted value was only 0.36%. The optimal process parameters for aluminum dross to prepare non-fired bricks are as follows: forming pressure is 18 MPa, mixing-water amount is 15% and particle size range is 80–130 mesh. The compressive strength of the prepared non-fired bricks is 24.66 MPa, which meets the requirement of MU20 non-fired bricks in Non-fired Rubbish Gangue Bricks.