Due to several complex factors such as the type, size, and shape of target object, vision-assisted robot grasping technology still faces serious challenges. In this research, a deep learning-based robot hand vision grasping algorithm was developed considering semi-structural environmental constraints. The proposed algorithm could build a deep learning network on the basis of the desired object, perform object recognition, category classification and position judgment, and complete robot hand-grasping tasks. The obtained experimental results demonstrated that the proposed algorithm effectively solved the problem of recognizing and classifying multi-category objects in a semi-structured environment, improving recognition rate and grasping rate and reducing collision rate.