Diagnosing benign and malignant glands in thyroid ultrasound images is considered a challenging issue. Recently, deep learning techniques have significantly resulted in extracting features from medical images and classifying them. Convolutional neural networks ignore the hierarchical structure of entities within images and do not pay attention to spatial information as well as the need for a large number of training samples. Capsule networks consist of different hierarchical capsules equivalent to the same layers in the convolutional neural networks. We propose a feature extraction method for ultrasound images based on the capsule network. Then, we combine those deep features with conventional features such as Histogram of Oriented Gradients and Local Binary Pattern together to form a hybrid feature space. We increase the accuracy percentage of a support vector machine (SVM) by balancing and reducing the data dimensions of samples. Since the SVM provides different training kernels according to the sample distribution method, the extracted textural features were categorized using each of these kernels to obtain the result. The parameters of classification evaluation using the researcher-made model have outperformed the other methods in this field. Experimental results showed that the combination of HOG, LBP, and CapsNet methods outperformed the others, with 83.95% accuracy in the SVM with a linear kernel.