Malignant liver tumors are considered as one of the most common cancers and a leading cause of cancer death worldwide. While using convolutional neural networks (CNNs) for feature extraction from ultrasound (US) images and tasks thereafter, most works focus on pre-trained architectures using transfer learning which can sometimes cause negative transfer and reduced performance in medical domain. A new method based on Pascal’s Triangle was developed for feature extraction using CNN. The convolutions and the kernels in Pascal’s Triangle based CNN (PT-CNN) are according to the coefficients at each level of Pascal’s Triangle. Due to the fuzzy nature of US images, the input layer takes a combination of the image and its neutrosophically pre-processed components as a single unit to improve noise robustness. The proposed CNN when implemented as a backbone for feature extraction for binary classification, gave validation accuracy > 90% and test accuracy >95% than other state-of-the-art CNN architectures. For tumor segmentation using Mask R-CNN framework, the aggregated feature maps from all convolutional layers were given as the input to the region proposal network for multiscale region proposals and to facilitate the detection of tumors of varying sizes. This gave an F1 score > 0.95 when compared with other architectures as backbone on Mask R-CNN framework and simple U-Net based segmentation. It is suggested that the promising results of PT-CNN as a feature extraction backbone could be further investigated in other domains.