Pointwise Convolution Research Articles

RB-Net: Training Highly Accurate and Efficient Binary Neural Networks With Reshaped Point-Wise Convolution and Balanced Activation

In this paper, we find that the conventional convolution operation becomes the bottleneck for extremely efficient binary neural networks (BNNs). To address this issue, we open up a new direction by introducing a reshaped point-wise convolution (RPC) to replace the conventional one to build BNNs. Specifically, we conduct a point-wise convolution after rearranging the spatial information into depth, with which at least <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.25\times $ </tex-math></inline-formula> computation reduction can be achieved. Such an efficient RPC allows us to explore more powerful representational capacity of BNNs under a given computation complexity budget. Moreover, we propose to use a balanced activation (BA) to adjust the distribution of the scaled activations after binarization, which enables significant performance improvement of BNNs. After integrating RPC and BA, the proposed network, dubbed as RB-Net, strikes a good trade-off between accuracy and efficiency, achieving superior performance with lower computational cost against the state-of-the-art BNN methods. Specifically, our RB-Net achieves 66.8% Top-1 accuracy with ResNet-18 backbone on ImageNet, exceeding the state-of-the-art Real-to-Binary Net (65.4%) by 1.4% while achieving more than <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3\times $ </tex-math></inline-formula> reduction (52M vs. 165M) in computational complexity.

Multi‐scale group‐fusion convolutional neural network for high‐resolution range profile target recognition

Convolution neural networks (CNNs) represent one of the workhorses of artificial intelligence applications. As a typical artificial intelligence application, a high-resolution range profile (HRRP) target recognition method based on CNNs has aroused a lot of research interest. Most CNNs use a relatively small and single-scale convolution kernel size to control the number of parameters and computational complexity, but recent studies indicate that CNNs with a small kernel size cannot extract enough spatial information, which hurts the recognition performance. Aiming at this problem, this paper proposes a multi-scale group-fusion one-dimensional convolution neural network (MSGF-1D-CNN) for HRRP target recognition. MSGF-1D-CNN utilises multi-scale group one-dimensional convolution (MSG 1D-Conv) and point-wise convolution (PW-Conv) to replace the standard convolution. Multi-scale group one-dimensional convolution can significantly reduce complexity and capture the information of targets within HRRP in different levels of detail to enhance feature extraction, while PW-Conv can realise the fusion of multi-scale features to help boosting recognition performance. Experiments on five mid-course ballistic targets in the HRRP dataset show that MSGF-1D-CNN has superior recognition performance, and the parameter number of the model is reduced by more than 2.4 times than standard 1D-CNN. Furthermore, MSGF-1D-CNN shows better performance on fine-grained HRRP target recognition and anti-noise robustness in most cases.

MiniNet: Dense squeeze with depthwise separable convolutions for image classification in resource-constrained autonomous systems

In recent years, artificial intelligence (AI) has been developed vigorously, and a great number of AI autonomous applications have been proposed. However, how to decrease computations and shorten training time with high accuracy under the limited hardware resource is a vital issue. In this paper, on the basis of MobileNet architecture, the dense squeeze with depthwise separable convolutions model is proposed, viz. MiniNet. MiniNet utilizes depthwise and pointwise convolutions, and is composed of the dense connection technique and the Squeeze-and-Excitation operations. The proposed MiniNet model is implemented and experimented with Keras. In experimental results, MiniNet is compared with three existing models, i.e., DenseNet, MobileNet, and SE-Inception-Resnet-v1. To validate that the proposed MiniNet model is provided with less computation and shorter training time, two types as well as large and small datasets are used. The experimental results showed that the proposed MiniNet model significantly reduces the number of parameters and shortens training time efficiently. MiniNet is superior to other models in terms of the lowest parameters, shortest training time, and highest accuracy when the dataset is small, especially.

Multitask Touch Gesture and Emotion Recognition Using Multiscale Spatiotemporal Convolutions With Attention Mechanism

During human-robot interaction, touch gesture and emotion recognition (TGER) can help robots perceive and infer human intentions and emotional states, and establish a more harmonious human-robot relationship. Conventional handcrafted feature-based machine learning methods and end-to-end deep learning networks have been used for TGER. However, existing TGER methods are mainly based on single-task models, i.e., the touch gestures and emotions are recognized separately with different models, which suffer from relatively low recognition accuracy, high computational complexity, or high memory burden. To solve these existing problems, this paper proposed a multi-scale spatiotemporal convolutions with attention mechanism for touch gesture and emotion recognition (MUSCAT), and established a multi-task TGER framework for an overall model construction. MUSCAT uses a factorized spatiotemporal convolutional neural network to sequentially extract spatial and temporal features, and multi-scale convolution is used in temporal convolution. The attention mechanisms are applied after spatial convolution, temporal convolution and point-wise convolution to respectively focus on the key information of spatial, temporal and channel dimensions. The proposed MUSCAT model has a deeper structure than the standard three-dimensional convolution, which gains better performance with fewer model parameters. Extensive experiments were conducted on two touch datasets, and the results verify the feasibility of the proposed method.

WeaveNet: Solution for Variable Input Sparsity Depth Completion

LIDARs produce depth measurements, which are relatively sparse when compared with cameras. Current state-of-the-art solutions for increasing the density of LIDAR-derived depth maps rely on training the models for specific input measurement density. This assumption can easily be violated. The goal of this work was to develop a solution capable of producing reasonably accurate depth predictions while using input with a very wide range of depth information densities. To that end, we defined a WeaveBlock capable of efficiently propagating depth information. To achieve this goal, WeaveBlocks utilize long and narrow horizontal and vertical convolution kernels together with MobileNet-inspired pointwise convolutions serving as computational kernels. In this paper, we present the WeaveNet architecture for guided (LIDAR and camera) and unguided (LIDAR only) depth completion as well as a non-standard network training procedure. We present the results of the network on the KITTI test and validation sets. We analyze the network performance at various levels of input sparsity by randomly removing between 0% and 99% of the LIDAR points from the network inputs, and in each case, we obtain reasonable quality output. Additionally, we show that our trained network weights can easily be reused with a different LIDAR sensor.

Lightweight Inception Networks for the Recognition and Detection of Rice Plant Diseases

Deep learning is a hot research topic in image processing and computer vision. It has been employed in many fields owing to the excellent performance. However, so far, deep learning methods have been rarely applied in the task of plant disease recognition, except that some existing work focuses on it from a public dataset of images zoomed on plant leaves. The main reasons behind the limited usage of deep learning models in plant disease recognition include: the large volume of deep learning models, which is difficult to deploy on embedded systems, the great computational complexity, which requires large memory overhead, the complex backgrounds of experimental materials, which are hard to train an efficient model, and others. To address these challenges, the paper proposes a valid lightweight network architecture, namely MobInc-Net, to perform the crop disease recognition and detection. In this study, the Inception module was enhanced by replacing the original convolutions with depth-wise and point-wise convolutions, and then the modified Inception (M-Inception) module paired with the pre-trained MobileNet was chosen as the backbone extractor to extract high-quality image features. After that, the completely linked Softmax layer with the actual number of categories and the SSD block were separately added behind the foundation network for classifying and detecting crop disease types. To train an efficient model, the two-stage transfer learning was applied in the model training. Experimental findings show that the proposed method can attain the desired performance with an average recognition accuracy of 99.21% on the public dataset and 97.89% on the local dataset.

Channel-Wise Average Pooling and 1D Pixel-Shuffle Denoising Autoencoder for Electrode Motion Artifact Removal in ECG

This paper presents a channel-wise average pooling and one dimension pixel-shuffle architecture for a denoising autoencoder (CPDAE) design that can be applied to efficiently remove electrode motion (EM) artifacts in an electrocardiogram (ECG) signal. The three advantages of the proposed design are as follows: (1) In the skip connection layer, less memory is needed to transfer the features extracted by the neural network; (2) Pixel shuffle and pixel unshuffle techniques with point-wise convolution are used to effectively reserve the key features generated from each layer in both the encoder and decoder; (3) Overall, fewer parameters are required to reconstruct the ECG signal. This paper describes three deep neural network models, namely CPDAELite, CPDAERegular, and CPDAEFull, which support various computational capacity and hardware arrangements. The three proposed structures involve an encoder and decoder with six, seven, and eight layers, respectively. Furthermore, the CPDAELite, CPDAERegular, and CPDAEFull structures require fewer multiply-accumulate operations—355.01, 56.96, and 14.69 million, respectively—and less parameter usage—2.69 million, 149.7 thousand, and 55.5 thousand, respectively. To evaluate the denoising performance, the MIT–BIH noise stress test database containing six signal-to-noise ratios (SNRs) of noisy ECGs was employed. The results demonstrated that the proposed models had a higher improvement of SNR and lower percentage root-mean-square difference than other state-of-the-art methods under various conditions of SNR.

Runtime Support for Accelerating CNN Models on Digital DRAM Processing-in-Memory Hardware

Processing-in-memory (PIM) provides promising solutions to the main memory bottleneck by placing computational logic in or near memory devices to reduce data movement overheads. Recent work explored how commercial DRAM can feature digital PIM logic while meeting fab-level energy and area constraints, and showed a significant speedup in the inference time of data-intensive deep learning models. However, convolutional neural network (CNN) models were not considered as main targets for the commercial DRAM-PIM due to their compute-intensive convolution layers. Moreover, recent studies revealed that the area and power constraints on memory die prevent DRAM-PIM from competing with GPUs and specialized accelerators in accelerating them. Recently, mobile CNN models have increasingly adopted a composition of depthwise and pointwise convolutions instead of such compute-intensive convolutions to reduce computation cost without accuracy drop. In this paper, we show that 1x1 convolution can be offloaded for PIM acceleration with integrated runtime support and without any hardware or algorithm changes. We provide further speedup with parallel execution on GPU and DRAM-PIM and code generation optimizations. Our solution achieves up to 35.2% (31.6% on average) speedup for all 1x1 convolutions for mobile CNN models against GPU.

Driver Distraction Detection Using Octave-Like Convolutional Neural Network

This study proposes a lightweight convolutional neural network with an octave-like convolution mixed block, called OLCMNet, for detecting driver distraction under a limited computational budget. The OLCM block uses point-wise convolution (PC) to expand feature maps into two sets of branches. In the low-frequency branches, we perform average pooling, depth-wise convolution (DC), and upsampling to obtain a low-resolution low-frequency feature map, reducing spatial redundancy and connection density. In the high-frequency branches, the expanded feature map with the original resolution is fed to the DC operator, gaining an apposite receptive field to capture fine details. The feature concatenation of the low-frequency and high-frequency branches is encoded sequentially by a squeeze-and-excitation (SE) module and PC operator, realizing feature global information fusion. Introducing another SE module at the last stage, the OLCMNet facilitates further sensitive information exchange between layers. In addition, with an augmented reality head-up display (ARHUD) platform, we create a Lilong Distracted Driving Behavior (LDDB) Dataset through a series of on-road experiments. Such a dataset contains 14808 videos collected from an infrared camera, covering six driving behaviors of 2468 participants. We manually annotate these videos at five frames per second, obtaining a total of 267378 images. Compared with the existing methods, the embedded hardware platform experiments indicate that OLCMNet hits acceptable trade-offs, namely, 89.53% accuracy for StateFarm Dataset and 95.98% accuracy LDDB Dataset when the latency is 32.8 ± 4.6ms.

Grouped Pointwise Convolutions Reduce Parameters in Convolutional Neural Networks

In DCNNs, the number of parameters in pointwise convolutions rapidly grows due to the multiplication of the number of filters by the number of input channels that come from the previous layer. Our proposal makes pointwise convolutions parameter efficient via grouping filters into parallel branches or groups, where each branch processes a fraction of the input channels. However, by doing so, the learning capability of the DCNN is degraded. To avoid this effect, we suggest interleaving the output of filters from different branches at intermediate layers of consecutive pointwise convolutions. We applied our improvement to the EfficientNet, DenseNet-BC L100, MobileNet and MobileNet V3 Large architectures. We trained these architectures with the CIFAR-10, CIFAR-100, Cropped-PlantDoc and The Oxford-IIIT Pet datasets. When training from scratch, we obtained similar test accuracies to the original EfficientNet and MobileNet V3 Large architectures while saving up to 90% of the parameters and 63% of the flops.

DSE-YOLO: Detail semantics enhancement YOLO for multi-stage strawberry detection

Multi-stage strawberry fruits detection is one of the important clues to estimate crop yields and assist robotic picking in modern agricultural production. However, it is difficult for detecting strawberries due to their small size, foreground-foreground class imbalance, and complex natural environment. Many works focus on how to detect fruits while ignoring multi-stage fruit detecting problems. In this paper, we propose DSE-YOLO (Detail-Semantics Enhancement You Only Look Once) to detect multi-stage strawberries. In DSE-YOLO, DSE (Detail-Semantics Enhancement) module is designed for detecting small fruits and distinguishing different stages of the fruit with higher accuracy, which utilize pointwise convolution and dilated convolution to extract various detail and semantics features in the horizontal and vertical dimensions. Exponentially Enhanced Binary Cross Entropy (EBCE) and Double Enhanced Mean Square Error (DEMSE) loss function are constructed to focus on small fruits, which can deal with foreground-foreground class imbalance problem. Experiments conducted on datasets demonstrate the superiority of DSE-YOLO over state-of-the-arts. The detection results had a mAP value of 86.58% and an F1-Score value of 81.59%, which demonstrates the effectiveness of the proposed model. Especially, DSE-YOLO can almost detect every stage of strawberry fruit accurately in the natural scene, which can provide an important theoretical basis and premise for automatic picking and monitoring system.

DMPCONV: DECOUPLING MULTI-BRANCH POINTWISE CONVOLUTIONS FOR LIGHT-WEIGHT REMOTE SENSING SCENE CLASSIFICATION

Abstract. The use of multi-branch architectures in off-the-shelf light-weight residual series neural networks can significantly improve their performance in remote sensing scene classification tasks. However, such architectures come at the expense of an increased number of parameters and calculations. In this paper, we propose the Decoupling Multi-branch Pointwise Convolutions (DMPConv), which works without a corresponding increase in parameters and calculations during inferencing, and at the same time, can maintain the same performance improvement ability as the multi-branch architectures. DMPConv can be decoupled into two states, the training-time DMPConv and the inferencing-time DMPConv. The training-time DMPConv enhances the expressivity of the network by using weighted multi-branch 1×1 convolutions. After training, we use structural reconstruction to convert the training-time DMPConv to the inferencing-time DMPConv, which has the same form as vanilla 1×1 convolution, so as to realize the inferencing-free. Extensive experiments were conducted on multiple remote sensing scene classification benchmarks, including Aerial Image data set and NWPU-RESISC45 data set to demonstrate the superiority of DMPConv.

Using lightweight convolutional neural network to track vibration displacement in rotating body video

The non-contact visual displacement measurement methods can detect problems such as rigid body deformation and structural wear of structural bodies due to long-term service life. However, the traditional visual vibration measurement methods are limited by the inherent sampling constraint of ordinary image sensors, which cannot achieve efficient vibration target identification and correlation tracking for rotating body targets in low-resolution videos under the premise of high sampling rate. Therefore, this paper uses high-speed industrial camera as the acquisition medium, introduces deep convolutional neural networks into the field of visual vibration measurement, on this basis, weighs the target detection accuracy and displacement tracking speed, uses the designed lightweight convolutional neural networks model to vibration displacement tracking in rotating body videos. We ensure that the computational efficiency and parameter quantity of the convolutional networks are solved with low loss accuracy. First of all, we take the lightweight convolutional neural network as the backbone network, replace the standard convolutional neural network with depthwise separable convolution and pointwise convolution. Considering the speed and accuracy advantages of deep learning algorithms for video object tracking, we estimate the heat map, object center offsets and bounding box sizes use an anchor-free detection algorithm on the network framework. In order to prevent the loss of vibrating target identity in rotating body videos, we use re-identification(re-ID) method to strengthen the correlation of target displacement between adjacent frames. In experiments with traditional visual vibration measurement and recent deep learning measurement methods for performance testing, the network model we designed can demonstrate absolute advantages that traditional convolutional neural networks do not have. On the one hand, our comparison of time–frequency characteristics at different speeds shows that the vibration displacement curve regressed by the lightweight convolutional neural network has a high degree of fit with the displacement signal obtained by the eddy current sensor;on the other hand, when the target object is blurred, the generalization ability of our algorithm is proved, which also reflects the engineering application value of visual vibration measurement in the field of vibration displacement tracking of rotating bodies.

A Novel Light-Weight Approach for the Classification of Different Types of Psoriasis Disease Using Depth Wise Separable Convolution Neural Networks

Objectives: The main objective of the work is to classify the psoriasis affected skin into one of different psoriasis types viz. erythrodermic, gutatte, inverse, nail, plaque, and pustular using depth wise separable convolution neural networks. Methods: To identify the type of psoriasis disorder, the experiment uses MobileNet machine learning architecture which is based on depthwise separable convolutions. In the preprocessing step, the input images are segmented into 224 pixels X 224 pixels with the help of the keras KerasImageDataGenerator function and in next step, these segmented images are fed as input to 28 layers of reconstructed MobileNet architecture. A series of convolutions and depth-wise separable convolutions layers are applied on the input images. The rectified linear activation function is applied to introduce non-linearity in the network. Adam optimizer algorithm is used for training the network. Categorical cross-entropy is used for the comparison of the accuracy of the experimental results with the existing work of classification of psoriasis disorder. Findings: Using MobileNet machine learning architecture, the experiments attained around 86% of classification efficiency, and an average F1􀀀 score of .94 as compared with VGG19 and ResNet-34, compared to the work done in(1). Novelty: The novelty of the work lies in the prediction of the types of the psoriasis disorder accurately with less turnaround time with an accuracy of 86% consuming low processing capability which can be implemented on low powered hand-held devices. Keywords: Machine Learning; Depth Wise Separable Convolution; Psoriasis Disorder; ReLU Activation; Pointwise convolutions

Battery health prognosis using improved temporal convolutional network modeling

Accurate estimation of the state of health (SOH) of lithium-ion batteries is the key to ensure the safe use of lithium-ion batteries. In practice, the application of traditional health features is hindered by incomplete charge and discharge. When the battery is stably charged, the voltage and temperature of the battery under different health states show similar spatial degradation trends. Therefore, the degradation trend of voltage and temperature can be directly used as the health feature sequence to reduce the error caused by manual feature extraction. In addition, a new model attention depthwise temporal convolutional network (AD-TCN) considering health characteristics is proposed for SOH estimation. Depthwise separable convolution operation is used to extend temporal convolutional network (TCN) to a model suitable for multivariate prediction. Depthwise convolution is used as feature extractor, and pointwise convolution recombines all features for regression prediction. In addition, the convolutional block attention module is used in the channel dimension and spatial dimension to selectively enhance or suppress the details. Experiments on NASA data sets show that this method has strong reliability and high prediction accuracy.

LRP-net: A lightweight recursive pyramid network for single image deraining

Single image deraining, as a low-level computer vision task, has been drawn extensive attention in recent years. Rain streaks can degrade subjective visibility quality, meanwhile, bring significant difficulties to subsequent high-level computer vision tasks such as object detection. Nowadays, deep-learning based methods, specifically Convolutional Neural Networks (CNN) based ones are adopted to remove the rain streaks and become the state-of-the-art. However, existing popular deep-networks have complicated branches and numerous layers, which strengthen the ability of removing rain streaks and result in high memory and computational cost inevitably. This restricts many applications in real-time and limited computation resource scene, especially on mobile or edge devices. To handle this issue, this paper proposes a novel Lightweight Recursive Pyramid network (LRP-Net) with a small number of parameters for single image deraining. To begin with, we propose a novel Lightweight Pyramid Deraining (LPD) block which consists of a multi-scale pyramid convolution for sufficient feature extraction and a pointwise convolution for feature fusion. Meanwhile, we also design a novel group convolution strategy in LPD for the sake of remarkable parameter reduction. Secondly, we combine a recursive deraining mechanism, a critical component that serves as a feature fusion iterator to construct our LRPNet a powerful and lightweight multi-stage model. In the benefit of the combination between the LPD block and recursive mechanism, the total number of parameters in LRP-Net is only 130 k, which is nearly a 40- reduction compared with the latest state-of-the-art models. The extensive experiments demonstrate the superiority of LRP-Net in both quantitative assessments and visual quality.

Local-global coupling relationship based low-light image enhancement

Convolutional neural networks have made significant progress in various fields of artificial intelligence. However, the convolution operation which is based on a shared parameter sliding window mechanism only focuses on modeling the local relationship and is insufficient to establish global relationships. Nonetheless, the local and global relationships are both crucial for feature representation. Therefore, this paper concentrates on how to efficiently construct and couple the local and global relationships to mine more abundant feature information and enhance the discriminability of features. To this end, a local-global coupling module is designed, which is composed of four basic components: feature extraction, local relationship branch based on depth-wise convolution (DWConv), global relationship branch based on multi-head self-attention (MHSA), and local-global relationship coupling based on point-wise convolution (PWConv). On this basis, a local-global relationship coupling with encoder-decoder structure neural network is proposed, which can efficaciously model the local-global coupling relationship, strengthen the discriminative ability of feature information, and improve the performance of the model. Extensive experiments on low-light image enhancement datasets demonstrate the effectiveness of the proposed method, which significantly outperforms the state-of-the-art counterparts. Furthermore, the importance and effectiveness of local-global relationship coupling are analyzed through ablation and extended experiments.

U-shaped GAN for Semi-Supervised Learning and Unsupervised Domain Adaptation in High Resolution Chest Radiograph Segmentation.

Deep learning has achieved considerable success in medical image segmentation. However, applying deep learning in clinical environments often involves two problems: (1) scarcity of annotated data as data annotation is time-consuming and (2) varying attributes of different datasets due to domain shift. To address these problems, we propose an improved generative adversarial network (GAN) segmentation model, called U-shaped GAN, for limited-annotated chest radiograph datasets. The semi-supervised learning approach and unsupervised domain adaptation (UDA) approach are modeled into a unified framework for effective segmentation. We improve GAN by replacing the traditional discriminator with a U-shaped net, which predicts each pixel a label. The proposed U-shaped net is designed with high resolution radiographs (1,024 × 1,024) for effective segmentation while taking computational burden into account. The pointwise convolution is applied to U-shaped GAN for dimensionality reduction, which decreases the number of feature maps while retaining their salient features. Moreover, we design the U-shaped net with a pretrained ResNet-50 as an encoder to reduce the computational burden of training the encoder from scratch. A semi-supervised learning approach is proposed learning from limited annotated data while exploiting additional unannotated data with a pixel-level loss. U-shaped GAN is extended to UDA by taking the source and target domain data as the annotated data and the unannotated data in the semi-supervised learning approach, respectively. Compared to the previous models dealing with the aforementioned problems separately, U-shaped GAN is compatible with varying data distributions of multiple medical centers, with efficient training and optimizing performance. U-shaped GAN can be generalized to chest radiograph segmentation for clinical deployment. We evaluate U-shaped GAN with two chest radiograph datasets. U-shaped GAN is shown to significantly outperform the state-of-the-art models.

Tensor neural networks via circulant convolution

Enough computation and storage resources have made Convolutional Neural Networks (CNNs) prosperous in many fields, but it is still challenging to deploy these models on mobile or embedded devices. Designing effective and efficient convolution modules can vastly reduce the number of parameters and computational complexity of CNNs to favor the deployment. This paper proposes a Circulant Convolution (CC) established on a tensor algebra operator of t-product. Specifically, a feature map can be parameterized by multilinear operation to make the channels positively coupled, and then a more compact parameter space and stronger parameterization capability can be simultaneously obtained. Replacing standard spatial convolution with CC can effectively reduce almost 9 times the number of network parameters and FLOPs, and the impact on accuracy is much smaller than other well-designed convolution modules for lightweight networks. CC can be embedded into existing network architectures as a plug-and-play module, and its topology structure can be easily extended to high-dimensional data. To favor the use of CC in CNNs, a circulant convolution module (CCM), also known as the bottleneck of CC, is also designed by combining CC and pointwise convolution. In further, a lightweight network CCMNet is constructed based on incorporating CC and CCM into an existing lightweight backbone. Extensive experiments on the benchmark dataset CIFAR10/100, ImageNet, Lung-CXR, and MS COCO, demonstrate that our proposed CC could significantly improve the performance of several advanced lightweight networks, and CCMNet is competitive with the start-of-the-art portable neural networks.

A New Pointwise Convolution in Deep Neural Networks Through Extremely Fast and Non Parametric Transforms

Some conventional transforms such as Discrete Walsh-Hadamard Transform (DWHT) and Discrete Cosine Transform (DCT) have been widely used as feature extractors in image processing but rarely applied in neural networks. However, we found that these conventional transforms can serve as a powerful feature extractor in channel dimension without any learnable parameters in deep neural networks. This paper firstly proposes to apply conventional transforms on pointwise convolution, showing that such transforms can significantly reduce the computational complexity of neural networks without accuracy degradation on various classification tasks and even on face detection task. Our comprehensive experiments show that the proposed DWHT-based model gained 1.49% accuracy increase with 79.4% reduced parameters and 49.4% reduced FLOPs compared with its baseline model on the CIFAR 100 dataset while achieving comparable accuracy under the condition that 81.4% of parameters and 49.4% of FLOPs reduced on SVHN dataset. Additionally, our DWHT-based model showed comparable accuracy with 89.2% reduced parameters and 26.5% reduced FLOPs compared to the baseline models on WIDER FACE and FDDB datasets.