Convolutional Filters Research Articles

Object tracking in infrared images using a deep learning model and a target-attention mechanism

Small object tracking in infrared images is widely utilized in various fields, such as video surveillance, infrared guidance, and unmanned aerial vehicle monitoring. The existing small target detection strategies in infrared images suffer from submerging the target in heavy cluttered infrared (IR) maritime images. To overcome this issue, we use the original image and the corresponding encoded image to apply our model. We use the local directional number patterns algorithm to encode the original image to represent more unique details. Our model is able to learn more informative and unique features from the original and encoded image for visual tracking. In this study, we explore the best convolutional filters to obtain the best possible visual tracking results by finding those inactive to the backgrounds while active in the target region. To this end, the attention mechanism for the feature extracting framework is investigated comprising a scale-sensitive feature generation component and a discriminative feature generation module based on the gradients of regression and scoring losses. Comprehensive experiments have demonstrated that our pipeline obtains competitive results compared to recently published papers.

Predicting lymphovascular invasion in clinically node-negative breast cancer detected by abbreviated magnetic resonance imaging: Transfer learning vs. radiomics.

ObjectiveTo compare the performance of abbreviated breast magnetic resonance imaging (AB-MRI)-based transfer learning (TL) algorithm and radionics analysis for lymphovascular invasion (LVI) prediction in patients with clinically node-negative invasive breast cancer (IBC).MethodsBetween November 2017 and October 2020, 233 clinically node-negative IBCs detected by AB-MRI were retrospectively enrolled. One hundred thirty IBCs from center 1 (37 LVI-positive and 93 LVI-negative) were assigned as the training cohort and 103 from center 2 (25 LVI-positive and 78 LVI-negative) as the validation cohort. Based on AB-MRI, a TL signature (TLS) and a radiomics signature (RS) were built with the least absolute shrinkage and selection operator (LASSO) logistic regression. Their diagnostic performances were validated and compared using areas under the receiver operating curve (AUCs), net reclassification improvement (NRI), integrated discrimination improvement (IDI), decision curve analysis (DCA), and stratification analysis. A convolutional filter visualization technique was used to map the response areas of LVI on the AB-MRI.ResultsIn the validation cohort, compared with RS, the TLS showed better capability in discriminating LVI-positive from LVI-negative lesions (AUC: 0.852 vs. 0.726, p < 0.001; IDI = 0.092, p < 0.001; NRI = 0.554, p < 0.001). The diagnostic performance of TLS was not affected by the menstrual state, molecular subtype, or contrast agent type (all p > 0.05). Moreover, DCA showed that the TLS added more net benefit than RS for clinical utility.ConclusionsAn AB-MRI-based TLS was superior to RS for preoperative LVI prediction in patients with clinically node-negative IBC.

Compiling CNNs with Cain: focal-plane processing for robot navigation

Focal-plane Sensor-processors (FPSPs) are a camera technology that enables low power, high frame rate computation in the image sensor itself, making them suitable for edge computation. To fit into the sensor array, FPSPs are highly resource-constrained, with limited instruction set and few registers - which makes developing complex algorithms difficult. In this work, we present Cain, a compiler for convolutional filters that targets SCAMP-5, a general-purpose FPSP. Cain generates code to evaluate multiple convolutional kernels at the same time. It generates code that avoids the need for hardware multipliers, while orchestrating the exploitation of common sub-terms—leading to a large reduction in instruction count compared to both straightforward and prior optimized approaches. We demonstrate the capability enabled by Cain on SCAMP-5 with robotic navigation for near-sensor high-speed and low-power computation, by using Cain to implement a neural network on the focal plane.

Mapping of gold-related alteration minerals and linear structures using ASTER data in the Giyani Greenstone Belt, South Africa

This paper focuses on the mapping of alteration minerals and linear structures associated with gold mineralization in the Giyani Greenstone Belt (GGB). Spectral Information Divergence (SID) and Linear Spectral Unmixing (LSU) classification algorithms were applied to the ASTER data using image extracted endmember spectra which match the spectral profiles of predominant alteration minerals (biotite and calcite) related to gold mineralization in the GGB. Directional convolution filters were also applied to the ASTER data to extract structural lineaments that may be related to gold mineralization. In addition, Fry analysis of known gold occurrences in the GGB was conducted to determine the major distribution patterns along which gold mineralization have occurred. LSU and SID results show several pixels classified as alteration minerals corresponding to known gold occurrences. SID results also show minerals overlapping into the gneissic rock, which is a rare host of gold mineralization in the study area. On the other hand, LSU results show high abundances of calcite and biotite generally confined within the mafic-ultramafic greenstone rocks, which have been hypothesized as the source of gold-mineralized fluids in the GGB. In general, LSU show distinct broad regions classified as alteration minerals corresponding to known gold occurrences that have been previously reported to host these alteration minerals. Directional convolution filters enabled the extraction of predominately NE-SW oriented linear structures, including the major shear zone which is associated with gold mineralization in the GGB. Fry analysis revealed three major distribution patterns: N-S, NE-SW, ENE-WSW along which gold mineralization have occurred. These patterns correspond to major lineaments associated with gold mineralization reported in the previous structural studies of the GGB.

Improved Deep Multi-Patch Hierarchical Network For Handling Saturation In Image Deblurring

The most active study topic in the field of computer vision is dealing with saturation in image deblurring. The Deep Multi Patch Framework (DMPHN) is primarily utilized in dynamic scene deblurring, and its fine-to-coarse hierarchical representation results in lower processing costs. The contribution of the framework in gathering and processing local residual information of blur at the courser levels in the hierarchy makes the dynamic scene deblurring successful. The existing framework uses the traditional CNN with the convolutional filter, which is a generalized linear model (GLM) that goes along with a nonlinear activation function to scan the input. When the samples of latent concepts are not linearly separable then the linear CNN cannot abstract the good representations. Because saturation in an image contradicts the linear properties of the blur model, the present Deep Multi Patch architecture with linear CNN is unable to collect the necessary local information of blur in an image. In CNN, the higher-level layer maps to the larger regions of the input image by combining the lower-level concepts from each local patch. Due to the framework’s incapacity to retain information at lower levels, noticeable ringing artifacts and poor deblurring performance resulted. We designed a nonlinear function approximator for this section, which was influenced by the good abstraction capabilities of ”Network-in-Network”. The nonlinear features of saturated pixels and blur are successfully captured by this upgraded DMPHN with a nonlinear function approximator. Compared to the current DMPHN model, the suggested approach greatly reduced the number of trainable parameters while improving the time cost. On the GoPro, Synthetic, and Natural Saturated Blurred Image datasets, the upgraded DMPHN network achieves superior deblurring performance with better PSNR and SSIM than the state-of-the-art.

Online chatter detection considering beat effect based on Inception and LSTM neural networks

Chatter is an unstable and self-excited vibration that adversely affects part quality and tool life in various machining processes. To achieve high-performance machining, chatter identification has attracted considerable interest from many researchers in recent decades. Nevertheless, most existing chatter detection approaches fail to consider the presence of the beat effect, which is an interference pattern caused by slightly different chatter frequencies. The neglect of the beat effect would seriously degrade the effectiveness of these methods and even result in false alarms. In this paper, a novel deep neural network combining the Inception module, long short-term memory (LSTM) and residual networks (ILR-DNN) is proposed for online chatter detection considering the presence of the beat effect. The ILR-DNN automatically extracts insightful features from two branches. One branch is a two-layer LSTM network that can capture temporal characteristics of the chatter development process from the raw cutting force. In the other branch, a two-layer Inception network consisting of multiple convolutional kernel size filters extracts multiscale features from the FFT spectrum of the cutting force. Afterwards, the extracted features are concatenated and go through a residual network to alleviate the gradient disappearing during deep-layer network training. Finally, the fully-connected and softmax layers are employed to make a final classification. Machining tests are carried out, and cutting forces are collected to validate the feasibility and effectiveness of the proposed chatter detection method. Results show that the proposed ILR-DNN achieves much better performance than other methods in distinguishing machining states, i.e., stable machining, chatter with the beat effect, and chatter without the beat effect.

Anisotropic SpiralNet for 3D Shape Completion and Denoising.

Three-dimensional mesh post-processing is an important task because low-precision hardware and a poor capture environment will inevitably lead to unordered point clouds with unwanted noise and holes that should be suitably corrected while preserving the original shapes and details. Although many 3D mesh data-processing approaches have been proposed over several decades, the resulting 3D mesh often has artifacts that must be removed and loses important original details that should otherwise be maintained. To address these issues, we propose a novel 3D mesh completion and denoising system with a deep learning framework that reconstructs a high-quality mesh structure from input mesh data with several holes and various types of noise. We build upon SpiralNet by using a variational deep autoencoder with anisotropic filters that apply different convolutional filters to each vertex of the 3D mesh. Experimental results show that the proposed method enhances the reconstruction quality and achieves better accuracy compared to previous neural network systems.

Temporal Context Modeling Network with Local-Global Complementary Architecture for Temporal Proposal Generation

Temporal Action Proposal Generation (TAPG) is a promising but challenging task with a wide range of practical applications. Although state-of-the-art methods have made significant progress in TAPG, most ignore the impact of the temporal scales of action and lack the exploitation of effective boundary contexts. In this paper, we propose a simple but effective unified framework named Temporal Context Modeling Network (TCMNet) that generates temporal action proposals. TCMNet innovatively uses convolutional filters with different dilation rates to address the temporal scale issue. Specifically, TCMNet contains a BaseNet with dilated convolutions (DBNet), an Action Completeness Module (ACM), and a Temporal Boundary Generator (TBG). The DBNet aims to model temporal information. It handles input video features through different dilated convolutional layers and outputs a feature sequence as the input of ACM and TBG. The ACM aims to evaluate the confidence scores of densely distributed proposals. The TBG is designed to enrich the boundary context of an action instance. The TBG can generate action boundaries with high precision and high recall through a local–global complementary structure. We conduct comprehensive evaluations on two challenging video benchmarks: ActivityNet-1.3 and THUMOS14. Extensive experiments demonstrate the effectiveness of the proposed TCMNet on tasks of temporal action proposal generation and temporal action detection.

In-sensor optoelectronic computing using electrostatically doped silicon

Complementary metal–oxide–semiconductor (CMOS) image sensors allow machines to interact with the visual world. In these sensors, image capture in front-end silicon photodiode arrays is separated from back-end image processing. To reduce the energy cost associated with transferring data between the sensing and computing units, in-sensor computing approaches are being developed where images are processed within the photodiode arrays. However, such methods require electrostatically doped photodiodes where photocurrents can be electrically modulated or programmed, and this is challenging in current CMOS image sensors that use chemically doped silicon photodiodes. Here we report in-sensor computing using electrostatically doped silicon photodiodes. We fabricate thousands of dual-gate silicon p–i–n photodiodes, which can be integrated into CMOS image sensors, at the wafer scale. With a 3 × 3 network of the electrostatically doped photodiodes, we demonstrate in-sensor image processing using seven different convolutional filters electrically programmed into the photodiode network. A network of dual-gate silicon p–i–n photodiodes, which are compatible with complementary metal–oxide–semiconductor fabrication processes, can perform in-sensor image processing by being electrically programmed into convolutional filters.

Convolutional kernel function algebra

Many systems for image manipulation, signal analysis, machine learning, and scientific computing make use of discrete convolutional filters that are known before computation begins. These contexts benefit from common sub-expression elimination to reduce the number of calculations required, both multiplications and additions. We present an algebra for describing convolutional kernels and filters at a sufficient level of abstraction to enable intuitive common sub-expression based optimizations through decomposing filters into smaller, repeated, kernels. This enables the creation of an enormous search space of potential implementations of filters via algebraic manipulation. We demonstrate how integral image and sliding window optimizations can be expressed in the context of common sub-expression elimination as well as show the direct use case for this algebra in massively SIMD multiply-free contexts such as in cellular processor arrays. We then show that this algebra is general enough to express and optimize kernels that use non-standard semi-rings to enable shortest path algorithms.

A novel compact design of convolutional layers with spatial transformation towards lower-rank representation for image classification

Convolutional neural networks (CNNs) usually come with numerous parameters and thus are not convenient for some situations, such as when the storage space is limited. Low-rank decomposition is one effective way for network compression or compaction. However, the current methods are far from theoretical optimal compression performance because the low-rankness of the commonly trained convolution filter sets is limited because of the versatility of convolution filters. We propose a novel compact design for convolutional layers with spatial transformation for achieving a much lower-rank form. The convolution filters in our design are generated using a predefined Tucker product form, followed by learnable individual spatial transformations on each filter. The low-rank (Tucker) part lowers the parameter capacity while the transformation part enhances the feature representation capacity. We validate our proposed approach on an image classification task. Our approach focuses on compressing parameters while also improving accuracy. We perform experiments on the MNIST, CIFAR10, CIFAR100, and ImageNet datasets. On the ImageNet dataset, our approach outperforms low-rank based state-of-the-arts by 2% to 6% in top-1 validation accuracy. Furthermore, our approach outperforms a series of low-rank-based state-of-the-arts on various datasets. The experiments validate the efficacy of our proposed method. Our code is available at https://github.com/liubc17/low_rank_compact_transformed.

Nemo: An Open-Source Transformer-Supercharged Benchmark for Fine-Grained Wildfire Smoke Detection

Deep-learning (DL)-based object detection algorithms can greatly benefit the community at large in fighting fires, advancing climate intelligence, and reducing health complications caused by hazardous smoke particles. Existing DL-based techniques, which are mostly based on convolutional networks, have proven to be effective in wildfire detection. However, there is still room for improvement. First, existing methods tend to have some commercial aspects, with limited publicly available data and models. In addition, studies aiming at the detection of wildfires at the incipient stage are rare. Smoke columns at this stage tend to be small, shallow, and often far from view, with low visibility. This makes finding and labeling enough data to train an efficient deep learning model very challenging. Finally, the inherent locality of convolution operators limits their ability to model long-range correlations between objects in an image. Recently, encoder–decoder transformers have emerged as interesting solutions beyond natural language processing to help capture global dependencies via self- and inter-attention mechanisms. We propose Nemo: a set of evolving, free, and open-source datasets, processed in standard COCO format, and wildfire smoke and fine-grained smoke density detectors, for use by the research community. We adapt Facebook’s DEtection TRansformer (DETR) to wildfire detection, which results in a much simpler technique, where the detection does not rely on convolution filters and anchors. Nemo is the first open-source benchmark for wildfire smoke density detection and Transformer-based wildfire smoke detection tailored to the early incipient stage. Two popular object detection algorithms (Faster R-CNN and RetinaNet) are used as alternatives and baselines for extensive evaluation. Our results confirm the superior performance of the transformer-based method in wildfire smoke detection across different object sizes. Moreover, we tested our model with 95 video sequences of wildfire starts from the public HPWREN database. Our model detected 97.9% of the fires in the incipient stage and 80% within 5 min from the start. On average, our model detected wildfire smoke within 3.6 min from the start, outperforming the baselines.

Discriminative Convolutional Sparse Coding of ECG Signals for Automated Recognition of Cardiac Arrhythmias

Electrocardiogram (ECG) is a common and powerful tool for studying heart function and diagnosing several abnormal arrhythmias. In this paper, we present a novel classification model that combines the discriminative convolutional sparse coding (DCSC) framework with the linear support vector machine (LSVM) classification strategy. In the training phase, most existing convolutional sparse coding frameworks are unsupervised in the sense that label information is ignored in the convolutional filter training stage. In this work, we explicitly incorporate a label consistency constraint called “discriminative sparse-code error” into the objective function to learn discriminative dictionary filters for sparse coding. The learned dictionary filters encourage signals from the same class to have similar sparse codes, and signals from different classes to have dissimilar sparse codes. To reduce the computational complexity, we propose to perform a max-pooling operation on the sparse coefficients. Using LSVM as a classifier, we examine the performance of the proposed classification system on the MIT-BIH arrhythmia database in accordance with the AAMI EC57 standard. The experimental results show that the proposed DCSC + LSVM algorithm can obtain 99.32% classification accuracy for cardiac arrhythmia recognition.

Application of Intelligent Analysis Technology of Football Video Based on Online Target Tracking Algorithm of Motion Characteristics in Football Training.

Because the scene of football video is fixed and simple, the events in football video, such as shooting and offside , also have clear semantics. At the same time, they have sufficient domain knowledge and have broad application prospects. The research framework of sports video intelligence analysis is usually regarded as a three-level framework, namely, the low-level feature layer, the middle-level key primitive generation layer, and the high-level event analysis layer. A MF_O2T (Moving Feature Online Target Tracking) algorithm is proposed. First, based on the marked first frame image, this algorithm extracts a set of standardized local images from the target areas of visible and infrared images as target convolution filters, removes the main color of the stadium by using HSV color space nonuniform quantization algorithm, and extracts the histogram of the main color of players in upper and lower blocks. Experimental results show that the algorithm designed in this paper has strong robustness, can better adapt to player tracking in different scenes of football video, and meets the real-time requirements of football training.

Deep learning neural network for lung cancer classification: enhanced optimization function

Convolutional neural network is widely used for image recognition in the medical area at nowadays. However, overall accuracy in predicting lung tumor is low and the processing time is high as the error occurred while reconstructing the CT image. The aim of this work is to increase the overall prediction accuracy along with reducing processing time by using multispace image in pooling layer of convolution neural network. The proposed method has the autoencoder system to improve the overall accuracy, and to predict lung cancer by using multispace image in pooling layer of convolution neural network and Adam Algorithm for optimization. First, the CT images were pre-processed by feeding image to the convolution filter and down sampled by using max pooling. Then, features are extracted using the autoencoder model based on convolutional neural network and multispace image reconstruction technique is used to reduce error while reconstructing the image which then results improved accuracy to predict lung nodule. Finally, the reconstructed images are taken as input for SoftMax classifier to classify the CT images. The state-of-art and proposed solutions were processed in Python Tensor Flow and It provides significant increase in accuracy in classification of lung cancer to 99.5 from 98.9 and decrease in processing time from 10 frames/second to 12 seconds/second. The proposed solution provides high classification accuracy along with less processing time compared to the state of art. For future research, large dataset can be implemented, and low pixel image can be processed to evaluate the classification.

Research on Segmentation of Brain Tumor in MRI Image Based on Convolutional Neural Network.

Brain tumors are the brain diseases with the highest mortality and prevalence, and magnetic resonance imaging has high-resolution and multiparameter. As the basis for realizing the quantitative analysis of brain tumors, automatic segmentation plays a vital role in diagnosis and treatment. A new network model is proposed to improve the accuracy of convolutional neural network segmentation of brain tumor regions and control the parameter space scale of the network model. The model first uses a convolutional layer composed of a series of 3D convolution filters to construct a backbone network for feature learning of input 3D MRI image blocks. Then, a pyramid structure constructed by a 3D convolutional layer is designed to extract and fuse features of tumor lesions and context information of different scales and then classify the fused feature at the voxel level to obtain segmentation results. Finally, a conditional random field is used to postprocess segmentation results for structured refinement. By designing massive ablation experiments to analyze the sensitivity of the essential modules of the comparison network, the results confirm that our method can better solve the problems faced by the traditional fully connected convolutional neural network.

Development of an intention estimation model based on gaze and face information for electric wheelchair operation support

AbstractIn recent years, various user interfaces have been developed to meet the diverse needs of physically disabled persons. In this paper, we developed a method to identify gazing and facial movements based on gaze time and eye/face information and developed an electric wheelchair that can be operated with the user's natural gazing and facial movements. This intention estimation model is composed of 1DCNN and LSTM layers. First, 1DCNN is used to extract features from gaze and face information, and then the extracted features are input to LSTM to estimate the user's intentions regarding movement. The evaluation experiments suggest that the combination of gaze and face information improves the estimation accuracy and contributes to the classification. Furthermore, it was confirmed that adding a convolutional filter layer to the LSTM layer improved the accuracy.

Sign Language Recognition Based on R(2+1)D With Spatial–Temporal–Channel Attention

Previous work utilized three-dimensional (3-D) convolutional neural networks (CNNs) tomodel the spatial appearance and temporal evolution concurrently for sign language recognition (SLR) and exhibited impressive performance. However, there are still challenges for 3-D CNN-based methods. First, motion information plays a more significant role than spatial content in sign language. Therefore, it is still questionable whether to treat space and time equally and model them jointly by heavy 3-D convolutions in a unified approach. Second, because of the interference from the highly redundant information in sign videos, it is still nontrivial to effectively extract discriminative spatiotemporal features related to sign language. In this study, deep R(2+1)D was adopted for separate spatial and temporal modeling and demonstrated that decomposing 3-D convolution filters into independent spatial and temporal convolutions facilitates the optimization process in SLR. A lightweight spatial–temporal–channel attention module, including two submodules called channel–temporal attention and spatial–temporal attention, was proposed to make the network concentrate on the significant information along spatial, temporal, and channel dimensions by combining squeeze and excitation attention with self-attention. By embedding this module into R(2+1)D, superior or comparable results to the state-of-the-art methods on the CSL-500, Jester, and EgoGesture datasets were obtained, which demonstrated the effectiveness of the proposed method.

Inner-Imaging Networks: Put Lenses Into Convolutional Structure.

Despite the tremendous success in computer vision, deep convolutional networks suffer from serious computation costs and redundancies. Although previous works address that by enhancing the diversities of filters, they have not considered the complementarity and the completeness of the internal convolutional structure. To respond to this problem, we propose a novel inner-imaging (InI) architecture, which allows relationships between channels to meet the above requirement. Specifically, we organize the channel signal points in groups using convolutional kernels to model both the intragroup and intergroup relationships simultaneously. A convolutional filter is a powerful tool for modeling spatial relations and organizing grouped signals, so the proposed methods map the channel signals onto a pseudoimage, like putting a lens into the internal convolution structure. Consequently, not only is the diversity of channels increased but also the complementarity and completeness can be explicitly enhanced. The proposed architecture is lightweight and easy to be implement. It provides an efficient self-organization strategy for convolutional networks to improve their performance. Extensive experiments are conducted on multiple benchmark datasets, including CIFAR, SVHN, and ImageNet. Experimental results verify the effectiveness of the InI mechanism with the most popular convolutional networks as the backbones.

An improved capsule network for glioma segmentation on MRI images: A curriculum learning approach

Glioma segmentation is an essential step in tumor identification and treatment planning. Glioma segmentation is a challenging task because it appears with blurred and irregular boundaries in a variety of shapes. In this paper, we propose an efficient and novel model for automatic glioma segmentation based on capsule neural networks. We improved the architecture and training of the SegCaps model, the first capsule-based segmentation network. The proposed architecture is improved by introducing dilation blocks in the primary capsule block to get deeper features while avoiding resolution reduction. The prediction layer of the network is also modified using one-dimensional convolution filters, enabling the network to not only maximize tumor existence likelihood but also regularize the capsule orientations within the tumor. Our main contribution, however, is to introduce an enhanced curriculum-based training algorithm into the proposed SegCaps model. We adapt the curriculum learning for the model by suggesting a new pacing mechanism based on a roulette-wheel selection algorithm that enriches randomness in the network and prevents bias. A hybrid dice loss function is also employed, which is better adapted to the introduced curriculum-based training procedure. We evaluated the performance of improved SegCaps on the BraTS2020, a multimodal benchmark dataset for brain tumor segmentation. The experimental results confirmed that the improvements yield a top-performing yet memory-efficient deep capsule architecture. The proposed model outperformed the best-reported accuracies on the BraTS2020, achieving improved dice scores of 85.16% and 81.88% for tumor core and enhancing tumor segmentation, respectively. Using 90%, fewer parameters than the popular U-Net also confirmed the high memory efficiency of our proposed model.