Types Of Convolutional Neural Networks Research Articles

Implementation on FingerScripter: Converting Fingerspelling into Text and Speech

Abstract: Communication might seem a simple act for normal people, but for the ones who are suffering from physical or psychological disability find it very difficult for communicating with others. This communication barrier needs to be removed for which there are various techniques that are being used, out of which some are verbal, non-verbal, visual and written. For mute and deaf people these communication barrier is a major problem which can adversely affect their lives adding to their already challenging lives. For this sign languages are the best possible option, as they bridge the gap between communication of these deaf and mute people with normal people. We have created a machine learning model in order to solve this issue which will identify these signs and interpret them to detect which alphabet is signified. The model will be detecting the American Sign Language (ASL) which is one of the most widely used sign language and a type of fingerspelling. The model will form words out of the identified alphabets which will also be then converted into speech. The key algorithm which will be used is a special type of Convolutional Neural Network (CNN) for image recognition. All the documentation regarding the model along with its working is provided in this paper. With high precision and accuracy the model aims to ease the communication of deaf and mute people.

Wide-TSNet: A Novel Hybrid Approach for Bitcoin Price Movement Classification

In this paper, we introduce Wide-TSNet, a novel hybrid approach for predicting Bitcoin prices using time-series data transformed into images. The method involves converting time-series data into Markov transition fields (MTFs), enhancing them using histogram equalization, and classifying them using Wide ResNets, a type of convolutional neural network (CNN). We propose a tripartite classification system to accurately represent Bitcoin price trends. In addition, we demonstrate the effectiveness of Wide-TSNet through various experiments, in which it achieves an Accuracy of approximately 94% and an F1 score of 90%. It is also shown that lightweight CNN models, such as SqueezeNet and EfficientNet, can be as effective as complex models under certain conditions. Furthermore, we investigate the efficacy of other image transformation methods, such as Gramian angular fields, in capturing the trends and volatility of Bitcoin prices and revealing patterns that are not visible in the raw data. Moreover, we assess the effect of image resolution on model performance, emphasizing the importance of this factor in image-based time-series classification. Our findings explore the intersection between finance, image processing, and deep learning, providing a robust methodology for financial time-series classification.

Convolutional Neural Network Approaches in Median Nerve Morphological Assessment from Ultrasound Images.

Ultrasound imaging has been used to investigate compression of the median nerve in carpal tunnel syndrome patients. Ultrasound imaging and the extraction of median nerve parameters from ultrasound images are crucial and are usually performed manually by experts. The manual annotation of ultrasound images relies on experience, and intra- and interrater reliability may vary among studies. In this study, two types of convolutional neural networks (CNNs), U-Net and SegNet, were used to extract the median nerve morphology. To the best of our knowledge, the application of these methods to ultrasound imaging of the median nerve has not yet been investigated. Spearman's correlation and Bland-Altman analyses were performed to investigate the correlation and agreement between manual annotation and CNN estimation, namely, the cross-sectional area, circumference, and diameter of the median nerve. The results showed that the intersection over union (IoU) of U-Net (0.717) was greater than that of SegNet (0.625). A few images in SegNet had an IoU below 0.6, decreasing the average IoU. In both models, the IoU decreased when the median nerve was elongated longitudinally with a blurred outline. The Bland-Altman analysis revealed that, in general, both the U-Net- and SegNet-estimated measurements showed 95% limits of agreement with manual annotation. These results show that these CNN models are promising tools for median nerve ultrasound imaging analysis.

Redundant node identification using Siamese neural network to enhance the availability in visual sensor network

SummaryMany monitoring applications related to surveillance, tracking and multipurpose visual monitoring have taken into consideration the use of wireless visual sensor networks. When sensors are deployed over a monitored field that could potentially damage the monitoring capability and availability in the visual sensor network. In order to overcome these issues, faulty nodes are identified and replaced by redundant node based on Siamese network in this research. Initially, camera nodes are randomly deployed in the visual sensor network, and the data are received from the network through gateway. To identify the redundant nodes, initially, the frames are divided into equivalent time slot, and then, Siamese network is utilized to identify the redundant nodes in a network. Siamese neural network is type of convolutional neural network that is utilized to recognize the similar images in the network. After that, faulty nodes are identified based on some parameters such as entropy, energy, transmission delay and network coverage. If the average energy, entropy, transmission delay and network coverage are below the threshold value, then the node is identified as faulty node. Finally, replace the faulty node with redundant node to enhance the availability in the visual sensor network for critical monitoring applications. The simulation analysis shows that the developed approach takes 772 s to identify redundant node and take 27 s to identify fake nodes and the developed method is executed at 1308 s. Thus, this prediction model helps to improve the coverage quality of target‐based monitoring in order to achieve availability.

Time-Constrained Ensemble Sensing With Heterogeneous IoT Devices in Intelligent Transportation Systems

Recently we have witnessed the rise of Artificial Intelligence of Things (AIoT) and the shift of sensing paradigm from cloud-centric to the edge-centric, which effectively improves the sensing capability of intelligence transportation systems. To improve the real-time sensing performance, in this work we propose an ensemble sensing based scheme to solve the time-constraint synchronized inference problem and achieve robust inference with heterogeneous IoT devices in intelligence transportation systems. We design and implement Ensen, which incorporates various novel techniques such as customized DNN model design, KD-based model training, and dynamic deep ensemble management, etc., to achieve improved accuracy and maximize the computational resource usage of the whole sensing group. Extensive evaluations on different types of common IoT devices have shown that Ensen achieves a robust performance and can be easily extended to different types of convolutional neural networks.

Feature recognition in multiple CNNs using sEMG images from a prototype comfort test

ObjectiveDeep learning-based CNN networks have recently been investigated to solve the problem of body posture recognition based on surface electromyographic signals (sEMG). Influenced by these studies, to develop a combined approach of sEMG and CNNs in the study of human-product interactions and the impact of body comfort, and to compare the advantages and disadvantages of various CNNs networks. MethodsIn this study, sEMG measurements were carried out by building a prototype usability experiment, and the data were divided into four categories, with two types of datasets: training and testing. Four CNNs, LeNet-5, VGGNet-11, InceptionNet V4, and DenseNet, were used for the recognition of sEMG images. ResultsDenseNet is another type of convolutional neural network with deep layers, which has a unique advantage over other algorithms. unique advantages over other algorithms. DenseNet has fewer layers and better accuracy than InceptionNet V4, but not only does it bypass enhanced feature reuse, but its network is easier to train and has some regularization effects, while also mitigating the problems of gradient disappearance and model degradation. ConclusionThese findings could lead to a more appropriate CNN model and a useful tool for developing comfort judgments of surface EMG signals, furthering the development of products that come into contact with the human body without the need for routine retraining.

Florets for Chiplets: Data Flow-aware High-Performance and Energy-efficient Network-on-Interposer for CNN Inference Tasks

Recent advances in 2.5D chiplet platforms provide a new avenue for compact scale-out implementations of emerging compute- and data-intensive applications including machine learning. Network-on-Interposer (NoI) enables integration of multiple chiplets on a 2.5D system. While these manycore platforms can deliver high computational throughput and energy efficiency by running multiple specialized tasks concurrently, conventional NoI architectures have a limited computational throughput due to their inherent multi-hop topologies. In this paper, we propose Floret, a novel NoI architecture based on space-filling curves (SFCs). The Floret architecture leverages suitable task mapping, exploits the data flow pattern, and optimizes the inter-chiplet data exchange to extract high performance for multiple types of convolutional neural network (CNN) inference tasks running concurrently. We demonstrate that the Floret architecture reduces the latency and energy up to 58% and 64%, respectively, compared to state-of-the-art NoI architectures while executing datacenter-scale workloads involving multiple CNN tasks simultaneously. Floret achieves high performance and significant energy savings with much lower fabrication cost by exploiting the data-flow awareness of the CNN inference tasks.

Deep Learning-Based Identification of Common Complication Features of Surgical Incisions

In recent years, due to the development of accelerated recovery after surgery and day surgery in the field of surgery, the average length-of-stay of patients has been shortened and patients stay at home for post-surgical recovery and healing of the surgical incisions. In order to identify, in a timely manner, the problems that may appear at the incision site and help patients prevent or reduce the anxiety they may experience after discharge, we used deep learning method in this study to classify the features of common complications of surgical incisions, hoping to realize patient-directed early identification of complications common to surgical incisions. A total of 1 224 postoperative photographs of patients' surgical incisions were taken and collected at a tertiary-care hospital between June 2021 and March 2022. The photographs were collated and categorized according to different features of complications of the surgical incisions. Then, the photographs were divided into training, validation, and test sets at the ratio of 8∶1∶1 and 4 types of convolutional neural networks were applied in the training and testing of the models. Through the training of multiple convolutional neural networks and the testing of the model performance on the basis of a test set of 300 surgical incision images, the average accuracy of the four ResNet classification network models, SE-ResNet101, ResNet50, ResNet101, and SE-ResNet50, for surgical incision classification was 0.941, 0.903, 0.896, and 0.918, respectively, the precision was 0.939, 0.898, 0.868, and 0.903, respectively, and the recall rate was 0.930, 0.880, 0.850, and 0.894, respectively, with the SE-Resnet101 network model showing the highest average accuracy of 0.941 for incision feature classification. Through the combined use of deep learning technology and images of surgical incisions, problematic features of surgical incisions can be effectively identified by examining surgical incision images. It is expected that patients will eventually be able to perform self-examination of surgical incisions on smart terminals.

Convolutional neural networks applied to microtomy: Identifying the trimming-end cutting routine on paraffin-embedded tissue blocks

In the field of histopathology, the microtomy procedure yields thin sections of tissue embedded in paraffin blocks, then to be further processed for diagnostic purposes. Within microtomy, trimming is an initial but critical process in which the excess paraffin covering the tissue of interest is removed by continuous cutting routines, until the tissue is suitably exposed and ready for sectioning. Trimming is currently a time-consuming process that is manually held by technicians. In this paper, we present a method to automatize this process, by analyzing tissue block surface images resulting from each cyclic cutting routine. Two types of Convolutional Neural Networks (CNNs) were fine-tuned: one for binary segmentation, the other for multi-class classification tasks, by exploring and optimizing lightweight architectures to provide fast analytical results on cost-effective edge computing. Two sequential online conditions followed the CNNs to output the current stage of the block surface and rule if the trimming-end cutting routine was reached. We compared the results obtained through our method with the ones obtained by three skilled technicians processing 75 tissue blocks. The proposed method identified the trimming-end cutting routine approximately as accurate as the technicians did, yielding up to 90% of trimmed blocks of optimal quality and evidencing the potential of this tool in future automated trimming instruments. We deployed our method to an Edge TPU hardware accelerator to showcase its capability to provide immediate and objective results at every microtomy station applied with an ad hoc hardware, potentially guaranteeing a throughput 50% higher than manual trimming.

Wind Power Forecasting Based on WaveNet and Multitask Learning

Accurately predicting the power output of wind turbines is crucial for ensuring the reliable and efficient operation of large-scale power systems. To address the inherent limitations of physical models, statistical models, and machine learning algorithms, we propose a novel framework for wind turbine power prediction. This framework combines a special type of convolutional neural network, WaveNet, with a multigate mixture-of-experts (MMoE) architecture. The integration aims to overcome the inherent limitations by effectively capturing and utilizing complex patterns and trends in the time series data. First, the maximum information coefficient (MIC) method is applied to handle data features, and the wavelet transform technique is employed to remove noise from the data. Subsequently, WaveNet utilizes its scalable convolutional network to extract representations of wind power data and effectively capture long-range temporal information. These representations are then fed into the MMoE architecture, which treats multistep time series prediction as a set of independent yet interrelated tasks, allowing for information sharing among different tasks to prevent error accumulation and improve prediction accuracy. We conducted predictions for various forecasting horizons and compared the performance of the proposed model against several benchmark models. The experimental results confirm the strong predictive capability of the WaveNet–MMoE framework.

(433) Eros and (25143) Itokawa surface properties from reflectance spectra

Context. Our knowledge of near-Earth asteroid (NEA) composition is important for planetary research, planetary defence, and future in-space resource utilisation. Upcoming space missions, for example, Hera, M-ARGO, or missions to the asteroid (99942) Apophis, will provide us with surface-resolved NEA reflectance spectra. Neural networks are useful tools for analysing reflectance spectra and determining material composition with high precision and low processing time. Aims. We applied neural-network models on disk-resolved spectra of the Eros and Itokawa asteroids observed by the NEAR Shoemaker and Hayabusa spacecraft. With this approach, the mineral variations or intensity of space weathering can be mapped. Methods. We built and tested two types of convolutional neural networks (CNNs). The first one was trained using asteroid reflectance spectra with known taxonomy classes. The other one used silicate reflectance spectra with assigned mineral abundances and compositions. Results. The reliability of the classification model depends on the resolution of reflectance spectra. Typical F1 score and Cohen's κC values decrease from about 0.90 for high-resolution spectra to about 0.70 for low-resolution spectra. The predicted silicate composition does not strongly depend on spectrum resolution and coverage of the 2-µm band of pyroxene. The typical root mean square error is between 6 and 10 percentage points. For the Eros and Itokawa asteroids, the predicted taxonomy classes favour the S-type and the predicted surface compositions are homogeneous and correspond to the composition of L/LL and LL ordinary chondrites, respectively. On the Itokawa surface, the model identified fresh spots that were connected with craters or coarse-grain areas. Conclusions. The neural network models trained with measured spectra of asteroids and silicate samples are suitable for deriving surface silicate mineralogy with a reasonable level of accuracy. The predicted surface mineralogy is comparable to the mineralogy of returned samples measured in the laboratory. Moreover, the taxonomical predictions can point out locations of fresher areas.

Patient selection for proton therapy using Normal Tissue Complication Probability with deep learning dose prediction for oropharyngeal cancer.

In cancer care, determining the most beneficial treatment technique is a key decision affecting the patient's survival and quality of life. Patient selection for proton therapy (PT) over conventional radiotherapy (XT) currently entails comparing manually generated treatment plans, which requires time andexpertise. We developed an automatic and fast tool, AI-PROTIPP (Artificial Intelligence Predictive Radiation Oncology Treatment Indication to Photons/Protons), that assesses quantitatively the benefits of each therapeutic option. Our method uses deep learning (DL) models to directly predict the dose distributions for a given patient for both XT and PT. By using models that estimate the Normal Tissue Complication Probability (NTCP), namely the likelihood of side effects to occur for a specific patient, AI-PROTIPP can propose a treatment selection quickly andautomatically. A database of 60 patients presenting oropharyngeal cancer, obtained from the Cliniques Universitaires Saint Luc in Belgium, was used in this study. For every patient, a PT plan and an XT plan were generated. The dose distributions were used to train the two dose DL prediction models (one for each modality). The model is based on U-Net architecture, a type of convolutional neural network currently considered as the state of the art for dose prediction models. A NTCP protocol used in the Dutch model-based approach, including grades II and III xerostomia and grades II and III dysphagia, was later applied in order to perform automatic treatment selection for each patient. The networks were trained using a nested cross-validation approach with 11-folds. We set aside three patients in an outer set and each fold consists of 47 patients in training, five in validation and five for testing. This method allowed us to assess our method on 55 patients (five patients per test times the number of folds). The treatment selection based on the DL-predicted doses reached an accuracy of 87.4% for the threshold parameters set by the Health Council of the Netherlands. The selected treatment is directly linked with these threshold parameters as they express the minimal gain brought by the PT treatment for a patient to be indicated to PT. To validate the performance of AI-PROTIPP in other conditions, we modulated these thresholds, and the accuracy was above 81% for all the considered cases. The difference in average cumulative NTCP per patient of predicted and clinical dose distributions is very similar (less than 1% difference). AI-PROTIPP shows that using DL dose prediction in combination with NTCP models to select PT for patients is feasible and can help to save time by avoiding the generation of treatment plans only used for the comparison. Moreover, DL models are transferable, allowing, in the future, experience to be shared with centers that would not have PT planningexpertise.

Automatic Modulation Recognition of Radiation Source Signals Based on Data Rearrangement and the 2D FFT

It is a challenge for automatic modulation recognition (AMR) methods for radiation source signals to work in environments with low signal-to-noise ratios (SNRs). This paper proposes a modulation feature extraction method based on data rearrangement and the 2D fast Fourier transform (FFT) (DR2D), and a DenseNet feature extraction network with early fusion is constructed to recognize the extracted modulation features. First, the input signal is preprocessed by DR2D to obtain three types of joint frequency feature bins with multiple time scales. Second, the feature fusion operation is performed on the inputs of the different layers of the proposed network. Finally, feature recognition is completed in the subsequent layers. The theoretical analysis and simulation results show that DR2D is a fast and robust preprocessing method for extracting the features of modulated radiation source signals with less computational complexity. The proposed DenseNet feature extraction network with early fusion can identify the extracted modulation features with less spatial complexity than other types of convolutional neural networks (CNNs) and performs well in low-SNR environments.

Vision-Based System for Measuring the Diameter of Wood Logs

Detecting and measuring objects with vision-based systems in uncontrolled environments is a difficult task that today, thanks to the development of increasingly advanced artificial intelligence-based techniques, can be solved with greater ease. In this context, the paper proposes a novel approach for the vision-based measurement of objects in uncontrolled environments using a specific type of Convolutional Neural Network (CNN) named YOLO (You Only Look Once) and a Direct Linear Transformation (DLT) process. The case study concerned designing a novel vision-based system for measuring the diameter of wood logs cut and loaded onto trucks. This problem has been occurring in the Swedish forestry industry. In fact, this operation is not carried out with computer vision algorithms because of the high variability of environmental conditions caused by the changing position of the sun, weather conditions, and the variability of truck positioning. To solve this problem, the YOLO network is proposed to locate logs while attempting to maintain a high Intersection over Union (IoU) value for the correct estimation of log size. Furthermore, in order to obtain accurate measurements, the DLT is used to convert into world coordinates the dimensions of the logs themselves. The proposed CNN-based solution is described after briefly introducing today’s methodologies adopted for wood bundle analysis. Particular attention is paid to both the training and the calibration steps. Results report that for 80% of cases, the error reported has been smaller than 4 cm, representing only 8% of the measurement, considering a mean log diameter for the application of 50 cm.

Deep learning for inverse problems with unknown operator

We consider ill-posed inverse problems where the forward operator T is unknown, and instead we have access to training data consisting of functions fi and their noisy images Tfi. This is a practically relevant and challenging problem which current methods are able to solve only under strong assumptions on the training set. Here we propose a new method that requires minimal assumptions on the data, and prove reconstruction rates that depend on the number of training points and the noise level. We show that, in the regime of “many” training data, the method is minimax optimal. The proposed method employs a type of convolutional neural networks (U-nets) and empirical risk minimization in order to “fit” the unknown operator. In a nutshell, our approach is based on two ideas: the first is to relate U-nets to multiscale decompositions such as wavelets, thereby linking them to the existing theory, and the second is to use the hierarchical structure of U-nets and the low number of parameters of convolutional neural nets to prove entropy bounds that are practically useful. A significant difference with the existing works on neural networks in nonparametric statistics is that we use them to approximate operators and not functions, which we argue is mathematically more natural and technically more convenient.

Biomedical Diagnosis of Breast Cancer Using Deep Learning and Multiple Classifiers.

Breast cancer is considered one of the deadliest diseases in women. Due to the risk and threat it poses, the world has agreed to hold a breast cancer awareness day in October, encouraging women to perform mammogram inspections. This inspection may prevent breast-cancer-related deaths or reduce the death rate. The identification and classification of breast cancer are challenging tasks. The most commonly known procedure of breast cancer detection is performed by using mammographic images. Recently implemented algorithms suffer from generating accuracy below expectations, and their computational complexity is high. To resolve these issues, this paper proposes a fully automated biomedical diagnosis system of breast cancer using an AlexNet, a type of Convolutional Neural Network (CNN), and multiple classifiers to identify and classify breast cancer. This system utilizes a neuro-fuzzy method, a segmentation algorithm, and various classifiers to reach a higher accuracy than other systems have achieved. Numerous features are extracted to detect and categorize breast cancer. Three datasets from Kaggle were tested to validate the proposed system. The performance evaluation is performed with quantitative and qualitative accuracy, precision, recall, specificity, and F-score. In addition, a comparative assessment is performed between the proposed system and some works of literature. This assessment shows that the presented algorithm provides better classification results and outperforms other systems in all parameters. Its average accuracy is over 98.6%, while other metrics are more than 98%. This research indicates that this approach can be applied to assist doctors in diagnosing breast cancer correctly.

Automated Registration for Dual-View X-Ray Mammography Using Convolutional Neural Networks.

Automated registration algorithms for a pair of 2D X-ray mammographic images taken from two standard imaging angles, namely, the craniocaudal (CC) and the mediolateral oblique (MLO) views, are developed. A fully convolutional neural network, a type of convolutional neural network (CNN), is employed to generate a pixel-level deformation field, which provides a mapping between masses in the two views. Novel distance-based regularization is employed, which contributes significantly to the performance. The developed techniques are tested using real 2D mammographic images, slices from real 3D mammographic images, and synthetic mammographic images. Architectural variations of the neural network are investigated and the performance is characterized from various aspects including image resolution, breast density, lesion size, lesion subtlety, and lesion Breast Imaging-Reporting and Data System (BI-RADS) category. Our network outperformed the state-of-the-art CNN-based and non-CNN-based registration techniques, and showed robust performance across various tissue/lesion characteristics. The proposed methods provide a useful automated tool for co-locating lesions between the CC and MLO views even in challenging cases. Our methods can aid clinicians to establish lesion correspondence quickly and accurately in the dual-view X-ray mammography, improving diagnostic capability.

Automated bow shock and magnetopause boundary detection with Cassini using threshold and deep learning methods

Two algorithms set for automatic detection of bow shock (BS) and magnetopause (MP) boundaries at Saturn using in situ magnetic field and plasma data acquired by the Cassini spacecraft are presented. Traditional threshold-based and modern deep learning algorithms were investigated for the task of boundary detection. Sections of Cassini’s orbits were pre-selected based on empirical BS and MP boundary models, and from outlier detection in magnetic field data using an autoencoder neural network. The threshold method was applied to pre-selected magnetic field and plasma data independently to compute parameters to which a threshold was applied to determine the presence of a boundary. The deep learning method used a type of convolutional neural network (CNN) called ResNet on images of magnetic field time series data and electron energy-time spectrograms to classify the presence of boundaries. 2012 data were held out of the training data to test and compare the algorithms on unseen data. The comparison showed that the CNN method applied to plasma data outperformed the threshold method. A final multiclass CNN classifier trained on plasma data obtained F1 scores of 92.1% ± 1.4% for BS crossings and 84.7% ± 1.9% for MP crossings on a corrected test dataset (from use of a bootstrap method). Reliable automated detection of boundary crossings could enable future spacecraft experiments like the PEP instrument on the upcoming JUICE spacecraft mission to dynamically adapt the best observing mode based on rapid classification of the boundary crossings as soon as it appears, thus yielding higher quality data and improved potential for scientific discovery.

Analysis of the Microcontroller Resources Using Specifics for Speech Recognition

The use of neural networks for information recognition, in particular, voice, expands the functional capabilities of embedded systems on microcontrollers. But it is necessary to take into account the limitations of the microcontroller resources. The purpose of the work is to analyze the impact of voice processing parameters and neural network architecture on the degree of microcontroller resources usage. To do this, a database of samples of the keyword, samples of other words and voices, and samples of noise are created, the probability of recognizing the keyword among other words and noises is evaluated, the dependence of the amount of memory used on the microcontroller and the decision-making time on the number MFC coefficients is established, the dependence of the amount of used memory of the microcontroller and the decision-making time on the type of convolutional neural network is established also. During the experiment, the Arduino Nano 33 BLE Sense development board was used. The neural network model was built and trained on the Edge Impulse software platform. To conduct the experiment, three groups of data with the names "hello", "unknown", "noise" were created. The group "hello" contains 94 examples of the word "hello" in English, spoken by a female voice. The "unknown" group contains 167 examples of other words pronounced by both female and male voices. The "noise" group contains 166 samples of noise and random sounds. According to Edge Impulse's recommendation, 80% of the samples from each of the data groups were used to train the neural network model, and 20% of the samples were used for testing. Analysis of the results shows that with an increase in the number of MFC coefficients and, accordingly, the accuracy of keyword recognition, the amount of program memory occupied by the code increases by 480 bytes (less than 1%). For the nRF52840 microcontroller, this is not a significant increase. The amount of RAM used during the experiment did not change. Although the calculation time of the accuracy of the code word definition increased by only 14 ms (less than 5%) with the increase in the number of MFC coefficients, the calculation procedure is quite long (approximately 0.3 s) compared to the sound sample length of 1 s. This can be a certain limitation when processing a sound signal with 32-bit microcontrollers. To analyze phrases or sentences, it is necessary to use more powerful microcontrollers or microprocessors. Based on the results of experimental research, it can be stated that the computing resources of 32-bit microcontrollers are quite sufficient for recognizing voice commands with the possibility of pre-digital processing of the sound signal, in particular, the use of low-frequency cepstral coefficients. The selection of the number of coefficients does not significantly affect the amount of used FLASH and RAM memory of the nRF52840 microcontroller. The comparison results show the superiority of the 2D network in the accuracy of the keyword definition for both 12 and 13 MFC coefficients. The use of a one-dimensional convolutional neural network for voice sample recognition in the conducted experiment provides memory savings of approximately 5%. The quality of keyword recognition with the number of MFC coefficients of 12 is approximately 0.7. For 17 MFC coefficients, the recognition quality is already 0.97. The amount of RAM used in the case of the 2D network has decreased slightly. Voice sample processing time for both types of networks is practically the same. Thus, 1D convolutional neural networks have certain advantages in microcontroller applications for voice processing and recognition. The limitation of voice recognition on the microcontroller is the sufficiently long processing time of the sound sample (approximately 0.3 s) with the duration of the sample itself being 1 s, this can be explained by a sufficiently low clock frequency of 64 MHz. Increasing the clock frequency will reduce the calculation time.

On-sensor binarized CNN inference with dynamic model swapping in pixel processor arrays.

Many types of Convolutional Neural Network (CNN) models and training methods have been proposed in recent years aiming to provide efficiency for embedded and edge devices with limited computation and memory resources. The wide variety of architectures makes this a complex task that has to balance generality with efficiency. Among the most interesting camera-sensor architectures are Pixel Processor Arrays (PPAs). This study presents two methods that are useful for embedded CNNs in general but particularly suitable for PPAs. The first is for training purely binarized CNNs, the second is for deploying larger models with a model swapping paradigm that loads model components dynamically. Specifically, this study trains and implements networks with batch normalization and adaptive threshold for binary activations. Then, we convert batch normalization and binary activations into a bias matrix which can be parallelly implemented by an add/sub operation. For dynamic model swapping, we propose to decompose applications that are beyond the capacity of a PPA into sub-tasks that can be solved by tree networks that can be loaded dynamically as needed. We demonstrate our approaches to various tasks including classification, localization, and coarse segmentation on a highly resource constrained PPA sensor-processor.