Hardware Resource Constraints Research Articles

Optimized k-Nearest neighbors search implementation on resource-constrained FPGA platforms

The k-Nearest Neighbors (kNN) algorithm is a fundamental machine learning classification technique with wide-ranging applications. Among various kNN implementation choices, FPGA-based heterogeneous systems have gained popularity due to FPGA's inherent parallelism, energy efficiency, and reconfigurability. However, implementing the kNN algorithm on resource-constrained embedded FPGA platforms, typically characterized by constrained programmable resources shared among various application-specific hardware units, necessitates a kNN accelerator architecture that balances high performance, hardware efficiency, and flexibility. To address this challenge, in this paper, we present a kNN hardware accelerator unit designed to optimize resource utilization by utilizing sequential, i.e. accumulation-based, instead of pipelined/parallel distance computations. The proposed architecture incorporates two key algorithmic optimizations to reduce the iteration count of the sequential distance computation loop: a dynamic lower bound enabling early termination of the distance computation and an online element selection that maximizes partial distance growth per iteration. We further enhance the accelerator's performance by incorporating multiple optimized sequential distance computation units, each dedicated to processing a segment of the training dataset. Our experiments demonstrate that the proposed approach is scalable, making it applicable to various hardware platforms and resource constraints. In particular, when implemented on an AMD Zynq device, the proposed single-core kNN accelerator occupies a mere 5 % of the FPGA's resources while delivering a speedup of 3 – 5 times compared to the kNN software implementation running on the accompanying ARM A9 processor. For the 8-core kNN accelerator, the resource utilization stands at 30 %, while the speedup factor ranges between 25 and 35.

A Low-Latency Noise-Aware Tone Mapping Operator for Hardware Implementation with a Locally Weighted Guided Filter

A tone mapping operator (TMO) is a module in the image signal processing pipeline that is used to convert high dynamic range images to low dynamic range images for display. Currently, state-of-the-art TMOs typically take complex algorithms and are implemented on graphics processing units, making it difficult to run with low latency on edge devices, and TMOs implemented in hardware circuits often lack additional noise suppression because of latency and hardware resource constraints. To address these issues, we proposed a low-latency noise-aware TMO for hardware implementation. Firstly, a locally weighted guided filter is proposed to decompose the luminance image into a base layer and a detail layer, with the weight function symmetric concerning the central pixel value of a window. Secondly, the mean and standard deviation of the basic layer and the detail layer are used to estimate the noise visibility according to the human visual characteristics. Finally, the gain for the detail layer is calculated to achieve adaptive noise suppression. In this process, luminance is first processed by the log2 function before being filtered and then symmetrically converted back to the linear domain by the exp2 function after compression. Meanwhile, the algorithms within the proposed TMO were optimized for hardware implementation to minimize latency and cache, achieving a low latency of 60.32 μs under video specification of 1080 P at 60 frames per second and objective metric smoothness in dark flat regions could be improved by more than 10% compared to similar methods.

In-Vehicle Speech Recognition for Voice-Driven UAV Control in a Collaborative Environment of MAV and UAV

Most conventional speech recognition systems have mainly concentrated on voice-driven control of personal user devices such as smartphones. Therefore, a speech recognition system used in a special environment needs to be developed in consideration of the environment. In this study, a speech recognition framework for voice-driven control of unmanned aerial vehicles (UAVs) is proposed in a collaborative environment between manned aerial vehicles (MAVs) and UAVs, where multiple MAVs and UAVs fly together, and pilots on board MAVs control multiple UAVs with their voices. Standard speech recognition systems consist of several modules, including front-end, recognition, and post-processing. Among them, this study focuses on recognition and post-processing modules in terms of in-vehicle speech recognition. In order to stably control UAVs via voice, it is necessary to handle the environmental conditions of the UAVs carefully. First, we define control commands that the MAV pilot delivers to UAVs and construct training data. Next, for the recognition module, we investigate an acoustic model suitable for the characteristics of the UAV control commands and the UAV system with hardware resource constraints. Finally, two approaches are proposed for post-processing: grammar network-based syntax analysis and transaction-based semantic analysis. For evaluation, we developed a speech recognition system in a collaborative simulation environment between a MAV and an UAV and successfully verified the validity of each module. As a result of recognition experiments of connected words consisting of two to five words, the recognition rates of hidden Markov model (HMM) and deep neural network (DNN)-based acoustic models were 98.2% and 98.4%, respectively. However, in terms of computational amount, the HMM model was about 100 times more efficient than DNN. In addition, the relative improvement in error rate with the proposed post-processing was about 65%.

An Enhanced Lightweight Network for Road Damage Detection Based on Deep Learning

Achieving accurate and efficient detection of road damage in complex scenes has always been a challenging task. In this paper, an enhanced lightweight network, E-EfficientDet, is proposed. Firstly, a feature extraction enhancement module (FEEM) is designed to increase the receptive field and improve the feature expression capability of the network, which can extract richer multi-scale feature information. Secondly, to promote the reuse of feature information between different layers in the network and take full advantage of multi-scale context information, four pyramid modules with different structures are designed based on the idea of semi-dense connection, among which the bidirectional feature pyramid network with longitudinal connection (LC-BiFPN) is more suitable for road damage detection. Finally, to meet the road damage detection tasks under different hardware resource constraints, the E-EfficientDet-D0~D2 networks are proposed in this paper based on the compound scaling strategy. Experimental results show that the detection accuracy of E-EfficientDet-D0 improves by 2.41% compared with the original EfficientDet-D0 on the publicly available road damage dataset and outperforms other networks such as YOLOv5s, YOLOv7-tiny, YOLOv4-tiny, Faster R-CNN, and SSD. Meanwhile, the detection speed of EfficientDet-D0 can reach 27.0 FPS, which meets the demand for real-time detection, and the model size is only 32.31 MB, which is suitable for deployment in mobile devices such as unmanned inspection carts, UAVs, and smartphones. In addition, the detection accuracy of E-EfficientDet-D2 can reach 57.51%, which is 4.39% higher than E-EfficientDet-D0, and the model size is 61.78 MB, which is suitable for practical application scenarios that require higher detection accuracy and better hardware performance.

Efficient Dynamic Reconfigurable CNN Accelerator for Edge Intelligence Computing on FPGA

This paper proposes an efficient dynamic reconfigurable CNN accelerator (EDRCA) for FPGAs to tackle the issues of limited hardware resources and low energy efficiency in the deployment of convolutional neural networks on embedded edge computing devices. First, a configuration layer sequence optimization method is proposed to minimize the configuration time overhead and improve performance. Second, accelerator templates for dynamic regions are designed to create a unified high-speed interface and enhance operational performance. The dynamic reconfigurable technology is applied on the Xilinx KV260 FPGA platform to design the EDRCA accelerator, resolving the hardware resource constraints in traditional accelerator design. The YOLOV2-TINY object detection network is used to test the EDRCA accelerator on the Xilinx KV260 platform using floating point data. Results at 250 MHz show a computing performance of 75.1929 GOPS, peak power consumption of 5.25 W, and power efficiency of 13.6219 GOPS/W, indicating the potential of the EDRCA accelerator for edge intelligence computing.

Memory-aware Partitioning, Scheduling, and Floorplanning for Partially Dynamically Reconfigurable Systems

Partially dynamic reconfiguration (PDR) technology can accelerate the reconfiguration process and overcome hardware resource constraints when facing the challenge of high performance with respect to applications and resources constraints on field-programmable gate arrays (FPGAs). On FPGAs with PDR technology, the available on-chip Block RAM (BRAM) resources may not satisfy the memory requirements for all data. If we reserve more BRAM resources, then the total area of the dynamically reconfigurable region (DRR) that is used for calculation will decrease, with a reduction in system performance. We propose a memory-aware optimization framework to search for the optimal solution considering partitioning, scheduling, and floorplanning, where we make a tradeoff between performance and on-chip memory resources utilization. We then propose methods for memory allocation: An ILP model and a heuristic algorithm are provided to determine the minimum memory requirements and the number of corresponding memory blocks for data, as well as to determine whether the memory block with its stored data is assigned on-chip or off-chip by formulating the problem into a 0-1 knapsack problem and solving it using dynamic programming. Experimental results show that the memory-aware optimization framework and methods of memory allocation can increase the amount of on-chip data access to 29.65% of the total data volume with guaranteed performance.

Lightweight Dual Mutual-Feedback Network for Artificial Intelligence in Medical Image Super-Resolution

As a result of hardware resource constraints, it is difficult to obtain medical images with a sufficient resolution to diagnose small lesions. Recently, super-resolution (SR) was introduced into the field of medicine to enhance and restore medical image details so as to help doctors make more accurate diagnoses of lesions. High-frequency information enhances the accuracy of the image reconstruction, which is demonstrated by deep SR networks. However, deep networks are not applicable to resource-constrained medical devices because they have too many parameters, which requires a lot of memory and higher processor computing power. For this reason, a lightweight SR network that demonstrates good performance is needed to improve the resolution of medical images. A feedback mechanism enables the previous layers to perceive high-frequency information of the latter layers, but no new parameters are introduced, which is rarely used in lightweight networks. Therefore, in this work, a lightweight dual mutual-feedback network (DMFN) is proposed for medical image super-resolution, which contains two back-projection units that operate in a dual mutual-feedback manner. The features generated by the up-projection unit are fed back into the down-projection unit and, simultaneously, the features generated by the down-projection unit are fed back into the up-projection unit. Moreover, a contrast-enhanced residual block (CRB) is proposed as each cell block used in projection units, which enhances the pixel contrast in the channel and spatial dimensions. Finally, we designed a unity feedback to down-sample the SR result as the inverse process of SR. Furthermore, we compared it with the input LR to narrow the solution space of the SR function. The final ablation studies and comparison results show that our DMFN performs well without utilizing a large amount of computing resources. Thus, it can be used in resource-constrained medical devices to obtain medical images with better resolutions.

Optimal Mapping of Spiking Neural Network to Neuromorphic Hardware for Edge-AI.

Neuromorphic hardware, the new generation of non-von Neumann computing system, implements spiking neurons and synapses to spiking neural network (SNN)-based applications. The energy-efficient property makes the neuromorphic hardware suitable for power-constrained environments where sensors and edge nodes of the internet of things (IoT) work. The mapping of SNNs onto neuromorphic hardware is challenging because a non-optimized mapping may result in a high network-on-chip (NoC) latency and energy consumption. In this paper, we propose NeuMap, a simple and fast toolchain, to map SNNs onto the multicore neuromorphic hardware. NeuMap first obtains the communication patterns of an SNN by calculation that simplifies the mapping process. Then, NeuMap exploits localized connections, divides the adjacent layers into a sub-network, and partitions each sub-network into multiple clusters while meeting the hardware resource constraints. Finally, we employ a meta-heuristics algorithm to search for the best cluster-to-core mapping scheme in the reduced searching space. We conduct experiments using six realistic SNN-based applications to evaluate NeuMap and two prior works (SpiNeMap and SNEAP). The experimental results show that, compared to SpiNeMap and SNEAP, NeuMap reduces the average energy consumption by 84% and 17% and has 55% and 12% lower spike latency, respectively.

Parallelism Versus Latency in Simplified Successive-Cancellation Decoding of Polar Codes

This paper characterizes the latency of the simplified successive-cancellation (SSC) decoding scheme for polar codes under hardware resource constraints. In particular, when the number of processing elements P that can perform SSC decoding operations in parallel is limited, as is the case in practice, the latency of SSC decoding is O(N1-1/μ + N/P log2 log2 N/P), where N is the block length of the code and μ is the scaling exponent of the channel. Three direct consequences of this bound are presented. First, in a fully-parallel implementation where P = N/2, the latency of SSC decoding is O(N1-1/μ), which is sublinear in the block length. This recovers a result from our earlier work. Second, in a fully-serial implementation where P = 1, the latency of SSC decoding scales as O(N log2 log2 N). The multiplicative constant is also calculated: we show that the latency of SSC decoding when P = 1 is given by (2 + o(1))N log2 log2 N. Third, in a semi-parallel implementation, the smallest P that gives the same latency as that of the fully-parallel implementation is P = N1/μ. The tightness of our bound on SSC decoding latency and the applicability of the foregoing results is validated through extensive simulations.

Real-time task scheduling for FPGA-based multicore systems with communication delay

Efficient scheduling is critical for achieving high performance in the multicore computing environment. This paper focuses on the scheduling problem for the real-time applications in the FPGA-based multicore structure with an objective to minimize the makespan under hardware resource constraints. To address this problem, we propose a priority-driven scheduling algorithm, called the Real-Time Priority-driven Algorithm (ReTPA). In this method, we first allocate the tasks into software and hardware computing units based on their hardware resource constraints. Then scheduling is performed to minimize the makespan. We further extend ReTPA with communication scheduling to improve the scheduling efficiency in the communication-intensive applications, which is called Communication-based Real-Time Priority-driven Algorithm (CReTPA). Our experimental results confirm the efficiency of the proposed algorithms in applications with different scales or different computation-communication ratios. Besides, we compare the two proposed algorithms with a classic priority-driven algorithm, HEFT, an advanced priority-driven algorithm, HETS, and two advanced genetic algorithms, GAA and MGAA, proposed by Abdallah et al. in 2019. The performance comparisons illustrate that our algorithms, ReTPA and CReTPA, both show the outstanding performance and CReTPA obtains the most reasonable solutions within the shortest scheduling time for communication-intensive applications.

Real-time object detector for low-end devices

YOLOv4-tiny is a lightweight network designed for low-end devices. It proposes a global model scaling technology, which uses the same scaling method with few convolution filters in each stage of the network, resulting in a small receptive field and low accuracy. To solve this problem, we use different scaling techniques in shallow and deep network stages. For the shallow network stage, the Simple-StemBlock scaling module is proposed to simplify network based on factors such as FLOPs and network fragmentation. The module can effectively reduce computation and improve the diversity of features. For the deep network stage, we consider depth of network and hardware resource constraints, the Depth-CSPBlock scaling module is designed to expand receptive field while keeping the computation as low as possible and layered residual connection is built in the module to enrich semantic information. Besides, mish activation function is adopted in the backbone for higher accuracy. The experimental results show that the accuracy of the proposed method achieves 23.2% AP on MSCOCO test-dev and 66.73 % AP50 on VOC datasets, compared with YOLOv4-tiny, the AP and AP50 increased by 3.3% and 3.7%, respectively. The speed of the proposed method on Jetson TX2 can reach 45.6 frames per second, which is 20% faster than YOLOv4-tiny.

A Low Coupling and Lightweight Algorithm for Ship Detection in Optical Remote Sensing Images

In recent years, many ship detection algorithms based on convolutional neural networks (CNNs) have been proposed to improve the performance of ship detection. However, with the increase in model complexity and size, it is challenging to deploy these models to resource-constrained edge platforms. In this letter, a low coupling algorithm that belongs to anchor-free methods is proposed for ship detection to reduce the model complexity and still obtain a competitive performance. The proposed low coupling network (LCNet) is easy to deploy and contributes to speeding up the inference and improving memory utilization. In addition, we propose a model compression process consisting of the quantization-aware training (QAT) method and a structural pruning method based on Taylor expansion, which can effectively reduce the model size according to hardware resource constraints. Comparative experimental results demonstrate that LCNet outperforms the state-of-the-art ship detection and natural object detection algorithms, with a 95.27% mAP and 88.91% F1 score on the HRSC2016 dataset. Our proposed model compression method also achieves a compression ratio of at least 80% with a negligible loss of performance.

A Decision-Making Method Providing Sustainability to FPGA-Based SoCs by Run-Time Structural Adaptation to Mode of Operation, Power Budget, and Die Temperature Variations

One of the growing areas of application of embedded systems in robotics, aerospace, military, etc. is autonomous mobile systems. Usually, such embedded systems have multitask multimodal workloads. These systems must sustain the required performance of their dynamic workloads in presence of varying power budget due to rechargeable power sources, varying die temperature due to varying workloads and/or external temperature, and varying hardware resources due to occurrence of hardware faults. This paper proposes a run-time decision-making method, called Decision Space Explorer, for FPGA-based Systems-on-Chip (SoCs) to support changing workload requirements while simultaneously mitigating unpredictable variations in power budget, die temperature, and hardware resource constraints. It is based on the concept of Run-Time Structural Adaptation (RTSA); whenever there is a change in a system’s set of constraints, Explorer selects a suitable hardware processing circuit for each active task at an appropriate operating frequency such that all the constraints are satisfied. Explorer has been experimentally deployed on the ARM Cortex-A9 core of Xilinx Zynq XC7Z020 SoC. Its worst-case decision-making time for different scenarios ranges from tens to hundreds of microseconds. Explorer is thus suitable for enabling RTSA in systems where specifications of multiple objectives must be maintained simultaneously, making them self-sustainable.

A Study on Seizure Detection of EEG Signals Represented in 2D.

A seizure is a neurological disorder caused by abnormal neuronal discharges in the brain, which severely reduces the quality of life of patients and often endangers their lives. Automatic seizure detection is an important research area in the treatment of seizure and is a prerequisite for seizure intervention. Deep learning has been widely used for automatic detection of seizures, and many related research works decomposed the electroencephalogram (EEG) raw signal with a time window to obtain EEG signal slices, then performed feature extraction on the slices, and represented the obtained features as input data for neural networks. There are various methods for EEG signal decomposition, feature extraction, and representation, and most of the studies have been based on fixed hardware resources for the design of the scheme, which reduces the adaptability of the scheme in different application scenarios and makes it difficult to optimize the algorithms in the scheme. To address the above issues, this paper proposes a deep learning-based model for seizure detection, mainly characterized by the two-dimensional representation of EEG features and the scalability of neural networks. The model modularizes the main steps of seizure detection and improves the adaptability of the model to different hardware resource constraints, in order to increase the convenience of the algorithm optimization or the replacement of each module. The proposed model consists of five parts, and the model was tested using two epilepsy datasets separately. The experimental results showed that the proposed model has strong generality and good classification accuracy for seizure detection.

Edge Intelligent Networking Optimization for Internet of Things in Smart City

Massive devices connected through fifth-generation (5G) networks constitute a ubiquitous Internet of Things (IoT), providing diverse service applications in a smart city. A robust network topology structure against cyber-attacks is the foundation of highly reliable service quality, especially in next-generation networks or beyond 5G (B5G) networks. Existing methods apply neural networks with deep reinforcement learning methods to advance the network topology. However, the reduction of unique hardware resource constraints and the application of edge intelligent networking capability of terminal nodes are emerging challenges for robustness optimization of IoT with 5G and B5G networks. To address these problems, we design a distributed learning framework utilizing edge intelligence, improving smart terminal nodes' networking capability, which is deployed on ordinary computers instead of specialized hardware such as GPUs. The proposed framework leveraging multi-core CPU and intelligent edge methods decreases the training time and economic cost and takes full advantage of computer resources. Furthermore, the best performing framework considers the distributed communication model of edge computing and optimizes the network topology by taking advantage of smart terminal nodes' contributions. We show that the framework succeeds in various topologies and outperforms compared with other state-of-art algorithms in improving the robustness for IoT topology in smart cities.

Evolutionary multi-level acyclic graph partitioning

Directed graphs are widely used to model data flow and execution dependencies in streaming applications. This enables the utilization of graph partitioning algorithms for the problem of parallelizing execution on multiprocessor architectures under hardware resource constraints. However due to program memory restrictions in embedded multiprocessor systems, applications need to be divided into parts without cyclic dependencies. We found that this can be done by a subsequent second graph partitioning step with an additional acyclicity constraint. We have four main contributions. First, we show that this more constrained version of the graph partitioning problem is NP-complete and present linear time heuristics. We then integrate them into an existing multi-level graph partitioning framework to better handle large graphs. This achieves a 9% reduction of the edge cut compared to the previous single-level algorithm. Based on this, we engineer an evolutionary algorithm to further reduce the cut, achieving a 30% reduction on average compared to the state of the art. Finally, we integrate the partitioning heuristics into a graph compiler for an embedded multiprocessor architecture and show that this can reduce the amount of communication for a real-world imaging application and thereby accelerate it by an average of 11%. It is shown that the compiler can emit optimized code for vastly different hardware platforms using the heuristics. In addition, we demonstrate how a custom fitness function for the evolutionary algorithm can be used to optimize other objectives like load balancing if the communication volume is not predominantly important on a given hardware platform.

A depthwise separable convolutional neural network for keyword spotting on an embedded system

A keyword spotting algorithm implemented on an embedded system using a depthwise separable convolutional neural network classifier is reported. The proposed system was derived from a high-complexity system with the goal to reduce complexity and to increase efficiency. In order to meet the requirements set by hardware resource constraints, a limited hyper-parameter grid search was performed, which showed that network complexity could be drastically reduced with little effect on classification accuracy. It was furthermore found that quantization of pre-trained networks using mixed and dynamic fixed point principles could reduce the memory footprint and computational requirements without lowering classification accuracy. Data augmentation techniques were used to increase network robustness in unseen acoustic conditions by mixing training data with realistic noise recordings. Finally, the system’s ability to detect keywords in a continuous audio stream was successfully demonstrated.

ROS: Resource-constrained Oracle Synthesis for Quantum Computers

We present a completely automatic synthesis framework for oracle functions, a central part in many quantum algorithms. The proposed framework for resource-constrained oracle synthesis (ROS) is a LUT-based hierarchical method in which every step is specifically tailored to address hardware resource constraints. ROS embeds a LUT mapper designed to simplify the successive synthesis steps, costing each LUT according to the resources used by its corresponding quantum circuit. In addition, the framework exploits a SAT-based quantum garbage management technique. Those two characteristics give ROS the ability to beat the state-of-the-art hierarchical method both in number of qubits and in number of operations. The efficiency of the framework is demonstrated by synthesizing quantum oracles for Grover's algorithm.

Task number maximization offloading strategy seamlessly adapted to UAV scenario

Mobile edge computing (MEC) has been proposed in recent years to process resource-intensive and delay-sensitive applications at the edge of mobile networks, which can break the hardware limitations and resource constraints at user equipment (UE). In order to fully use the MEC server resource, how to maximize the number of offloaded tasks is meaningful especially for crowded place or disaster area. In this paper, an optimal partial offloading scheme POSMU (Partial Offloading Strategy Maximizing the User task number) is proposed to obtain the optimal offloading ratio, local computing frequency, transmission power and MEC server computing frequency for each UE. The problem is formulated as a mixed integer nonlinear programming problem (MINLP), which is NP-hard and challenging to solve. As such, we convert the problem into multiple nonlinear programming problems (NLPs) and propose an efficient algorithm to solve them by applying the block coordinate descent (BCD) as well as convex optimization techniques. Besides, we can seamlessly apply POSMU to UAV (Unmanned Aerial Vehicle) enabled MEC system by analyzing the 3D communication model. The optimality of POSMU is illustrated in numerical results, and POSMU can approximately maximize the number of offloaded tasks compared to other schemes.

Development and Validation of an EEG-Based Real-Time Emotion Recognition System Using Edge AI Computing Platform With Convolutional Neural Network System-on-Chip Design

This study proposed an electroencephalogram (EEG)-based real-time emotion recognition hardware system architecture based on multiphase convolutional neural network (CNN) algorithm implemented on a 28-nm technology chip and on field programmable gate array (FPGA) for binary and quaternary classification. Sample entropy, differential asymmetry, short-time Fourier transform, and a channel reconstruction method were used for emotion feature extraction. In this work, six EEG channels were selected (FP1, FP2, F3, F4, F7, and F8), and EEG images were generated from spectrogram fusions. The complete CNN architecture included training and acceleration for efficient artificial intelligence (AI) edge application, and we proposed a multiphase CNN execution method to accommodate hardware resource constraints. Datasets of 32 subjects from the DEAP database were used to validate the proposed design, exhibiting mean accuracies for valance binary classification and valance-arousal quaternary classification of 83.36% and 76.67%, respectively. The core area and total power consumption of the CNN chip were $1.83 \,\, \times \,\, 1.83~mm^{2}$ , respectively, and $76.61~mW$ . The chip operation was validated using ADVANTEST V93000 PS1600, and the training process and real-time classification processing time took 0.12495 ms and 0.02634 ms for each EEG image, respectively. The proposed EEG-based real-time emotion recognition system included a dry electrode EEG headset, feature extraction processor, CNN chip platform, and graphical user interface, and the execution time costed $450~ms$ for each emotional state recognition.