Hand-crafted Design Research Articles

FusionFormer: A Concise Unified Feature Fusion Transformer for 3D Pose Estimation

Depth uncertainty is a core challenge in 3D human pose estimation, especially when the camera parameters are unknown. Previous methods try to reduce the impact of depth uncertainty by multi-view and/or multi-frame feature fusion to utilize more spatial and temporal information. However, they generally lead to marginal improvements and their performance still cannot match the camera-parameter-required methods. The reason is that their handcrafted fusion schemes cannot fuse the features flexibly, e.g., the multi-view and/or multi-frame features are fused separately. Moreover, the diverse and complicated fusion schemes make the principle for developing effective fusion schemes unclear and also raises an open problem that whether there exist more simple and elegant fusion schemes. To address these issues, this paper proposes an extremely concise unified feature fusion transformer (FusionFormer) with minimized handcrafted design for 3D pose estimation. FusionFormer fuses both the multi-view and multi-frame features in a unified fusion scheme, in which all the features are accessible to each other and thus can be fused flexibly. Experimental results on several mainstream datasets demonstrate that FusionFormer achieves state-of-the-art performance. To our best knowledge, this is the first camera-parameter-free method to outperform the existing camera-parameter-required methods, revealing the tremendous potential of camera-parameter-free models. These impressive experimental results together with our concise feature fusion scheme resolve the above open problem. Another appealing feature of FusionFormer we observe is that benefiting from its effective fusion scheme, we can achieve impressive performance with smaller model size and less FLOPs.

Hyper-Heuristic Approach for Tuning Parameter Adaptation in Differential Evolution

Differential evolution (DE) is one of the most promising black-box numerical optimization methods. However, DE algorithms suffer from the problem of control parameter settings. Various adaptation methods have been proposed, with success history-based adaptation being the most popular. However, hand-crafted designs are known to suffer from human perception bias. In this study, our aim is to design automatically a parameter adaptation method for DE with the use of the hyper-heuristic approach. In particular, we consider the adaptation of scaling factor F, which is the most sensitive parameter of DE algorithms. In order to propose a flexible approach, a Taylor series expansion is used to represent the dependence between the success rate of the algorithm during its run and the scaling factor value. Moreover, two Taylor series are used for the mean of the random distribution for sampling F and its standard deviation. Unlike most studies, the Student’s t distribution is applied, and the number of degrees of freedom is also tuned. As a tuning method, another DE algorithm is used. The experiments performed on a recently proposed L-NTADE algorithm and two benchmark sets, CEC 2017 and CEC 2022, show that there is a relatively simple adaptation technique with the scaling factor changing between 0.4 and 0.6, which enables us to achieve high performance in most scenarios. It is shown that the automatically designed heuristic can be efficiently approximated by two simple equations, without a loss of efficiency.

Design of Wide‐Beam Leaky‐Wave Antenna Arrays Based on the Bilinear Transformation of IIR Digital Filters and the Z Transform

Abstract In the pioneering work, the radiation diagrams of leaky wave antenna arrays can achieve attenuation nulls and gains at specific angles by manually placing the zeros and the poles in the Z domain of the corresponding discrete linear time‐invariant (LTI) system. This handcrafted design procedure does not allow radiation diagrams with wide beams since the interaction between poles involved in the wide beam and their corresponding leaky modes cannot be easily handled. To overcome this limitation, this paper describes a novel method for designing radiation diagrams of leaky‐wave antenna arrays based on the theory of IIR discrete filters. The proposed method relies on the design of discrete filters with the prototypes of analog low‐pass filters defined by Butterworth and Chebyshev type I polynomials, whose roots along with the bilinear transformation provide the location of the poles and the zeros of the discrete LTI system and, therefore, the parameters of the leaky‐wave antenna array. Results with different designs and a comparison with other approaches show the utility and effectiveness of this novel method to design wide‐beam leaky‐wave antenna arrays.

PUPS: Point Cloud Unified Panoptic Segmentation

Point cloud panoptic segmentation is a challenging task that seeks a holistic solution for both semantic and instance segmentation to predict groupings of coherent points. Previous approaches treat semantic and instance segmentation as surrogate tasks, and they either use clustering methods or bounding boxes to gather instance groupings with costly computation and hand-craft designs in the instance segmentation task. In this paper, we propose a simple but effective point cloud unified panoptic segmentation (PUPS) framework, which use a set of point-level classifiers to directly predict semantic and instance groupings in an end-to-end manner. To realize PUPS, we introduce bipartite matching to our training pipeline so that our classifiers are able to exclusively predict groupings of instances, getting rid of hand-crafted designs, e.g. anchors and Non-Maximum Suppression (NMS). In order to achieve better grouping results, we utilize a transformer decoder to iteratively refine the point classifiers and develop a context-aware CutMix augmentation to overcome the class imbalance problem. As a result, PUPS achieves 1st place on the leader board of SemanticKITTI panoptic segmentation task and state-of-the-art results on nuScenes.

Optical neural network via loose neuron array and functional learning

This research proposes a deep-learning paradigm, termed functional learning (FL), to physically train a loose neuron array, a group of non-handcrafted, non-differentiable, and loosely connected physical neurons whose connections and gradients are beyond explicit expression. The paradigm targets training non-differentiable hardware, and therefore solves many interdisciplinary challenges at once: the precise modeling and control of high-dimensional systems, the on-site calibration of multimodal hardware imperfectness, and the end-to-end training of non-differentiable and modeless physical neurons through implicit gradient propagation. It offers a methodology to build hardware without handcrafted design, strict fabrication, and precise assembling, thus forging paths for hardware design, chip manufacturing, physical neuron training, and system control. In addition, the functional learning paradigm is numerically and physically verified with an original light field neural network (LFNN). It realizes a programmable incoherent optical neural network, a well-known challenge that delivers light-speed, high-bandwidth, and power-efficient neural network inference via processing parallel visible light signals in the free space. As a promising supplement to existing power- and bandwidth-constrained digital neural networks, light field neural network has various potential applications: brain-inspired optical computation, high-bandwidth power-efficient neural network inference, and light-speed programmable lens/displays/detectors that operate in visible light.

Deep Learning Techniques for Visual SLAM: A Survey

Visual Simultaneous Localization and Mapping (VSLAM) has attracted considerable attention in recent years. This task involves using visual sensors to localize a robot while simultaneously constructing an internal representation of its environment. Traditional VSLAM methods involve the laborious hand-crafted design of visual features and complex geometric models. As a result, they are generally limited to simple environments with easily identifiable textures. Recent years, however, have witnessed the development of deep learning techniques for VSLAM. This is primarily due to their capability of modeling complex features of the environment in a completely data-driven manner. In this paper, we present a survey of relevant deep learning-based VSLAM methods and suggest a new taxonomy for the subject. We also discuss some of the current challenges and possible directions for this field of study.

A Planning-and-Control Framework for Aerial Manipulation of Articulated Objects

While the variety of applications for Aerial Manipulators (AMs) has increased over the last years, they are mostly limited to push-and-slide tasks. More complex manipulations of dynamic environments are poorly addressed and still require handcrafted designs of hardware, control, and trajectory planning. In this letter we focus on the active manipulation of articulated objects with AMs. We present a novel planning and control approach that allows the AM to execute complex interaction maneuvers with as little as possible priors given by the operator. Our framework combines sampling-based predictive control to generate pose trajectories with an impedance controller for compliant behaviours, applied to a fully-actuated flying platform. The framework leverages a physics engine to simulate the dynamics of the platform and the environment in order to find optimal motions to execute manipulation tasks. Experiments on two selected examples of pulling open a door and of turning a valve show the feasibility of the proposed approach.

Activity Graph Based Convolutional Neural Network for Human Activity Recognition Using Acceleration and Gyroscope Data

Human activity recognition (HAR) using smartphone sensors have been recently studied in various applications including healthcare, fitness, and smart home. Their recognition accuracy often depends on high-quality feature design and effectiveness of classification algorithms, where existing work mostly replies on laborious hand-crafted design and shallow feature learning architecture. Recent deep learning techniques demonstrate outstanding effectiveness in performing automatic feature learning and outperform traditional models in terms of accuracy. But their performance is limited by the quality and volumes of available labelled data. It is challenging to achieve accurate multisubject HAR with only smartphone sensing data. This article proposes a novel optimal activity graph generation model incorporating a deep learning framework for automatic and accurate HAR with multiple subjects using only acceleration and gyroscope data. The activity graph generation model presents a multisensory integration mechanism with three-steps sorting algorithms for producing optimal activity graphs containing alignments of neighbored signals in their width and height. Then, we propose a deep convolutional neural network to automatically learn distinguishable features from the graphs for HAR. By leveraging superior presentation of correlations between human activities and neighbored signals alignments via optimal activity graphs, the learned features are endowed with more discriminative power. The experimental evaluation was carried out on several benchmark datasets (i.e., UCI, USCHAD, and UTD-MHAD). The results showed that our approach improved the average recognition accuracy by about 5% when compared with other state-of-the-art HAR methods. Particularly towards multisubject HAR cases (UTD-MHAD dataset with 21 subjects), it achieved up to 10% accuracy gain over other methods. These improvements show the advantage and potential of our method dealing with complex HAR problems with multiple subjects using limited sensing data.

MD3D: Mixture-Density-Based 3D Object Detection in Point Clouds

The design factors of anchor boxes, such as shape, placement, and target assignment policy, greatly influence the performance and latency of the 3D object detectors. Unlike image-based 2D anchors, 3D anchors must be placed in a 3D space and determined differently for each class of different sizes. This imposes a significant burden on the design complexity. To tackle this issue, various studies have been conducted on how to set the anchor form. However, for practical reasons, anchor-based methods select the anchor design by compromising between performance and latency. Consequently, only objects that are similar in shape and size to an anchor can obtain high accuracy. In this paper, we propose a Mixture-Density-based 3D Object Detection (MD3D) in point clouds to predict the distribution of 3D bounding boxes using a Gaussian Mixture Model (GMM). With an anchor-free detection head, MD3D requires few hand-crafted design factors and eliminates the inefficiency of separating the regression channel for each class, and thus offering both latency and memory benefits. MD3D is designed to utilize various types of feature encoding; therfore, it can be applied flexibly by replacing only the detection head of the existing detectors. Experimental results on the KITTI and Waymo open datasets show that the proposed method outperforms its counterparts that are based on the conventional anchor-based detection head in its overall performance, latency, and memory. The code is publicly available at https://github.com/sky77764/MD3D.

End-to-End Deep Reinforcement Learning for Image-Based UAV Autonomous Control

To achieve the perception-based autonomous control of UAVs, schemes with onboard sensing and computing are popular in state-of-the-art work, which often consist of several separated modules with respective complicated algorithms. Most methods depend on handcrafted designs and prior models with little capacity for adaptation and generalization. Inspired by the research on deep reinforcement learning, this paper proposes a new end-to-end autonomous control method to simplify the separate modules in the traditional control pipeline into a single neural network. An image-based reinforcement learning framework is established, depending on the design of the network architecture and the reward function. Training is performed with model-free algorithms developed according to the specific mission, and the control policy network can map the input image directly to the continuous actuator control command. A simulation environment for the scenario of UAV landing was built. In addition, the results under different typical cases, including both the small and large initial lateral or heading angle offsets, show that the proposed end-to-end method is feasible for perception-based autonomous control.

GEOMETRIC SCATTERING ATTENTION NETWORKS.

Geometric scattering has recently gained recognition in graph representation learning, and recent work has shown that integrating scattering features in graph convolution networks (GCNs) can alleviate the typical oversmoothing of features in node representation learning. However, scattering often relies on handcrafted design, requiring careful selection of frequency bands via a cascade of wavelet transforms, as well as an effective weight sharing scheme to combine low- and band-pass information. Here, we introduce a new attention-based architecture to produce adaptive task-driven node representations by implicitly learning node-wise weights for combining multiple scattering and GCN channels in the network. We show the resulting geometric scattering attention network (GSAN) outperforms previous networks in semi-supervised node classification, while also enabling a spectral study of extracted information by examining node-wise attention weights.

Real-Time Semantic Segmentation via Auto Depth, Downsampling Joint Decision and Feature Aggregation

To satisfy the stringent requirements for computational resources in the field of real-time semantic segmentation, most approaches focus on the hand-crafted design of light-weight segmentation networks. To enjoy the ability of model auto-design, Neural Architecture Search (NAS) has been introduced to search for the optimal building blocks of networks automatically. However, the network depth, downsampling strategy, and feature aggregation method are still set in advance and nonadjustable during searching. Moreover, these key properties are highly correlated and essential for a remarkable real-time segmentation model. In this paper, we propose a joint search framework, called AutoRTNet, to automate all the aforementioned key properties in semantic segmentation. Specifically, we propose hyper-cells to jointly decide the network depth and the downsampling strategy via a novel cell-level pruning process. Furthermore, we propose an aggregation cell to achieve automatic multi-scale feature aggregation. Extensive experimental results on Cityscapes and CamVid datasets demonstrate that the proposed AutoRTNet achieves the new state-of-the-art trade-off between accuracy and speed. Notably, our AutoRTNet achieves 73.9% mIoU on Cityscapes and 110.0 FPS on an NVIDIA TitanXP GPU card with input images at a resolution of $$768 \times 1536$$ .

Feature detection and description for image matching: from hand-crafted design to deep learning

ABSTRACT In feature based image matching, distinctive features in images are detected and represented by feature descriptors. Matching is then carried out by assessing the similarity of the descriptors of potentially conjugate points. In this paper, we first shortly discuss the general framework. Then, we review feature detection as well as the determination of affine shape and orientation of local features, before analyzing feature description in more detail. In the feature description review, the general framework of local feature description is presented first. Then, the review discusses the evolution from hand-crafted feature descriptors, e.g. SIFT (Scale Invariant Feature Transform), to machine learning and deep learning based descriptors. The machine learning models, the training loss and the respective training data of learning-based algorithms are looked at in more detail; subsequently the various advantages and challenges of the different approaches are discussed. Finally, we present and assess some current research directions before concluding the paper.

Deep Unfolding With Weighted ℓ₂ Minimization for Compressive Sensing

Compressive sensing (CS) aims to accurately reconstruct high-dimensional signals from a small number of measurements by exploiting signal sparsity and structural priors. However, signal priors utilized in existing CS reconstruction algorithms rely mainly on hand-crafted design, which often cannot offer the best sparsity-undersampling tradeoff because high-order structural priors of signals are hard to be captured in this manner. In this article, a new recovery guarantee of the unified CS reconstruction model-weighted ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> minimization (WL1M) is derived, which indicates universal priors could hardly lead to the optimal selection of the weights. Motivated by the analysis, we propose a deep unfolding network for the general WL1M model. The proposed deep unfolding-based WL1M (D-WL1M) integrates universal priors with learning capability so that all of the parameters, including the crucial weights, can be learned from training data. We demonstrate the proposed D-WL1M outperforms several state-of-the-art CS-based methods and deep learning-based methods by a large margin via the experiments on the Caltech-256 image data set.

Self-Supervised Learning for Autonomous Vehicles Perception: A Conciliation Between Analytical and Learning Methods

The interest in autonomous driving has continuously increased in the last two decades. However, to be adopted, such critical systems need to be safe. Concerning the perception of the ego-vehicle environment, the literature has investigated two different types of methods. On the one hand, traditional analytical methods generally rely on handcrafted designs and features while on the other hand, learning methods aim at designing their own appropriate representation of the observed scene.

Sparse Signal Reconstruction With Statistical Prior Information: A Data-Driven Method

Weighted $\ell _{1}$ minimization (WL1M) is a general and powerful framework for reconstructing sparse signals from underdetermined measurements. The performance improvement of WL1M owes to the incorporation of additional structural priors of signals by means of its weights. However, the selection of weights relies on hand-crafted designs in existing works, so that high-order structural priors of signals are hard to be captured. This paper proposes a data-driven method, namely RBM-WL1M, to alleviate this situation. In the RBM-WL1M, restricted Boltzmann machines (RBMs) are employed to learn the prior distribution of the signals from training data; furthermore, utilizing the RBM, high frequency support set and non-zero probabilities for each of the entries in signals can be estimated effectively, which are used to appropriately select the weights. In our experiments, the proposed framework demonstrates superior performance over several state-of-the-art CS methods on the Physikalisch-Technische Bundesanstalt(PTB) Diagnostic ECG Data set.

Design of a (480, 240) CMOS analog low-density parity-check decoder

Digital low-density parity-check (LDPC) decoders can hardly meet the power-limits brought by the new application scenarios. The analog LDPC decoder, which is an application of the analog computation technology, is considered to have the potential to address this issue to some extent. However, due to the lack of automation tools and analog stopping criteria, the analog LDPC decoders suffer from costly handcraft design and additional decoding delay, and are not feasible to practical applications. To address these issues, a decoder architecture using reusable building blocks is designed to lower the handcraft design, and a probability stopping criterion that is specially designed for analog decoder is further planned and implemented to reduce the decoding delay. Then, a (480,240) CMOS analog LDPC decoder is designed and fabricated in a 0.35-μm CMOS technology. Experimental results show that the decoder prototype can achieve 50Mbps throughput when the power consumption is about 86.3mW, and the decoding delay can be reduced by at most 93% compared with using the preset maximum decoding delay in existing works.

Aggressive Pipelining of Irregular Applications on Reconfigurable Hardware

CPU-FPGA heterogeneous platforms offer a promising solution for high-performance and energy-efficient computing systems by providing specialized accelerators with post-silicon reconfigurability. To unleash the power of FPGA, however, the programmability gap has to be filled so that applications specified in high-level programming languages can be efficiently mapped and scheduled on FPGA. The above problem is even more challenging for irregular applications, in which the execution dependency can only be determined at run time. Thus over-serialized accelerators are generated from existing works that rely on compile time analysis to schedule the computation. In this work, we propose a comprehensive software-hardware co-design framework, which captures parallelism in irregular applications and aggressively schedules pipelined execution on reconfigurable platform. Based on an inherently parallel abstraction packaging parallelism for runtime schedule, our framework significantly differs from existing works that tend to schedule executions at compile time. An irregular application is formulated as a set of tasks with their dependencies specified as rules describing the conditions under which a subset of tasks can be executed concurrently. Then datapaths on FPGA will be generated by transforming applications in the formulation into task pipelines orchestrated by evaluating rules at runtime, which could exploit fine-grained pipeline parallelism as handcrafted accelerators do. An evaluation shows that this framework is able to produce datapath with its quality close to handcrafted designs. Experiments show that generated accelerators are dramatically more efficient than those created by current high-level synthesis tools. Meanwhile, accelerators generated for a set of irregular applications attain 0.5x~1.9x performance compared to equivalent software implementations we selected on a server-grade 10-core processor, with the memory subsystem remaining as the bottleneck.

Design, Simulation, and Implementation of a CMOS Analog Decoder for (480,240) Low-Density Parity-Check Code

The analog low-density parity-check (LDPC) decoder, which is a specific application of the probabilistic computing, is considered to be a promising solution for power-constrained applications. However, due to the lack of efficient electronic design automation tools and reliable circuit model, the analog LDPC decoders suffer from costly hand-craft design cycle, and are unable to provide enough coding gains for practical applications. In this paper, we present an implementation of a (480,240) CMOS analog LDPC decoder, which is the longest implemented code to date using the analog approach. We first propose an analog LDPC decoder architecture, which is constructed by the reusable modules and can significantly reduce the hardware complexity. And then, we present a mixed behavioral and structural model for the analog LDPC decoding circuits, which can reliably and efficiently predict the error-correcting performance. Finally, the experimental results show that the decoder prototype, which is fabricated in a 0.35- $\mu \text{m}$ CMOS technology, can achieve a throughput higher than 50 Mbps with the power consumption of 86.3 mW for the decoder core, and can offer a superior 6.3-dB coding gain at the bit error rate of $10^{-6}$ when the tested throughput is 5 Mbps. The proposed analog LDPC decoder is suitable for the power-limited applications with moderate throughput and certain coding gains.

Procedural generation of non-player characters in massively multiplayer online strategy games

This paper presents a coevolutionary framework for procedural generation of NPCs in MMORTS games. In this context, players need to defeat environmental troops in battle to gather resources and achieve their goals. The benchmarked game has several balance problems related to these battle activities, mostly caused by the handcraft design of complex game content. To solve this problem, the algorithm takes player modeled heroes as input and returns a solution evolved to win. By this way, the players need to think better in a new way to conquer the victory, adding new levels of challenge, keeping the game enjoyable. We present a new mathematical model to evaluate the solutions, based only on the number of soldiers on the input and output, making it easy to extend to other contexts. The results show it is possible to procedurally generate thousands of new efficient and fair builds, without violating the game rules. Moreover, our analysis of the results was able to identify unbalanced characteristics in the game design and we suggested simple way to fix it.