Two-stage Architecture Research Articles

The two-stage detection-after-segmentation model improves the accuracy of identifying subdiaphragmatic lesions

Chest X-rays (CXRs) are primarily used to detect lung lesions. While the abdominal portion of CXRs can sometimes reveal critical conditions, research in this area is limited. To address this, we introduce a two-stage architecture that separates the abdominal region from the CXR and detects abdominal lesions using a specialized dataset. We compared the performance of our method on whole CXRs versus isolated abdominal regions. First, we created masks for the right upper quadrant (RUQ), left upper quadrant (LUQ), and upper abdomen (ABD) regions and trained corresponding segmentation models for each area. For detecting abdominal lesions, we curated a dataset of 5,996 images, categorized into 19 classes based on anatomical locations, air patterns, and levels of stomach or bowel dilation. The detection process was initially conducted on the original images, followed by the three regional areas, RUQ, LUQ, and ABD, extracted by the segmentation models. The results showed that the detection model trained on the entire ABD region achieved the highest accuracy, followed closely by the RUQ and LUQ models. In contrast, the CXR model had the lowest accuracy. This study highlights that the two-stage architecture effectively manages distinct segmentation and detection tasks in CXRs, offering a promising avenue for more accurate diagnoses. It also suggests that an optimal ratio between the sizes of the target lesions and the input images may exist.

Hybrid Transformer and Convolution for Image Compressed Sensing

In recent years, deep unfolding networks (DUNs) have received widespread attention in the field of compressed sensing (CS) reconstruction due to their good interpretability and strong mapping capabilities. However, existing DUNs often improve the reconstruction effect at the expense of a large number of parameters, and there is the problem of information loss in long-distance feature transmission. Based on the above problems, we propose an unfolded network architecture that mixes Transformer and large kernel convolution to achieve sparse sampling and reconstruction of natural images, namely, a reconstruction network based on Transformer and convolution (TCR-Net). The Transformer framework has the inherent ability to capture global context through a self-attention mechanism, which can effectively solve the challenge of long-range dependence on features. TCR-Net is an end-to-end two-stage architecture. First, a data-driven pre-trained encoder is used to complete the sparse representation and basic feature extraction of image information. Second, a new attention mechanism is introduced to replace the self-attention mechanism in Transformer, and a hybrid Transformer and convolution module based on optimization-inspired is designed. Its iterative process leads to the unfolding framework, which approximates the original image stage by stage. Experimental results show that TCR-Net outperforms existing state-of-the-art CS methods while maintaining fast computational speed. Specifically, when the CS ratio is 0.10, the average PSNR on the test set used in this paper is improved by at least 0.8%, the average SSIM is improved by at least 1.5%, and the processing speed is higher than 70FPS. These quantitative results show that our method has high computational efficiency while ensuring high-quality image restoration.

Personality prediction via multi-task transformer architecture combined with image aesthetics

Abstract Social media has found its path into the daily lives of people. There are several ways that users communicate in which liking and sharing images stands out. Each image shared by a user can be analyzed from aesthetic and personality traits views. In recent studies, it has been proved that personality traits impact personalized image aesthetics assessment. In this article, the same pattern was studied from a different perspective. So, we evaluated the impact of image aesthetics on personality traits to check if there is any relation between them in this form. Hence, in a two-stage architecture, we have leveraged image aesthetics to predict the personality traits of users. The first stage includes a multi-task deep learning paradigm that consists of an encoder/decoder in which the core of the network is a Swin Transformer. The second stage combines image aesthetics and personality traits with an attention mechanism for personality trait prediction. The results showed that the proposed method had achieved an average Spearman Rank Order Correlation Coefficient (SROCC) of 0.776 in image aesthetic on the Flickr-AES database and an average SROCC of 0.6730 on the PsychoFlickr database, which outperformed related SOTA (State of the Art) studies. The average accuracy performance of the first stage was boosted by 7.02 per cent in the second stage, considering the influence of image aesthetics on personality trait prediction.

Cyber Intrusion Detection System Using Deep Learning Approach

Abstract: Technological developments in network communications have led to a remarkable increase in network traffic and an explosion in the use of linked devices across a number of commercial fields. Systems for detecting intrusions that can recognize malicious assaults from traffic data might be useful instruments for protecting company assets from illegal access. This project suggests a two-stage architecture for an intrusion detection system, where an auto encoder (AE) and the grey wolf algorithm (GWO) choose features. It is evaluated using the Bot-Iot and NSL-KDD datasets, yielding better accuracy levels for binary and multiclass attack categorization. This method outperforms the latest intrusion detection techniques in terms of categorization using an ideal selection of traffic characteristics.

Choroidal Layer Analysis in OCT images via Ambiguous Boundary-aware Attention

Optical Coherence Tomography (OCT) is a commonly used retina imaging technique, and it is capable of revealing the morphology of the choroid. However, the segmentation and quantitative analysis of the sublayers and vessels in choroid are rarely explored, primarily due to the indistinct boundaries of choroidal sublayers, and imbalanced distribution of vessels observed in OCT imagery. In this paper, we propose a novel two-stage architecture called Choroidal Layer Analysis network (CLA), that may be considered the first attempt in this research community for joint segmentation of choroidal sublayers and choroidal vessels in OCT images. CLA employs the encoder–decoder network with the residual U-shape module as the backbone. In order to empower the ability of the segmentation model to identify the inconspicuous boundaries of choroidal sublayers, we introduce an Ambiguous Boundary Attention block (ABA) into the bottleneck of the encoder–decoder network in the first stage. For more accurate segmentation of large choroidal vessels with ambiguous contours and imbalanced spatial distribution, the second stage introduces an active contour-based loss to refine the contours of choroidal vessels simultaneously with precise identification of each vessel via contextual modeling. To train, test and validate the proposed model, we conducted a choroidal segmentation dataset containing 800 OCT images, with their sublayers and large choroidal vessels manually annotated. Experimental results demonstrate the superiority of the proposed approach compared with other state-of-the-art segmentation networks in large margins. It is worth noting that we also reconstructed the large choroidal vessels in three-dimensional (3D) based on the segmentation results, and multiple 3D morphological parameters were calculated. The statistical analysis of these parameters demonstrates significant differences between the healthy control and high myopia group, and this further confirms the proposed work may facilitate subsequent disease understanding and clinical decision-making.

ZOZI-Seg: A transformer and UNet cascade network with Zoom-Out and Zoom-In scheme for aortic dissection segmentation in enhanced CT images

Background & objectiveAortic dissection (AD) is a serious condition requiring rapid and accurate diagnosis. In this study, we aimed to improve the diagnostic accuracy of AD by presenting a novel method for aortic segmentation in computed tomography images that uses a combination of a transformer and a UNet cascade network with a Zoom-Out and Zoom-In scheme (ZOZI-seg). MethodsThe proposed method segments each compartment of the aorta, comprising the true lumen (TL), false lumen (FL), and thrombosis (TH) using a cascade strategy that captures both the global context (anatomical structure) and the local detail texture based on the dynamic patch size with ZOZI schemes. The ZOZI-seg model has a two-stage architecture using both a “3D transformer for panoptic context-awareness” and a “3D UNet for localized texture refinement.” The unique ZOZI strategies for patching were demonstrated in an ablation study. The performance of our proposed ZOZI-seg model was tested using a dataset from Asan Medical Center and compared with those of existing models such as nnUNet and nnFormer. ResultsIn terms of segmentation accuracy, our method yielded better results, with Dice similarity coefficients (DSCs) of 0.917, 0.882, and 0.630 for TL, FL, and TH, respectively. Furthermore, we indirectly compared our model with those in previous studies using an external dataset to evaluate its robustness and generalizability. ConclusionsThis approach may help in the diagnosis and treatment of AD in different clinical situations and provide a strong basis for further research and clinical applications.

A two-stage transformer based network for motor imagery classification

Brain-computer interfaces (BCIs) are used to understand brain functioning and develop therapies for neurological and neurodegenerative disorders. Therefore, BCIs are crucial in rehabilitating motor dysfunction and advancing motor imagery applications. For motor imagery, electroencephalogram (EEG) signals are used to classify the subject's intention of moving a body part without actually moving it. This paper presents a two-stage transformer-based architecture that employs handcrafted features and deep learning techniques to enhance the classification performance on benchmarked EEG signals. Stage-1 is built on parallel convolution based EEGNet, multi-head attention, and separable temporal convolution networks for spatiotemporal feature extraction. Further, for enhanced classification, in stage-2, additional features and embeddings extracted from stage-1 are used to train TabNet. In addition, a novel channel cluster swapping data augmentation technique is also developed to handle the issue of limited samples for training deep learning architectures. The developed two-stage architecture offered an average classification accuracy of 88.5 % and 88.3 % on the BCI Competition IV-2a and IV-2b datasets, respectively, which is approximately 3.0 % superior over similar recent reported works.

Transfer learning for auto-segmentation of 17 organs-at-risk in the head and neck: Bridging the gap between institutional and public datasets.

Auto-segmentation of organs-at-risk (OARs) in the head and neck (HN) on computed tomography (CT) images is a time-consuming component of the radiation therapy pipeline that suffers from inter-observer variability. Deep learning (DL) has shown state-of-the-art results in CT auto-segmentation, with larger and more diverse datasets showing better segmentation performance. Institutional CT auto-segmentation datasets have been small historically (n<50) due to the time required for manual curation of images and anatomical labels. Recently, large public CT auto-segmentation datasets (n>1000 aggregated) have become available through online repositories such as The Cancer Imaging Archive. Transfer learning is a technique applied when training samples are scarce, but a large dataset from a closely related domain is available. The purpose of this study was to investigate whether a large public dataset could be used in place of an institutional dataset (n>500), or to augment performance via transfer learning, when building HN OAR auto-segmentation models for institutional use. Auto-segmentation models were trained on a large public dataset (public models) and a smaller institutional dataset (institutional models). The public models were fine-tuned on the institutional dataset using transfer learning (transfer models). We assessed both public model generalizability and transfer model performance by comparison with institutional models. Additionally, the effect of institutional dataset size on both transfer and institutional models was investigated. All DL models used a high-resolution, two-stage architecture based on the popular 3D U-Net. Model performance was evaluated using five geometric measures: the dice similarity coefficient (DSC), surface DSC, 95th percentile Hausdorff distance, mean surface distance (MSD), and added path length. For a small subset of OARs (left/right optic nerve, spinal cord, left submandibular), the public models performed significantly better (p<0.05) than, or showed no significant difference to, the institutional models under most of the metrics examined. For the remaining OARs, the public models were inferior to the institutional models, although performance differences were small (DSC≤0.03, MSD<0.5mm) for seven OARs (brainstem, left/right lens, left/right parotid, mandible, right submandibular). The transfer models performed significantly better than the institutional models for seven OARs (brainstem, right lens, left/right optic nerve, left/right parotid, spinal cord) with a small margin of improvement (DSC≤0.02, MSD<0.4mm). When numbers of institutional training samples were limited, public and transfer models outperformed the institutional models for most OARs (brainstem, left/right lens, left/right optic nerve, left/right parotid, spinal cord, and left/right submandibular). Training auto-segmentation models with public data alone was suitable for a small number of OARs. Using only public data incurred a small performance deficit for most other OARs, when compared with institutional data alone, but may be preferable over time-consuming curation of a large institutional dataset. When a large institutional dataset was available, transfer learning with models pretrained on a large public dataset provided a modest performance improvement for several OARs. When numbers of institutional samples were limited, using the public dataset alone, or as a pretrained model, was beneficial for most OARs.

CETR: CenterNet-Vision transformer model for wheat head detection

&lt;p&gt;Wheat head detection is a critical task in precision agriculture for estimating crop yield and optimizing agricultural practices. Conventional object detection architectures often struggle with detecting densely packed and overlapping wheat heads in complex agricultural field images. To address this challenge, a novel CEnternet-vision TRansformer model for Wheat Head Detection (CETR) is proposed. CETR model combines the strengths of two cutting-edge technologies—CenterNet and Vision Transformer. A dataset of agricultural farm images labeled with precise wheat head annotations is used to train and evaluate the CETR model. Comprehensive experiments were conducted to compare CETR’s performance against convolutional neural network model commonly used in agricultural applications. The higher mAP value of 0.8318 for CETR compared against AlexNet, VGG19, ResNet152 and MobileNet indicates that the CETR model is more effective in detecting wheat heads in agricultural images. It achieves a higher precision in predicting bounding boxes that align well with the ground truth, resulting in more accurate and reliable wheat head detection. The higher performance of CETR can be attributed to the combination of CenterNet and ViT as a two-stage architecture taking advantage of both methods. Moreover, the transformer-based architecture of CETR enables better generalization across different agricultural environments, making it a suitable solution for automated agricultural applications.&lt;/p&gt;

Weakly supervised large-scale pancreatic cancer detection using multi-instance learning.

Early detection of pancreatic cancer continues to be a challenge due to the difficulty in accurately identifying specific signs or symptoms that might correlate with the onset of pancreatic cancer. Unlike breast or colon or prostate cancer where screening tests are often useful in identifying cancerous development, there are no tests to diagnose pancreatic cancers. As a result, most pancreatic cancers are diagnosed at an advanced stage, where treatment options, whether systemic therapy, radiation, or surgical interventions, offer limited efficacy. A two-stage weakly supervised deep learning-based model has been proposed to identify pancreatic tumors using computed tomography (CT) images from Henry Ford Health (HFH) and publicly available Memorial Sloan Kettering Cancer Center (MSKCC) data sets. In the first stage, the nnU-Net supervised segmentation model was used to crop an area in the location of the pancreas, which was trained on the MSKCC repository of 281 patient image sets with established pancreatic tumors. In the second stage, a multi-instance learning-based weakly supervised classification model was applied on the cropped pancreas region to segregate pancreatic tumors from normal appearing pancreas. The model was trained, tested, and validated on images obtained from an HFH repository with 463 cases and 2,882 controls. The proposed deep learning model, the two-stage architecture, offers an accuracy of 0.907 0.01, sensitivity of 0.905 0.01, specificity of 0.908 0.02, and AUC (ROC) 0.903 0.01. The two-stage framework can automatically differentiate pancreatic tumor from non-tumor pancreas with improved accuracy on the HFH dataset. The proposed two-stage deep learning architecture shows significantly enhanced performance for predicting the presence of a tumor in the pancreas using CT images compared with other reported studies in the literature.

Video-Based Multiphysiological Disentanglement and Remote Robust Estimation for Respiration.

Remote noncontact respiratory rate estimation by facial visual information has great research significance, providing valuable priors for health monitoring, clinical diagnosis, and anti-fraud. However, existing studies suffer from disturbances in epidermal specular reflections induced by head movements and facial expressions. Furthermore, diffuse reflections of light in the skin-colored subcutaneous tissue caused by multiple time-varying physiological signals independent of breathing are entangled with the intention of the respiratory process, leading to confusion in current research. To address these issues, this article proposes a novel network for natural light video-based remote respiration estimation. Specifically, our model consists of a two-stage architecture that progressively implements vital measurements. The first stage adopts an encoder-decoder structure to recharacterize the facial motion frame differences of the input video based on the gradient binary state of the respiratory signal during inspiration and expiration. Then, the obtained generative mapping, which is disentangled from various time-varying interferences and is only linearly related to the respiratory state, is combined with the facial appearance in the second stage. To further improve the robustness of our algorithm, we design a targeted long-term temporal attention module and embed it between the two stages to enhance the network's ability to model the breathing cycle that occupies ultra many frames and to mine hidden timing change clues. We train and validate the proposed network on a series of publicly available respiration estimation datasets, and the experimental results demonstrate its competitiveness against the state-of-the-art breathing and physiological prediction frameworks.

Detecting fake reviewers from the social context with a graph neural network method

With the development of mobile Web technologies, people can easily seek advice from social media before making purchases or decisions. Some companies employ expert writers to fabricate reviews or use automated techniques to improve the appeal of their products or services, or to undermine the credibility of their rivals. This obstructs the detection of fake reviews and reviewers. This paper proposes a novel graph neural network-based framework for detecting spammers, who originate fake reviews in discussion forums to capture information from different social network combinations in various subgraphs. These subgraphs include a complete social context graph, homogeneous user–user subgraph, and heterogeneous user–post subgraph. A novel two-stage architecture with focal loss was designed to create a training model. This model can be applied to solve the issue of imbalance data classification. The proposed framework was applied to evaluate a ground truth dataset collected from an actual fraudulent review event on a discussion forum. The experimental results show that this aggregate social context representation method can be effectively applied to detect fake reviewers.

An intelligent fault detection and diagnosis monitoring system for reactor operational resilience: Fault diagnosis

Various advanced reactor designs proposed in recent years envision deployment scenarios which feature reactor operations with significantly reduced staffing or even completely autonomous frameworks to reduce the operations and management costs of the plants. Many SMR and microreactor designs feature extended fuel cycles which limit inspection intervals, reduced access to critical components, and load-following capabilities that expose the reactor unit to different transients. Safe and reliable semi- or fully autonomous operations under these challenging operational regimes must be enabled through an on-line monitoring (OLM) system that effectively detects and diagnoses malfunctions in the reactor plant. This work presents the development of the Fault Diagnosis Module of the previously proposed data-driven OLM system for reactor operations: the Fault Detection and Diagnosis Monitoring System (FDDMS). The Fault Diagnosis Module monitors various sensor signatures from multiple systems and components at a nuclear power plant and accurately diagnoses the type and location of a malfunction. When integrated into the complete FDDMS methodology, the Fault Diagnosis Module can provide power transient dependent fault characterization by using separate convolutional neural network (CNN) models for steady state, ramping up in power, and ramping down in power operations, making the FDDMS especially applicable to load-following operational strategies. Two separate diagnosis approaches were explored in this paper for the Fault Diagnosis Module architecture: Hierarchical and End-to-End. The Hierarchical approach is a two-stage architecture in which the first stage uses a single CNN to classify the plant subsystem where the malfunction initiated. Subsequently in the second stage, a separate CNN is used for each subsystem to describe the specific fault type. The End-to-End approach uses a single CNN to directly classify the fault type from the overall list. To efficiently utilize computational resources, the hyperband intelligent hyperparameter optimization method to generate optimal CNN architectures. The diagnosis approaches were compared in terms of precision rate, recall rate, F1-score, and total accuracy for the possible fault types. Both diagnosis approaches produced good and satisfactory performance by generating total diagnosis accuracies above 99% on 17 different malfunction scenarios for all three power transient datasets. Additionally, robustness against noisy sensor data was tested, with models maintaining 99–100% accuracy at various levels of noise, and an illustrative real-time application of the methodology is provided.

A novel wavelet-transform-based convolution classification network for cervical lymph node metastasis of papillary thyroid carcinoma in ultrasound images.

Preoperative assessment of cervical lymph nodes metastasis (CLNM) for accurate qualitative and locating diagnosis is important for choosing the best treatment option for patients with papillary thyroid cancer. Non-destructive, non-invasive ultrasound is currently the imaging method of choice for lymph node metastatic assessment. For lymph node characteristics and ultrasound images, this paper proposes a multitasking network framework for diagnosing metastatic lymph nodes in ultrasound images, in which localization module not only provides information on the location of lymph nodes to focus on the peripheral and self regions of lymph nodes, but also provides structural features of lymph nodes for subsequent classification module. In the classification module, we design a novel wavelet-transform-based convolution network. Wavelet transform is introduced into the deep learning convolution module to analyze ultrasound images in both spatial and frequency domains, which effectively enriches the feature information and improves the classification performance of the model without increasing the model parameters. We collected 510 patient data (N = 1376) from Shanghai Sixth People's Hospital regarding ultrasound lymph nodes in the neck, as well as used three publicly available ultrasound datasets, including SCUI2020 (N = 2914), DDTI (N = 480), and BUSI (N = 780). Compared to the optimal two-stage model, our model has improved its accuracy and AUC indexes by 5.83% and 4%, which outperforms the two-stage architectures and also surpasses the latest classification networks.

Deep neural architecture for natural language image synthesis for Tamil text using BASEGAN and hybrid super resolution GAN (HSRGAN)

Tamil is a language that has the most extended history and is a conventional language of India. It has antique origins and a distinct tradition. A study reveals that at the beginning of the twenty-first century, more than 66 million people spoke Tamil. In the present time, image synthesis from text emerged as a promising advancement in computer vision applications. The research work done so far in intelligent systems is trained in universal language but still has not achieved the desired development level in regional languages. Regional languages have a greater scope for developing applications and will enhance more research areas to be explored, ruling out the barrier. The current work using Auto Encoders failed at the point of providing vivid information along with essential descriptions of the synthesised images. The work aims to generate embedding vectors using a language model headed by image synthesis using GAN (Generative Adversarial Network) architecture. The proposed method is divided into two stages: designing a language model TBERTBASECASE model for generating embedding vectors. Synthesising images using Generative Adversarial Network called BASEGAN, the resolution has been improved through two-stage architecture named HYBRID SUPER RESOLUTION GAN. The work uses Oxford-102 and CUB-200 datasets. The framework efficiency has been measured using F1 Score, Fréchet inception distance (FID), and Inception Score (IS). Language and image synthesis architecture proposed can bridge the gap between the research ideas in regional languages.

Personalized Ambient Pollution Estimation Based on Stationary-Camera-Taken Images Under Cross-Camera Information Sharing in Smart City

Timely and high-density air quality monitoring is essential for the development of future smart cities. The images captured from widely deployed stationary cameras can be transferred quickly via the Internet of Things (IoT) to facilitate ambient pollution estimation anytime anywhere. Imaged-based air pollution estimation is normally formulated as a supervised learning problem, relying on an extended number of image samples. However, individual stationary cameras can offer only very limited samples and scenes, whilst locally trained estimation models can easily overfit. A global method was proposed to address this challenge. The global model was trained via images captured from different cameras. However, such a model is less effective in extracting local features from scenes. A personalized method is therefore proposed to improve not only the generalization of the estimation model but also to preserve the local characteristics of individual cameras. Our personalized method consists of a two-stage architecture: (1) images from different cameras are used to train the global estimation model to avoid overfitting due to fixed scenes and small sample size; (2) the global model is further refined by images captured from individual cameras separately for adapting local characteristics. To evaluate our proposed personalized method, a large dataset was constructed, based on stationary camera-taken images captured in Hong Kong, consisting of different pollution measurements, including PM2.5, PM10, NO2 and O3. As compared to the local model, our proposed personalized model has reduced average MAE by 5.68% and average SMAPE by 6.82%, and improved average r by 4.69%.

A Pareto-based two-stage evolutionary algorithm for flexible job shop scheduling problem with worker cooperation flexibility

The previous studies on the flexible job shop scheduling problems (FJSP) with machine flexibility and worker flexibility normally assume that each machine is operated by one worker at any time. However, it is not accurate in many cases because many workers may be required for machines in processing complex operations. Hence, this paper studies a universal version, i.e., FJSP with worker cooperation flexibility (FJSPWC), which defines that each machine can be used only if their required workers are prepared. A mixed-integer linear programming model tuned by CPLEX is established for the problem aiming to collaboratively minimize the makespan, maximum workload of machines and maximum workload of workers. To solve the problem efficiently, a Pareto-based two-stage evolutionary algorithm (PTEA) is proposed. In the PTEA, a well-tailored initialization operator and the NSGA-II structure are designed for global exploration in the first stage, and a competitive objective-based local search operator is developed to improve its local search ability and accelerate the convergence in the second stage. Extensive experiments based on fifty-eight newly formulated benchmarks are carried out to validate the effectiveness of the well-designed initialization operator and two-stage architecture. Comprehensive experiments are performed to evaluate the proposed PTEA, and the results reveal that the PTEA is superior to four comparison algorithms concerning the distribution, convergence, and overall performance.

Unifying Convolution and Transformer for Efficient Concealed Object Detection in Passive Millimeter-Wave Images

Passive millimeter-wave (PMMW) imaging is an ideal technique for concealed object detection in non-contact security inspection scenarios, and has been widely used in railway stations and airports. However, there are still several challenges that limit the accuracy of detection in PMMW images: low resolution, small objects and complex background interference. The existing deep learning-based methods mainly adopt two-stage architecture with convolutional neural networks (CNN) as the backbone for feature extraction, while the low speed of two-stage architecture and limited receptive field of CNN impede the further improvement of the intelligent inspection system. In this paper, we propose a one-stage anchor-free detector that combines the merits of CNN and transformer to solve these problems, and we avoid the tradeoff between accuracy and computational complexity that most hierarchical transformers make by constricting self-attention within pre-defined local windows. Specifically, we design a novel backbone to model low-level local and high-level global features at different scales via CNN and transformer. We also introduce object queries to guide the detector to perceive the concealed objects from the background noise, and these learnable queries are further utilized to form a full-size object-aware attention mechanism. Besides, to select the optimal positive samples for the anchor-free detector, we propose a novel label assignment strategy by employing Gaussian distribution to adaptively model the objects with various shapes. Experimental results on our self-developed PMMW imager demonstrate that the proposed method outperforms the state-of-the-art methods in terms of accuracy and speed.

An End-to-End Online Traffic-Risk Incident Prediction in First-Person Dash Camera Videos

Predicting traffic risk incidents in first-person helps to ensure a safety reaction can occur before the incident happens for a wide range of driving scenarios and conditions. One challenge to building advanced driver assistance systems is to create an early warning system for the driver to react safely and accurately while perceiving the diversity of traffic-risk predictions in real-world applications. In this paper, we aim to bridge the gap by investigating two key research questions regarding the driver’s current status of driving through online videos and the types of other moving objects that lead to dangerous situations. To address these problems, we proposed an end-to-end two-stage architecture: in the first stage, unsupervised learning is applied to collect all suspicious events on actual driving; in the second stage, supervised learning is used to classify all suspicious event results from the first stage to a common event type. To enrich the classification type, the metadata from the result of the first stage is sent to the second stage to handle the data limitation while training our classification model. Through the online situation, our method runs 9.60 fps on average with 1.44 fps on standard deviation. Our quantitative evaluation shows that our method reaches 81.87% and 73.43% for the average F1-score on labeled data of CST-S3D and real driving datasets, respectively. Furthermore, the proposed method has the potential to assist distribution companies in evaluating the driving performance of their driver by automatically monitoring near-miss events and analyzing driving patterns for training programs to reduce future accidents.

Higher-order memory guided temporal random walk for dynamic heterogeneous network embedding

Network embedding (NE) aims at learning node embeddings via structure-based sampling. However, there are complex patterns in network structure (heterogeneity, higher-order dependence, dynamics) in the real world. The existing methods suffer from high dependence and constraints on manually designed higher-order structures and loss of fine-grained temporal information. To solve the above challenges, we propose a novel higher-order memory guided temporal random walk for dynamic heterogeneous network embedding (HoMo-DyHNE). The proposed model is a two-stage architecture consisting of a meta-structure-independent random walk algorithm namely HoMo-TRW with transition vectors and higher-order memory, and a Hawkes-based featured Skip-gram (HFSG) incorporating a multivariate Hawkes point process to measure the history-current association intensity. Extensive experiments demonstrate the superior effectiveness of our proposed method.