Input Stream Research Articles

Interactive Dashboard with Visual Sensing and Fast Reactivity

These days, technology is growing rapidly, and the market has been introduced with lots of fascinating ways to interact with computers. The advancement of deep learning models and hardware technology also enables more applications with fancy features to be built. The importance of hand gesture recognition has increased due to the prevalence of touchless applications. However, developing an efficient recognition system needs to overcome the challenges of hand segmentation, local hand shape representation, global body configuration representation, and a gesture sequence model. This paper proposed an interactive dashboard that could react to hand gestures. This is also an initiative of the Tunku Abdul Rahman University College (TAR UC) Smart Campus project. Deep learning models were investigated in this research and the optimal model was selected for the dashboard. In addition, 20BN Jester Dataset was used for the dashboard development. To set up a more user-friendly dashboard, the data communication stream between the captured input stream and commands among the devices were studied. As to achieve higher responsiveness from the dashboard, evaluation on data communication protocols which were used to pass the input data were included in the study.

Exploring the improvement potentials in an existing industrial olefin plant through advanced exergy-based analyses

The production of light olefins via the thermal cracking of hydrocarbons is the most energy-demanding process in the petrochemical industry, and an important producer of greenhouse gasses worldwide. But, which sections or components of this process are major contributors to its energy-intensiveness, and high environmental impacts? And, how these problems can be avoided and to what extent?Presently, conventional and advanced exergetic, exergoeconomic, and exergoenvironmental analyses are introduced as the most promising methods which could answer these questions, therefore, in this work, are used to identify the real priorities of improvement in an olefin plant in the south of Iran (Assaluyeh), by first simulating the plant in ASPEN HYSYS simulator. The environmental impacts of components and input streams of plant were calculated using Eco-Indicator 99 method of Life Cycle Assessment (LCA). Results of the study indicated that although the cracking furnace causes 55.3% of total exergy destruction of the plant, only 15.36% of it is avoidable endogenous. Furthermore, the considerable share of steam system in the total exergy destruction of plant (as the second exergy destructive section with a share of 14%) showed that ignoring this zone of an olefin plant in the exergy-based analyses leads to major mistakes to diagnose the priorities of improvements. The results showed that considering prespecified costs and environmental impacts for internal material or energy streams produced within the process introduces a significant error to identify the improvement priorities regarding the economic and environmental aspects.

Rethinking statistical learning as a continuous dynamic stochastic process, from the motor systems perspective.

The brain integrates streams of sensory input and builds accurate predictions, while arriving at stable percepts under disparate time scales. This stochastic process bears different unfolding dynamics for different people, yet statistical learning (SL) currently averages out, as noise, individual fluctuations in data streams registered from the brain as the person learns. We here adopt a new analytical approach that instead of averaging out fluctuations in continuous electroencephalographic (EEG)-based data streams, takes these gross data as the important signals. Our new approach reassesses how individuals dynamically learn predictive information in stable and unstable environments. We find neural correlates for two types of learners in a visuomotor task: narrow-variance learners, who retain explicit knowledge of the regularity embedded in the stimuli. They seem to use an error-correction strategy steadily present in both stable and unstable environments. This strategy can be captured by current optimization-based computational frameworks. In contrast, broad-variance learners emerge only in the unstable environment. Local analyses of the moment-by-moment fluctuations, naïve to the overall outcome, reveal an initial period of memoryless learning, well characterized by a continuous gamma process starting out exponentially distributed whereby all future events are equally probable, with high signal (mean) to noise (variance) ratio. The empirically derived continuous Gamma process smoothly converges to predictive Gaussian signatures comparable to those observed for the error-corrective mode that is captured by current optimization-driven computational models. We coin this initially seemingly purposeless stage exploratory. Globally, we examine a posteriori the fluctuations in distributions' shapes over the empirically estimated stochastic signatures. We then confirm that the exploratory mode of those learners, free of expectation, random and memoryless, but with high signal, precedes the acquisition of the error-correction mode boasting smooth transition from exponential to symmetric distributions' shapes. This early naïve phase of the learning process has been overlooked by current models driven by expected, predictive information and error-based learning. Our work demonstrates that (statistical) learning is a highly dynamic and stochastic process, unfolding at different time scales, and evolving distinct learning strategies on demand.

3D-VDNet: Exploiting the vertical distribution characteristics of point clouds for 3D object detection and augmentation

• Vertical distribution characteristics (VDCs) of point clouds are robust to point-sparsity and provide semantic information. • Decorating the pillars with VDCs and semantic label is conducive to robust and discriminative feature extraction. • The adaptive object augmentation (AOA) paradigm overcomes the conflict between augmented objects and corresponding scenes. • 3D-VDNet significantly outperforms the baseline in all classes (Car, Pedestrian, and Cyclist). • The AOA paradigm with a stable performance across detector types (voxel- or point-based). Accurate 3D object detection is limited by the sparsity of LiDAR-based point clouds. The vertical distribution characteristics (VDCs) of point clouds in pillars are robust to point-sparsity and provide informative semantic information on objects. Based on this, we propose a novel 3D object detection framework where the VDCs of point clouds are exploited to optimize feature extraction and object augmentation. More specifically, a Spatial Feature Aggregation module is proposed to perform robust feature extraction by decorating pillars with the VDCs. To spatially enhance semantic embeddings, we employ VDCs to construct a voxelized semantic map, acting as an additional input stream. Moreover, we develop an Adaptive Object Augmentation (AOA) paradigm, which adopts the VDC searching of suitable ground regions to “paste” virtual objects, thus avoiding conflicts with new scenes. Extensive experiments on the KITTI dataset demonstrate that our framework can significantly outperform the baseline, achieving 3.74%/1.59% moderate AP improvements on the Car 3D/BEV benchmarks with 38 FPS inference speed. Furthermore, we prove the stable performance of our AOA module across different detectors.

Immature olfactory sensory neurons provide behaviourally relevant sensory input to the olfactory bulb

Postnatal neurogenesis provides an opportunity to understand how newborn neurons integrate into circuits to restore function. Newborn olfactory sensory neurons (OSNs) wire into highly organized olfactory bulb (OB) circuits throughout life, enabling lifelong plasticity and regeneration. Immature OSNs form functional synapses capable of evoking firing in OB projection neurons but what contribution, if any, they make to odor processing is unknown. Here, we show that immature OSNs provide odor input to the mouse OB, where they form monosynaptic connections with excitatory neurons. Importantly, immature OSNs respond as selectively to odorants as mature OSNs and exhibit graded responses across a wider range of odorant concentrations than mature OSNs, suggesting that immature and mature OSNs provide distinct odor input streams. Furthermore, mice can successfully perform odor detection and discrimination tasks using sensory input from immature OSNs alone. Together, our findings suggest that immature OSNs play a previously unappreciated role in olfactory-guided behavior.

Improving GPU performance in multimedia applications through FPGA based adaptive DMA controller

Purpose Deep learning techniques are unavoidable in a variety of domains such as health care, computer vision, cyber-security and so on. These algorithms demand high data transfers but require bottlenecks in achieving the high speed and low latency synchronization while being implemented in the real hardware architectures. Though direct memory access controller (DMAC) has gained a brighter light of research for achieving bulk data transfers, existing direct memory access (DMA) systems continue to face the challenges of achieving high-speed communication. The purpose of this study is to develop an adaptive-configured DMA architecture for bulk data transfer with high throughput and less time-delayed computation. Design/methodology/approach The proposed methodology consists of a heterogeneous computing system integrated with specialized hardware and software. For the hardware, the authors propose an field programmable gate array (FPGA)-based DMAC, which transfers the data to the graphics processing unit (GPU) using PCI-Express. The workload characterization technique is designed using Python software and is implementable for the advanced risk machine Cortex architecture with a suitable communication interface. This module offloads the input streams of data to the FPGA and initiates the FPGA for the control flow of data to the GPU that can achieve efficient processing. Findings This paper presents an evaluation of a configurable workload-based DMA controller for collecting the data from the input devices and concurrently applying it to the GPU architecture, bypassing the hardware and software extraneous copies and bottlenecks via PCI Express. It also investigates the usage of adaptive DMA memory buffer allocation and workload characterization techniques. The proposed DMA architecture is compared with the other existing DMA architectures in which the performance of the proposed DMAC outperforms traditional DMA by achieving 96% throughput and 50% less latency synchronization. Originality/value The proposed gated recurrent unit has produced 95.6% accuracy in characterization of the workloads into heavy, medium and normal. The proposed model has outperformed the other algorithms and proves its strength for workload characterization.

Sensory Target Detection at Local and Global Timescales Reveals a Hierarchy of Supramodal Dynamics in the Human Cortex.

To ensure survival in a dynamic environment, the human neocortex monitors input streams from different sensory organs for important sensory events. Which principles govern whether different senses share common or modality-specific brain networks for sensory target detection? We examined whether complex targets evoke sustained supramodal activity while simple targets rely on modality-specific networks with short-lived supramodal contributions. In a series of hierarchical multisensory target detection studies (n = 77, of either sex) using EEG, we applied a temporal cross-decoding approach to dissociate supramodal and modality-specific cortical dynamics elicited by rule-based global and feature-based local sensory deviations within and between the visual, somatosensory, and auditory modality. Our data show that each sense implements a cortical hierarchy orchestrating supramodal target detection responses, which operate at local and global timescales in successive processing stages. Across different sensory modalities, simple feature-based sensory deviations presented in temporal vicinity to a monotonous input stream triggered a mismatch negativity-like local signal which decayed quickly and early, whereas complex rule-based targets tracked across time evoked a P3b-like global neural response which generalized across a late time window. Converging results from temporal cross-modality decoding analyses across different datasets, we reveal that global neural responses are sustained in a supramodal higher-order network, whereas local neural responses canonically thought to rely on modality-specific regions evolve into short-lived supramodal activity. Together, our findings demonstrate that cortical organization largely follows a gradient in which short-lived modality-specific as well as supramodal processes dominate local responses, whereas higher-order processes encode temporally extended abstract supramodal information fed forward from modality-specific cortices.SIGNIFICANCE STATEMENT Each sense supports a cortical hierarchy of processes tracking deviant sensory events at multiple timescales. Conflicting evidence produced a lively debate around which of these processes are supramodal. Here, we manipulated the temporal complexity of auditory, tactile, and visual targets to determine whether cortical local and global ERP responses to sensory targets share cortical dynamics between the senses. Using temporal cross-decoding, we found that temporally complex targets elicit a supramodal sustained response. Conversely, local responses to temporally confined targets typically considered modality-specific rely on early short-lived supramodal activation. Our finding provides evidence for a supramodal gradient supporting sensory target detection in the cortex, with implications for multiple fields in which these responses are studied (e.g., predictive coding, consciousness, and attention).

A signal packet router for the upgrade of the Muon spectrometer at the ATLAS experiment

We present the design, characterization, and production of a signal packet router for the upgrade of the ATLAS forward muon detector. The router serves as a packet switch in one of the primary trigger paths. It handles up to 12 serial inputs at 4.8 Gbps/input from on-detector electronics and outputs four 4.8 Gbps to the trigger processing circuits. In the 12-to-4 packet switching process, the input streams are decoded and synchronized to a common clock domain for data packets forwarding and NULL packets suppression. Challenges in the design include operations of multi-giga hertz serial links with low and constant latency (<100 ns) for packet routing, as well as mitigation of radiation-induced effects in the harsh radiation environment over 10 years of detector operation. The Router design has been fully characterized and demonstrated to meet all specifications. In total 256 Routers were installed in the detector. Its design concept can be used in other similar applications, particularly in a radiation environment.

Toward Autonomous Production of mRNA-Therapeutics in the Light of Advanced Process Control and Traditional Control Strategies for Chromatography

mRNA-based therapeutics are predicted to have a bright future. Recently, a B2C study was published highlighting the critical bottlenecks of mRNA manufacturing. The study focused on supply bottlenecks of various chemicals as well as shortages of skilled personnel. The assessment of existing messenger ribonucleic acid (mRNA) vaccine processing shows the need for continuous manufacturing processes that are capable of about 80% chemical reduction and more than 70% personnel at factor five more efficient equipment utilization. The key technology to solve these problems is both a higher degree of automation and the maximization of process throughput. In this paper, the application of a quality-by-design process development approach is demonstrated, using process models as digital twins. Their systematic application leads to both robust optimized process parameters, with an increase in productivity of up to 108%, and sophisticated control concepts, preventing batch failures and minimizing the operating workload in terms of personnel and chemicals’ consumption. The approach thereby provides a data-driven decision basis for the industrialization of such processes, which fulfills the regulatory requirements of the approval authorities and paves the way for PAT integration. In the process investigated, it was shown that conventional PID-based controls can regulate fluctuations in the input streams sufficiently well. Model-based control based on digital twins may have potential above all in a further increase in productivity, but is not mandatory to implement for the industrialization of continuous mRNA manufacturing.

Overall particles flow pattern in a two-zone gas-solid fluidized bed with a secondary-gas stream

Fluidized bed reactors with multiple input streams have been developed over the last decades to satisfy industries in the scope of multiphase catalytic reactions, and two-zone fluidized beds have been developed to perform reaction and catalyst regeneration simultaneously. However, the presence of the secondary gas nozzle in the bed significantly impacts the hydrodynamics and efficiency. Particle image velocimetry (PIV) and digital image analysis (DIA) techniques have been used to study different particle flow patterns in a two-zone pseudo-2D fluidized bed with lateral gas injection. The secondary flow nozzles' location has been changed, and different particle flow patterns have been observed in the fluidized bed. It was found that at low excess gas flow rates, when the lateral nozzle is located at the middle part of the bed, particles move downwardly at the bed center and upwardly near the walls; while when the lateral nozzle is located at the top part of the bed, particles have a reverse movement trend. At higher gas velocities for both mentioned nozzle locations, particles move upwardly at the bed center and downwardly near the walls. The formation of large bubbles was less probable when the nozzle was located at the middle of the bed.

Affective context and its uncertainty drive momentary affective experience.

Affect fluctuates in a moment-to-moment fashion, reflecting the continuous relationship between the individual and the environment. Despite substantial research, there remain important open questions regarding how a stream of sensory input is dynamically represented in experienced affect. Here, approaching affect as a temporally dependent process, we show that momentary affect is shaped by a combination of the affective impact of stimuli (i.e., visual images for the current studies) and previously experienced affect. We also found that this temporal dependency is influenced by uncertainty of the affective context. Participants in each trial viewed sequentially presented images and subsequently reported their affective experience, which was modeled based on images' normative affect ratings and participants' previously reported affect. Study 1 showed that self-reported valence and arousal in a given trial is partly shaped by the affective impact of the given images and previously experienced affect. In Study 2, we manipulated context uncertainty by controlling occurrence probabilities for normatively pleasant and unpleasant images in separate blocks. Increasing context uncertainty (i.e., random occurrence of pleasant and unpleasant images) was associated with increased negative affect. In addition, the relative contribution of the most recent image to experienced pleasantness increased with increasing context uncertainty. Taken together, these findings provide clear behavioral evidence that momentary affect is a temporally dependent and continuous process, which reflects the affective impact of recent input variables and the previous internal state, and that this process is sensitive to the affective context and its uncertainty. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Control theory applications when describing centrifugal concentration processes for mineral raw materials

The article discusses an approach aimed at developing an application methodology for the principles and models used in the theory of process control automation. The problems solved in the article are as follows: to try and represent the gravity concentration process (centrifugal concentration) in the form of a transfer function and to establish its parameters. In this case, the limited amount of data obtained in the course of the experiments is the main limitation for building models in the form of differential equations. An original processing and application methodology is proposed for the averaged data used when designing the model. This approach may be applicable to other ore preparation and processing stages as it involves equipment operation using the black box principle. The study is based on replacement of instantaneous characteristics of the input stream with their averaged values and subsequent integration, allowing the creation of a numerical model that coincides with the actual data. Transfer functions and their parameters for tailings and concentrate were established separately. MatLAB software with the Simulink package was used to verify the calculations. As a result, the transfer function was found for a KC-CVD6 device used at a gold deposit and satisfactory simulation results were obtained. The work uses automatic systems identification, system analysis, and mathematical modeling methods, namely, numerical solutions of differential equations using the MatLAB desktop environment with the Simulink package.

Research of autoencoder-based user biometric verification with motion patterns

In the current research, we continue our previous study regarding motion-based user biometric verification, which consumes sensory data. Sensory-based verification systems empower the continuous authentication narrative – as physiological biometric methods mainly based on photo or video input meet a lot of difficulties in implementation. The research aims to analyze how various components of sensor data from an accelerometer affect and contribute to defining the process of unique person motion patterns and understanding how it may express the human behavioral patterns with different activity types. The study used the recurrent long-short-term-memory autoencoder as a baseline model. The choice of model was based on our previous research. The research results have shown that various data components contribute differently to the verification process depending on the type of activity. However, we conclude that a single sensor data source may not be enough for a robust authentication system. The multimodal authentication system should be proposed to utilize and aggregate the input streams from multiple sensors as further research.

Release of Endogenous Nutrients Drives the Transformation of Nitrogen and Phosphorous in the Shallow Plateau of Lake Jian in Southwestern China

Eutrophication remediation is an ongoing priority for protecting aquatic ecosystems, especially in plateau lakes with fragile ecologies and special tectonic environments. However, current strategies to control the phosphorus (P) and nitrogen (N) levels in eutrophication sites have been mainly guided by laboratory experiments or literature reviews without in-field analyses of the geochemical processes associated with the hydrological and eutrophic characteristics of lakes. This study analyzed the water quality parameters of 50 sites at Lake Jian in May 2019, a moderate eutrophication shallow plateau lake, based on dissolved/sedimentary nitrogen, phosphorous and organic matter, grain size, C/N ratios and stable isotope ratios of δ13C or δ15N in sediments. The results showed that the average total nitrogen (TN) and total phosphorus (TP) concentrations in the lake water were 0.57 mg/L and 0.071 mg/L, respectively. The TN and TP contents of surface sediment ranged from 2.15 to 9.55 g/kg and 0.76 to 1.74 g/kg, respectively. Stable isotope and grain source analysis indicated that N in sediments mainly existed in organic matter form and P mainly occurred as inorganic mineral adsorption. Endogenous pollution contributed to &gt;20% of TN. Furthermore, our findings showed that phosphorus was the nutrient that limited eutrophication at Lake Jian, unlike other eutrophic shallow lakes. In contrast, the nutrient levels in the sediment and input streams belonged entirely to the N-limitation state. The distinctness in release intensity of N and P could modify the N/P limitation in the lake, which affects algae growth and nutrient control. These results demonstrated that reducing exogenous nutrients might not effectively mitigate lake eutrophication due to their endogenous recycling; thus, detailed nutrient monitoring is needed to preserve aquatic ecosystems.

A machine learning approach for clinker quality prediction and nonlinear model predictive control design for a rotary cement kiln

AbstractCement manufacturing is energy‐intensive (5Gj/t) and comprises a significant portion of the energy footprint of concrete systems. Incorporating modern monitoring, simulation and control systems will allow lower energy use, lower environmental impact, and lower costs of this widely used construction material. One of the goals of the CESMII roadmap project on the Smart Manufacturing of Cement included developing an analytical process model for clinker quality that includes the chemistry of the kiln feed and accounts for critical process variables. This predictive model will be used in nonlinear model predictive control system designed to significantly reduce process energy use while maintaining or improving product quality. In the cement manufacturing plant used in this study, the kiln feed (meal) is tested every 12 h and used to estimate the mineral composition of the cement kiln output (clinker) using the stoichiometry‐based Bogue's model and the expertise of the plant operators. During kiln operation, kiln output (clinker) is sampled and tested every 2 h to measure its chemical and mineral composition. The predicted and measured values of the clinker composition are used by the plant operators to adjust the kiln input stream and the production process characteristics to maintain stable operation and uniform product quality. However, the time delay between prediction and testing, along with inaccuracies inherent in the Bogue's model have made any process changes designed to minimize energy use problematic, especially in‐light of potential clinker quality issues that process changes often pose. A new analytical model that integrates quality information and process operation information has been developed from data collected from 2 years of production from an operating cement facility. To make the model fuel‐type‐independent, consumed heat energy was computed in the model instead of fuel type and amount. A Feedforward Network was trained and tailored from collected data. Many data‐based simulations were conducted to quantitatively evaluate the proposed model and the 5‐fold cross‐validation procedure was used to test the models. The resulting predictive model was shown to have a low root mean square error (MSE) with respect to the estimated clinker mineral composition compared to that using the industry standard “Bogue’ model”. The end goal of this work was to develop a single machine learning tool that allows the use of quality control data and process control variables to improve energy efficiency of the process in a continuous fashion. The proposed nonlinear model predictive control system (NMPC) can generate predicted kiln production characteristics based on manipulated variables in manner that accurately follows the target product quality values. Simulation results also show that the proposed model produced accurate predictions of kiln outputs that fell within the required constraints, while manipulating control variables within typical operational ranges.

Characterization of a high bandwidth readout chain for the CMS phase-2 pixel upgrade

The CMS collaboration is building a new inner tracking pixel detector for the High-Luminosity LHC. Each pixel readout chip will be controlled with a single serial input stream at 160 Mb/s and will send out data via four current mode logic (CML) 1.28 Gb/s outputs. The readout chips will be grouped in modules and connected with up to 1.6 meters long low-mass electrical links to Low-Power Gigabit Transceivers (lpGBT) and Versatile Link PLUS Transceiver (VTRx+) modules that send the data optically to off-detector electronics at 10 Gb/s. The characterization of these components and system tests of the readout chain are presented.

Multimodal Event Processing: A Neural-Symbolic Paradigm for the Internet of Multimedia Things

With the Internet of Multimedia Things (IoMT) becoming a reality, new approaches are needed to process real-time multimodal event streams. Existing approaches to event processing have limited consideration for the challenges of multimodal events, including the need for complex content extraction, and increased computational and memory costs. This article explores event processing as a basis for processing real-time IoMT data. This article introduces the multimodal event processing (MEP) paradigm, which provides a formal basis for native approaches to neural multimodal content analysis (i.e., computer vision, linguistics, and audio) with symbolic event processing rules to support real-time queries over multimodal data streams using the multimodal event processing language to express single, primitive multimodal, and complex multimodal event patterns. The content of multimodal streams is represented using multimodal event knowledge graphs to capture the semantic, spatial, and temporal content of the multimodal streams. The approach is implemented and evaluated within a MEP engine using single and multimodal queries achieving near real-time performance with a throughput of ~30 frames processed per second (fps) and subsecond latency of 0.075–0.30 s for video streams of 30 fps input rate. Support for high input stream rates (45 fps) is achieved through content-aware load-shedding techniques with a ~127X latency improvement resulting in only a minor decrease in accuracy.

High-Level Design Optimizations for Implementing Data Stream Sketch Frequency Estimators on FPGAs

This paper presents simple yet effective optimizations for implementing data stream frequency estimation sketch kernels using High-Level Synthesis (HLS). The paper addresses design issues common to sketches utilizing large portions of the embedded RAM resources in a Field Programmable Gate Array (FPGA). First, a solution based on Load-Store Queue (LSQ) architecture is proposed for resolving the memory dependencies associated with the hash tables in a frequency estimation sketch. Second, performance fine-tuning through high-level pragmas is explored to achieve the best possible throughput. Finally, a technique based on pre-processing the data stream in a small cache memory prior to updating the sketch is evaluated to reduce the dynamic power consumption. Using an Intel HLS compiler, a proposed optimized hardware version of the popular Count-Min sketch utilizing 80% of the embedded RAM in an Intel Arria 10 FPGA, achieved more than 3x the throughput of an unoptimized baseline implementation. Furthermore, the sketch update rate is significantly reduced when the input stream is skewed. This, in turn, minimizes the effect of high throughput on dynamic power consumption. Compared to FPGA sketches in the published literature, the presented sketch is the most well-rounded sketch in terms of features and versatility. In terms of throughput, the presented sketch is on a par with the fastest sketches fine-tuned at the Register Transfer Level (RTL).

A Graph Skeleton Transformer Network for Action Recognition

Skeleton-based action recognition is a research hotspot in the field of computer vision. Currently, the mainstream method is based on Graph Convolutional Networks (GCNs). Although there are many advantages of GCNs, GCNs mainly rely on graph topologies to draw dependencies between the joints, which are limited in capturing long-distance dependencies. Meanwhile, Transformer-based methods have been applied to skeleton-based action recognition because they effectively capture long-distance dependencies. However, existing Transformer-based methods lose the inherent connection information of human skeleton joints because they do not yet focus on initial graph structure information. This paper aims to improve the accuracy of skeleton-based action recognition. Therefore, a Graph Skeleton Transformer network (GSTN) for action recognition is proposed, which is based on Transformer architecture to extract global features, while using undirected graph information represented by the symmetric matrix to extract local features. Two encodings are utilized in feature processing to improve joints’ semantic and centrality features. In the process of multi-stream fusion strategies, a grid-search-based method is used to assign weights to each input stream to optimize the fusion results. We tested our method using three action recognition datasets: NTU RGB+D 60, NTU RGB+D 120, and NW-UCLA. The experimental results show that our model’s accuracy is comparable to state-of-the-art approaches.

Depth Estimation for Integral Imaging Microscopy Using a 3D–2D CNN with a Weighted Median Filter

This study proposes a robust depth map framework based on a convolutional neural network (CNN) to calculate disparities using multi-direction epipolar plane images (EPIs). A combination of three-dimensional (3D) and two-dimensional (2D) CNN-based deep learning networks is used to extract the features from each input stream separately. The 3D convolutional blocks are adapted according to the disparity of different directions of epipolar images, and 2D-CNNs are employed to minimize data loss. Finally, the multi-stream networks are merged to restore the depth information. A fully convolutional approach is scalable, which can handle any size of input and is less prone to overfitting. However, there is some noise in the direction of the edge. A weighted median filtering (WMF) is used to acquire the boundary information and improve the accuracy of the results to overcome this issue. Experimental results indicate that the suggested deep learning network architecture outperforms other architectures in terms of depth estimation accuracy.