3D Streaming Research Articles

Automated Laryngeal Invasion Detector of Boluses in Videofluoroscopic Swallowing Study Videos Using Action Recognition-Based Networks.

We aimed to develop an automated detector that determines laryngeal invasion during swallowing. Laryngeal invasion, which causes significant clinical problems, is defined as two or more points on the penetration-aspiration scale (PAS). We applied two three-dimensional (3D) stream networks for action recognition in videofluoroscopic swallowing study (VFSS) videos. To detect laryngeal invasion (PAS 2 or higher scores) in VFSS videos, we employed two 3D stream networks for action recognition. To establish the robustness of our model, we compared its performance with those of various current image classification-based architectures. The proposed model achieved an accuracy of 92.10%. Precision, recall, and F1 scores for detecting laryngeal invasion (≥PAS 2) in VFSS videos were 0.9470 each. The accuracy of our model in identifying laryngeal invasion surpassed that of other updated image classification models (60.58% for ResNet101, 60.19% for Swin-Transformer, 63.33% for EfficientNet-B2, and 31.17% for HRNet-W32). Our model is the first automated detector of laryngeal invasion in VFSS videos based on video action recognition networks. Considering its high and balanced performance, it may serve as an effective screening tool before clinicians review VFSS videos, ultimately reducing the burden on clinicians.

N-DEPTH: Neural Depth Encoding for Compression-Resilient 3D Video Streaming

Recent advancements in 3D data capture have enabled the real-time acquisition of high-resolution 3D range data, even in mobile devices. However, this type of high bit-depth data remains difficult to efficiently transmit over a standard broadband connection. The most successful techniques for tackling this data problem thus far have been image-based depth encoding schemes that leverage modern image and video codecs. To our knowledge, no published work has directly optimized the end-to-end losses of a depth encoding scheme sandwiched around a lossy image compression codec. We present N-DEPTH, a compression-resilient neural depth encoding method that leverages deep learning to efficiently encode depth maps into 24-bit RGB representations that minimize end-to-end depth reconstruction errors when compressed with JPEG. N-DEPTH’s learned robustness to lossy compression expands to video codecs as well. Compared to an existing state-of-the-art encoding method, N-DEPTH achieves smaller file sizes and lower errors across a large range of compression qualities, in both image (JPEG) and video (H.264) formats. For example, reconstructions from N-DEPTH encodings stored with JPEG had dramatically lower error while still offering 29.8%-smaller file sizes. When H.264 video was used to target a 10 Mbps bit rate, N-DEPTH reconstructions had 85.1%-lower root mean square error (RMSE) and 15.3%-lower mean absolute error (MAE). Overall, our method offers an efficient and robust solution for emerging 3D streaming and 3D telepresence applications, enabling high-quality 3D depth data storage and transmission.

Cellular Energy Cycle Mediates an Advection-Like Forward Cell Flow to Support Collective Invasion.

Collective cell migration is a model for nonequilibrium biological dynamics, which is important for morphogenesis, pattern formation, and cancer metastasis. The current understanding of cellular collective dynamics is based primarily on cells moving within a 2D epithelial monolayer. However, solid tumors often invade surrounding tissues in the form of a stream-like 3D structure, and how biophysical cues are integrated at the cellular level to give rise to this collective streaming remains unclear. Here, it is shown that cell cycle-mediated bioenergetics drive a forward advective flow of cells and energy to the front to support 3D collective invasion. The cell division cycle mediates a corresponding energy cycle such that cellular adenosine triphosphate (ATP) energy peaks just before division. A reaction-advection-diffusion (RAD) type model coupled with experimental measurements further indicates that most cells enter an active division cycle at rear positions during 3D streaming. Once the cells progress to a later stage toward division, the high intracellular energy allows them to preferentially stream toward the tip and become leader cells. This energy-driven cellular flow may be a fundamental characteristic of 3D collective dynamics based on thermodynamic principles important for not only cancer invasion but also tissue morphogenesis.

Microparticle focusing and micromixing with two-dimensional acoustic waves

Acoustofluidics has recently been popularized as a crucial element of lab-on-a-chip (LoC) platforms to efficiently manipulate microparticles and continuous matter alike. In this study, a three-dimensional (3D) numerical model is proposed to simulate the focusing of polystyrene microparticles with three diameters and micromixing of dilute species using two orthogonally oriented standing waves, contrasting them with one-dimensional (1D) waves. The limiting velocity method is modified to explore the 3D acoustic streaming in a symmetric microchannel. In contrast to 1D standing acoustic waves, the simultaneous excitation of two orthogonal waves generates an acoustic streaming velocity field that does not counteract the radiation force. The obtained results show that the focusing efficiency of 5-μm particles reaches 97% with two dimensional (2D) standing acoustic waves, which was unachievable using 1D waves. Moreover, by reducing the flow rate to 1 μL min−1, the focusing of critical microparticle diameter peaked at 94%, indicating an approximately 9% improvement over a flow rate of 2.5 μL min−1. Increasing the viscosity of the background fluid resulted in 16% better 2D focusing with a single vortex compared to other cases, and higher amplitudes did not change focusing efficiency with a single vortex, while reducing efficiency in other cases. Finally, using 2D acoustic waves remarkably improved the mixing efficiency of dilute species, underscoring the advantage of 2D acoustic waves over their 1D counterpart. The proposed numerical model can play a meaningful role in cutting fabrication costs of next-generation LoC devices by identifying the most crucial parameters influencing acoustofluidic matter transport.

Video-Assisted Telescope Operating Monitor 3D System in Microsurgical Varicocelectomy: A Preliminary Report.

Video-assisted telescope operating monitor (VITOM®) with 3D Visualization technology was developed and has been used with favorable results by several surgical specialties. Our study aims to be a preliminary report for initial experience using the VITOM® 3D system for microsurgical varicocelectomy on varicocele patients. We performed 35 microsurgical varicocelectomy procedures using the VITOM® 3D system on varying types and grades of varicoceles. The surgeon had the option of using either a 2.5 or 3.5 magnifying loupe in addition to the exoscope for each operation evaluated. The exoscope is a standalone camera head with an integrated 3D telescope and remote control with zoom and focus functions. It is connected to the 3D monitor via a mechanical holding arm. During the procedure, surgeons, assistants, and observers were able to view the 3D high-definition stream displayed on a 26-inch 3D monitor at a convenient viewing angle and distance. The varicocele ligation was performed using a Carl Zeiss Meditec AG microscope. There were 35 patients with varicocele aged 31.51years old on average, which were included in this report. Most patients had grade 3 bilateral varicocele (n = 13, 37.1%). All procedures were performed without any intraoperative complications. After the procedures, only a few patients suffered from postoperative complications. Three patients suffered scrotal edema (8.6%), while another had hydrocele (2.9%). The postoperative pain results were also very minimal from .89 1day after the operation to .26 3days after the operation. The VITOM® 3D system showed promise in microsurgical varicocelectomy.

NUMERICAL THREE-DIMENSIONAL MODEL OF ULTRASONIC COAGULATION OF AEROSOL PARTICLES IN VORTEX ACOUSTIC STREAMING

Separation of highly dispersed systems with huge liquid-gas or liquid-solid interfaces is relevant for practical tasks of gas purification from the most highly dispersed and difficult-to-detect dispersed fraction PM2.5, and separation of nanoparticles (including their small agglomerates) in fine chemical technology processes. One of the most effective ways to separate highly dispersed systems with a large interface surface is to combine each of the closed subsurfaces (surfaces of individual dispersed particles) under the influence of hydrodynamic effects in the gas phase, arising both near the interface surfaces and at a considerable distance from them, due to the superposition of ultrasonic vibrations. Since the efficiency of ultrasonic coagulation decreases with a large distance between closed subsurfaces from each other in PM2.5 aerosol and the small size of these surfaces, it is necessary to create conditions for the emergence of new nonlinear effects that contribute to the local compaction of the dispersed fraction. In a resonant and significantly inhomogeneous ultrasonic field (with a scale of inhomogeneity on the order of the wavelength), vortex acoustic flows arise, which, due to inertial forces, locally compact the dispersed phase in the form of an increase in the concentration of aerosol particles. A numerical model of ultrasonic coagulation of PM2.5 aerosol particles in three-dimensional (3D) vortex acoustic streaming is proposed in this paper. The model is designed to identify the possibility of increasing the efficiency of ultrasonic coagulation in 3D streaming by virtue of the following mechanisms: (1) local increase in concentration caused by the inertial transfer of particles to the periphery of 3D vortices in the gas phase; (2) increase in the frequency of particle collisions due to 3D turbulent disturbances in ultrasonic fields; and (3) increase in productivity and ensuring uninterrupted implementation of the process in a flow mode owing to transfer of particles between the streamlines of the main vortices initiated by ultrasonic vibrations. The listed mechanisms for increasing the efficiency of coagulation in 3D streaming are taken into consideration by introducing two stream functions, considering turbulent chaotic disturbances of the flow resulting in dispersion of particle velocities. It was possible to establish based on numerical analysis of the model using the example of PM2.5 that laminar vortex flows begin to influence at sound pressure level from 160-165 dB, and turbulent disturbances make an additional contribution in the range of sound pressure levels from 140-170 dB. At the same time, as a result of 3D turbulent disturbances, the efficiency of coagulation reaches almost 100% at a sound pressure level 5 dB lower than with laminar flows (sound pressure amplitude, 3 times lower).

Preview the Products on Display at Display Week 2023

Preview the Products on Display at Display Week 2023

Interpolation of 3D data streams with C2 PH quintic splines

The construction of smooth spatial paths with Pythagorean-hodograph (PH) quintic splines is proposed. To facilitate real-time computations, an efficient local data stream interpolation algorithm is introduced to successively construct each spline segment as a quintic PH biarc interpolating second- and first-order Hermite data at the initial and final end-point, respectively. A C2 smooth connection between successive spline segments is obtained by taking the locally required second-order derivative information from the previous segment. Consequently, the data stream spline interpolant is globally C2 continuous and can be constructed for arbitrary C1 Hermite data configurations. A simple and effective selection of the free parameters that arise in the local interpolation problem is proposed. The developed theoretical analysis proves its fourth approximation order while a selection of numerical examples confirms the same accuracy for the spline extension of the scheme. In addition, the performances of the method are also validated by considering its application to point stream interpolation with automatically generated first-order derivative information.

Motion Estimation Role in the Context of 3D Video

The 3D end-to-end video system (i.e., 3D acquisition, processing, streaming, error concealment, virtual/augmented reality handling, content retrieval, rendering, and displaying) still needs improvements. This paper scrutinizes the motion compensation/motion estimation (MCME) impact in the 3D video (3DV) from the end-to-end users' point of view deeply. The concepts of motion vectors (MVs) and disparities are very close, and they help to ameliorate all the stages of the end-to-end 3DV system. The high-efficiency video coding (HEVC) video codec standard is taken into consideration to evaluate the emergent trend towards computational treatment throughout the cloud whenever possible. The tight bond between movement and depth affects 3D information recovery from these cues and optimizes the performance of algorithms and standards from several parts of the 3D system. Still, 3DV lacks support for engaging interactive 3DV services. Better bit allocation strategies also ameliorate all 3D pipeline stages while being attentive to cloud-based deployments for 3D streaming.

Investigation of effective parameters on streaming-induced acoustophoretic particle manipulation in a microchannel via three-dimensional numerical simulation

Particle manipulation using ultrasonic standing waves has gained increased attention in recent years as it is efficient and noninvasive. In order to predict the effects of acoustic streaming on the concentration of particles in the actual microchannel geometry, this paper presents a 3D numerical study on the transient motion of microparticles suspended in a liquid-filled microchannel, considering the mixed standing and traveling waves. The motion was generated by the acoustic radiation force and acoustic streaming-induced drag force arising from an imposed bulk acoustic wave and the hydrodynamic drag. The acoustic streaming patterns in the 3D microchannel were investigated using the limiting velocity method. In addition, the effects of the 3D streaming pattern in an acoustofluidic device on the acoustophoretic motion of microparticles were evaluated. The concentration of polystyrene particles was simulated for many particles with diameters of 0.5, 2, and 5 μm released from random initial locations. The obtained results indicate a balance between the flow rate and the particle diameter to achieve the highest concentration percentage. Increasing the height increased the concentration of large 5-μm-diameter particles to more than 80%. By doubling the length of the piezoelectrically actuated region, the concentration of 2-μm particles improved by approximately 20%. Finally, increasing the viscosity of the fluid by using a 50% glycerol-in-water mixture resulted in a greater effect of acoustic streaming. This study can provide helpful guidance for optimizing the design of acoustofluidic devices to enhance experiments.

Survey on some key technologies of virtual tourism system based on Web3D

<p>Some key technologies on how to build large-scale virtual tourism systssems comprehensively on Web browsers and mobiles were analyzed and the current R&D status on Web3D virtual tourism was surveyed insightfully. Then, some methods were summarized, including 3D trees or plants modeling, 3D architectural modeling, 3D Virtual Human behavior modeling, virtual agents path planning, collision detection and progressive transmission strategy suitable for developing large scale Web3D tourism scenarios. Also, some bottleneck problems of Web3D virtual tourism system were investigated. At the same time, the lightweight 3D engine, the lightweight 3D modeling, the lightweight 3D streaming and P2P based progressive transmission of huge Web3D tourism contents would become much helpful to breakthrough those bottlenecks of Web3D tourism systems were pointed out. In addition, all kinds of Web3D engines in terms of lightweight, realism and efficiency that would be a good reference for developers to choose during various applications were compared comprehensively. Finally, the prospect of future investigation of Web3D tourism system is presented, which will be going on in terms of four characteristics lightweight, high-speed, realism, beauty. </p>

Mol* Viewer: modern web app for 3D visualization and analysis of large biomolecular structures.

Large biomolecular structures are being determined experimentally on a daily basis using established techniques such as crystallography and electron microscopy. In addition, emerging integrative or hybrid methods (I/HM) are producing structural models of huge macromolecular machines and assemblies, sometimes containing 100s of millions of non-hydrogen atoms. The performance requirements for visualization and analysis tools delivering these data are increasing rapidly. Significant progress in developing online, web-native three-dimensional (3D) visualization tools was previously accomplished with the introduction of the LiteMol suite and NGL Viewers. Thereafter, Mol* development was jointly initiated by PDBe and RCSB PDB to combine and build on the strengths of LiteMol (developed by PDBe) and NGL (developed by RCSB PDB). The web-native Mol* Viewer enables 3D visualization and streaming of macromolecular coordinate and experimental data, together with capabilities for displaying structure quality, functional, or biological context annotations. High-performance graphics and data management allows users to simultaneously visualise up to hundreds of (superimposed) protein structures, stream molecular dynamics simulation trajectories, render cell-level models, or display huge I/HM structures. It is the primary 3D structure viewer used by PDBe and RCSB PDB. It can be easily integrated into third-party services. Mol* Viewer is open source and freely available at https://molstar.org/.

54‐6: Holographic 3D Telepresence System with Light Field 3D Displays and Depth Cameras over a LAN

A 3D teleconferencing system is demonstrated, based on the Looking Glass light field displays and Microsoft Azure Kinect depth cameras over a gigabit Local Area Network (LAN). Full‐duplex (two‐way) real‐time 3D streaming of a full colour point cloud video with audio is achieved at 30 fps at 173 Mbps.

Spatio-Temporal Learning for Video Deblurring based on Two-Stream Generative Adversarial Network

Video-deblurring has achieved excellent results by using deep learning approaches. How to capture the dynamic spatio-temporal information in the videos is crucial on deblurring. In this paper, we propose a two-stream DeblurGAN which combines a 3D stream with a 2D stream to deblur. The 3D convolution provides spatial and temporal invariance to restore the foreground of frames, while the 2D convolution is sufficient to deal with spatial features, given a relatively consistant background. Thus, our model takes advantage of the great processing power of the 3D stream to handle the foreground which usually contains more dynamical motion blur, and the advantage of the simplicity of the 2D stream to handle the mostly consistent background. We have the full advantage of combining both the 3D convolution and the 2D convolution. Then we take the two-stream model as the generator and adopt the adversarial learning. We test our model on the VideoDeblurring and GOPRO datasets, and compare with other methods which we have listed. Our method outperforms others in the Peak Signal-to-Noise Ratio (PSNR), especially shows a good performance handling the foreground with obvious motion blur.

Orthogonalization-Guided Feature Fusion Network for Multimodal 2D+3D Facial Expression Recognition

As 2D and 3D data present different views of the same face, the features extracted from them can be both complementary and redundant. In this paper, we present a novel and efficient orthogonalization-guided feature fusion network, namely OGF <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^2$</tex-math></inline-formula> Net, to fuse the features extracted from 2D and 3D faces for facial expression recognition. While 2D texture maps are fed into a 2D feature extraction pipeline (FE2DNet), the attribute maps generated from 3D data are concatenated as input of the 3D feature extraction pipeline (FE3DNet). The two networks are separately trained at the first stage and frozen in the second stage for late feature fusion, which can well address the unavailability of a large number of 3D+2D face pairs. To reduce the redundancies among features extracted from 2D and 3D streams, we design an orthogonal loss-guided feature fusion network to orthogonalize the features before fusing them. Experimental results show that the proposed method significantly outperforms the state-of-the-art algorithms on both the BU-3DFE and Bosphorus databases. While accuracies as high as 89.05% (P1 protocol) and 89.07% (P2 protocol) are achieved on the BU-3DFE database, an accuracy of 89.28% is achieved on the Bosphorus database. The complexity analysis also suggests that our approach achieves a higher processing speed while simultaneously requiring lower memory costs.

Simulation of Thermo Diffusion on Three-Dimensional Flow of a Micropolar Liquid on an Inclined Convective Surface with Nonlinear Stretching Sheet

The present research explores the features of thermal and solutal transport of a 3D micropolar liquid stream on an elongated convectively heated inclined sheet taking Soret effect. Mathematical modelling is designed with the aid of suitable scaling analysis on the governing PDEs conceiving the small magnetic Reynolds number. The resultant set of coupled nonlinear ODEs are derived with MATLAB to obtain computational solutions. Impression of the emerged flow parameters on the three boundary layers is graphically traced and deliberated. The parameters of magnetic field and stretching ratio and power law index diminished frictional drag. Hike in rate of thermal diffusion is prevailed with stronger surface heat convective and Prandtl numbers. Outcomes are collated with the data available in the literature and found to agree very closely as a limiting case.

Spatio-Temporal Representation of an Electoencephalogram for Emotion Recognition Using a Three-Dimensional Convolutional Neural Network.

Emotion recognition plays an important role in the field of human–computer interaction (HCI). An electroencephalogram (EEG) is widely used to estimate human emotion owing to its convenience and mobility. Deep neural network (DNN) approaches using an EEG for emotion recognition have recently shown remarkable improvement in terms of their recognition accuracy. However, most studies in this field still require a separate process for extracting handcrafted features despite the ability of a DNN to extract meaningful features by itself. In this paper, we propose a novel method for recognizing an emotion based on the use of three-dimensional convolutional neural networks (3D CNNs), with an efficient representation of the spatio-temporal representations of EEG signals. First, we spatially reconstruct raw EEG signals represented as stacks of one-dimensional (1D) time series data to two-dimensional (2D) EEG frames according to the original electrode position. We then represent a 3D EEG stream by concatenating the 2D EEG frames to the time axis. These 3D reconstructions of the raw EEG signals can be efficiently combined with 3D CNNs, which have shown a remarkable feature representation from spatio-temporal data. Herein, we demonstrate the accuracy of the emotional classification of the proposed method through extensive experiments on the DEAP (a Dataset for Emotion Analysis using EEG, Physiological, and video signals) dataset. Experimental results show that the proposed method achieves a classification accuracy of 99.11%, 99.74%, and 99.73% in the binary classification of valence and arousal, and, in four-class classification, respectively. We investigate the spatio-temporal effectiveness of the proposed method by comparing it to several types of input methods with 2D/3D CNN. We then verify the best performing shape of both the kernel and input data experimentally. We verify that an efficient representation of an EEG and a network that fully takes advantage of the data characteristics can outperform methods that apply handcrafted features.

Lidar Data Reduction for Unmanned Systems Navigation in Urban Canyon

This paper introduces a novel protocol for managing low altitude 3D aeronautical chart data to address the unique navigational challenges and collision risks associated with populated urban environments. Based on the Open Geospatial Consortium (OGC) 3D Tiles standard for geospatial data delivery, the proposed extension, called 3D Tiles Nav., uses a navigation-centric packet structure which automatically decomposes the navigable regions of space into hyperlocal navigation cells and encodes environmental surfaces that are potentially visible from each cell. The developed method is sensor agnostic and provides the ability to quickly and conservatively encode visibility directly from a region by enabling an expanded approach to viewshed analysis. In this approach, the navigation cells themselves are used to represent the intrinsic positional uncertainty often needed for navigation. Furthermore, we present in detail this new data format and its unique features as well as a candidate framework illustrating how an Unmanned Traffic Management (UTM) system could support trajectory-based operations and performance-based navigation in the urban canyon. Our results, experiments, and simulations conclude that this data reorganization enables 3D map streaming using less bandwidth and efficient 3D map-matching systems with limited on-board compute, storage, and sensor resources.

A Cloud Based Disaster Management System

The combination of wireless sensor networks (WSNs) and 3D virtual environments opens a new paradigm for their use in natural disaster management applications. It is important to have a realistic virtual environment based on datasets received from WSNs to prepare a backup rescue scenario with an acceptable response time. This paper describes a complete cloud-based system that collects data from wireless sensor nodes deployed in real environments and then builds a 3D environment in near real-time to reflect the incident detected by sensors (fire, gas leaking, etc.). The system’s purpose is to be used as a training environment for a rescue team to develop various rescue plans before they are applied in real emergency situations. The proposed cloud architecture combines 3D data streaming and sensor data collection to build an efficient network infrastructure that meets the strict network latency requirements for 3D mobile disaster applications. As compared to other existing systems, the proposed system is truly complete. First, it collects data from sensor nodes and then transfers it using an enhanced Routing Protocol for Low-Power and Lossy Networks (RLP). A 3D modular visualizer with a dynamic game engine was also developed in the cloud for near-real time 3D rendering. This is an advantage for highly-complex rendering algorithms and less powerful devices. An Extensible Markup Language (XML) atomic action concept was used to inject 3D scene modifications into the game engine without stopping or restarting the engine. Finally, a multi-objective multiple traveling salesman problem (AHP-MTSP) algorithm is proposed to generate an efficient rescue plan by assigning robots and multiple unmanned aerial vehicles to disaster target locations, while minimizing a set of predefined objectives that depend on the situation. The results demonstrate that immediate feedback obtained from the reconstructed 3D environment can help to investigate what–if scenarios, allowing for the preparation of effective rescue plans with an appropriate management effort.

Stereo Vision Based Robot for Remote Monitoring with VR Support

The machine vision systems have been playing a significant role in visual monitoring systems. With the help of stereovision and machine learning, it will be able to mimic human-like visual system and behaviour towards the environment. In this paper, we present a stereo vision based 3-DOF robot which will be used to monitor places from remote using cloud server and internet devices. The 3-DOF robot will transmit human-like head movements, i.e., yaw, pitch, roll and produce 3D stereoscopic video and stream it in Real-time. This video stream is sent to the user through any generic internet devices with VR box support, i.e., smartphones giving the user a First-person real-time 3D experience and transfers the head motion of the user to the robot also in Real-time. The robot will also be able to track moving objects and faces as a target using deep neural networks which enables it to be a standalone monitoring robot. The user will be able to choose specific subjects to monitor in a space. The stereovision enables us to track the depth information of different objects detected and will be used to track human interest objects with its distances and sent to the cloud. A full working prototype is developed which showcases the capabilities of a monitoring system based on stereo vision, robotics, and machine learning.