Image Stream Research Articles

X-ray fluoroscopy guided localization and steering of miniature robots using virtual reality enhancement

In developing medical interventions using untethered milli- and microrobots, ensuring safety and effectiveness relies on robust methods for real-time robot detection, tracking, and precise localization within the body. The inherent non-transparency of human tissues significantly challenges these efforts, as traditional imaging systems like fluoroscopy often lack crucial anatomical details, potentially compromising intervention safety and efficacy. To address this technological gap, in this study, we build a virtual reality environment housing an exact digital replica (digital twin) of the operational workspace and a robot avatar. We synchronize the virtual and real workspaces and continuously send the robot position data derived from the image stream into the digital twin with short average delay time around 20–25 ms. This allows the operator to steer the robot by tracking its avatar within the digital twin with near real-time temporal resolution. We demonstrate the feasibility of this approach with millirobots steered in confined phantoms. Our concept demonstration herein can pave the way for not only improved procedural safety by complementing fluoroscopic guidance with virtual reality enhancement, but also provides a platform for incorporating various additional real-time derivative data, e.g., instantaneous robot velocity, intraoperative physiological data obtained from the patient, e.g., blood flow rate, and pre-operative physical simulation models, e.g., periodic body motions, to further refine robot control capacity.

Autonomous Cargo Grabbing System for Drones Using Cameras and ROS

Abstract. With the rapid development of low-altitude economy, the functional requirements for logistics UAVs are constantly improving. In this paper, an unmanned aerial vehicle system (UAV) is proposed to realize autonomous grasping of goods. First of all, a new type of grasping device is proposed, which can be easily mounted on the UAV. The camera image stream mounted on the UAV was obtained through camera calibration and corrected. Subsequently, the object recognition algorithm was used to process the image frames, successfully identify the goods to be captured, and calculate the position and direction relationship between the UAV and the target cargo. Next, the speed of the drone is adjusted by the PID controller to achieve a stable landing above the target cargo. Then, the drone is equipped with a robotic claw innovatively designed by our team to complete the gripping of the goods. Subsequently, the drone lifted off and began to deliver cargo. Throughout the grabbing process, a laptop equipped with Ubuntu and ROS systems communicates with the drone via Wi-Fi. The feasibility of the whole system was proved by experiments.

Emotional stimuli boost incidental learning through predictive processing.

Extracting regularities and probabilities from the environment is a fundamental and critical ability in an ever-changing surrounding. Previous findings showed that people are highly efficient in learning these regularities and that emotional stimuli are better learned than neutral ones. Yet, the generality and the underlying mechanism of this benefit are poorly understood. Here, participants viewed a stream of images with negative and neutral valence. Unbeknownst, the items recurred in regularity as triplets. Then, to assess learning, a surprised familiarity test was conducted. The results of Experiment 1, using two sets of stimuli, found better statistical learning for negative triplets than for neutral triplets. Experiment 2 revealed similar benefits even when only a single negative item was in the triplet at the second or third position, suggesting the advantage is not cumulative. We speculated that the predictability of the negative items is driving the effect. Consequently, Experiment 3 confirmed that the memory for neutral items preceding negative items was better than for neutral items preceding neutral items. Together, these findings provide novel insights into the mechanism of how the learning of incidental temporal associations is influenced by negative stimuli and the role of predictability in the negative valence benefit.

Tele-education in point-of-care ultrasound training

BackgroundTraditionally, ultrasound skills have been taught through a one-on-one approach, where instructors physically guide learners’ hands at the bedside or in the workshop. However, this method is frequently challenged by scheduling and cost limitations. Our objective was to create a tele-education model for point-of-care ultrasound training and evaluate its effectiveness and its impact on learners’ perceived workload compared to conventional education and self-directed learning methods.MethodsWe conducted a 3-arm randomized trial, comparing tele-education (TE), conventional education (CE) and self-directed learning (SL) methods. All subjects underwent online didactic lectures prior to a hands-on ultrasound workshop. The TE group utilized an ultrasound machine equipped with a speakerphone, a webcam for direct visualization of learner’s hand maneuvers, and an analog-to-video converter for the real-time streaming of ultrasound images. This configuration enabled remote instructors to provide immediate verbal feedback to learners. In contrast, the CE group received in-person coaching, while the SL group had no instructors present. Following the coaching session, subjects completed a scenario-based skill test and a survey on the National Aeronautics and Space Administration task load index (NASA-TLX) to measure their ultrasound competency and perceived workload, respectively.ResultsTwenty-seven ultrasound novices were randomly allocated into 3 groups. The median skill test score of TE, CE, and SL was 22 [interquartile range (IQR): 18–28], 24 [IQR: 21–31], and 16 [IQR: 15–18], respectively (p < 0.01). Pairwise comparisons of median test scores of 3 groups demonstrated a statistical significance in comparisons of TE vs. SL (22 vs. 16, p = 0.01) and CE vs. SL (24 vs. 16, p < 0.01), but not in TE vs. CE (22 vs. 24, p = 0.56). There was no statistical significance observed in the median NASA-TLX scores among the 3 groups; 54 [IQR:47–61] in TE, 57 [IQR:22–64] in CE, and 66 [IQR: 66–72] in SL (p = 0.05).ConclusionsOur tele-education model was more effective than self-directed learning. There was no statistically significant difference in effectiveness between the tele-education and the conventional education groups. Importantly, tele-education did not impose a significantly higher workload on learners compared to conventional education or self-directed learning. Tele-education has a substantial potential as an alternative to conventional ultrasound training.

A Computationally Efficient Neuronal Model for Collision Detection with Contrast Polarity-Specific Feed-Forward Inhibition

Animals utilize their well-evolved dynamic vision systems to perceive and evade collision threats. Driven by biological research, bio-inspired models based on lobula giant movement detectors (LGMDs) address certain gaps in constructing artificial collision-detecting vision systems with robust selectivity, offering reliable, low-cost, and miniaturized collision sensors across various scenes. Recent progress in neuroscience has revealed the energetic advantages of dendritic arrangements presynaptic to the LGMDs, which receive contrast polarity-specific signals on separate dendritic fields. Specifically, feed-forward inhibitory inputs arise from parallel ON/OFF pathways interacting with excitation. However, none of the previous research has investigated the evolution of a computational LGMD model with feed-forward inhibition (FFI) separated by opposite polarity. This study fills this vacancy by presenting an optimized neuronal model where FFI is divided into ON/OFF channels, each with distinct synaptic connections. To align with the energy efficiency of biological systems, we introduce an activation function associated with neural computation of FFI and interactions between local excitation and lateral inhibition within ON/OFF channels, ignoring non-active signal processing. This approach significantly improves the time efficiency of the LGMD model, focusing only on substantial luminance changes in image streams. The proposed neuronal model not only accelerates visual processing in relatively stationary scenes but also maintains robust selectivity to ON/OFF-contrast looming stimuli. Additionally, it can suppress translational motion to a moderate extent. Comparative testing with state-of-the-art based on ON/OFF channels was conducted systematically using a range of visual stimuli, including indoor structured and complex outdoor scenes. The results demonstrated significant time savings in silico while retaining original collision selectivity. Furthermore, the optimized model was implemented in the embedded vision system of a micro-mobile robot, achieving the highest success ratio of collision avoidance at 97.51% while nearly halving the processing time compared with previous models. This highlights a robust and parsimonious collision-sensing mode that effectively addresses real-world challenges.

Gesture Recognition of Filipino Sign Language Using Convolutional and Long Short-Term Memory Deep Neural Networks

In response to the recent formalization of Filipino Sign Language (FSL) and the lack of comprehensive studies, this paper introduces a real-time FSL gesture recognition system. Unlike existing systems, which are often limited to static signs and asynchronous recognition, it offers dynamic gesture capturing and recognition of 10 common expressions and five transactional inquiries. To this end, the system sequentially employs cropping, contrast adjustment, grayscale conversion, resizing, and normalization of input image streams. These steps serve to extract the region of interest, reduce the computational load, ensure uniform input size, and maintain consistent pixel value distribution. Subsequently, a Convolutional Neural Network and Long-Short Term Memory (CNN-LSTM) model was employed to recognize nuances of real-time FSL gestures. The results demonstrate the superiority of the proposed technique over existing FSL recognition systems, achieving an impressive average accuracy, recall, and precision rate of 98%, marking an 11.3% improvement in accuracy. Furthermore, this article also explores lightweight conversion methods, including post-quantization and quantization-aware training, to facilitate the deployment of the model on resource-constrained platforms. The lightweight models show a significant reduction in model size and memory utilization with respect to the base model when executed in a Raspberry Pi minicomputer. Lastly, the lightweight model trained with the quantization-aware technique (99%) outperforms the post-quantization approach (97%), showing a notable 2% improvement in accuracy.

Separation of ventilation and perfusion of electrical impedance tomography image streams using multi-dimensional ensemble empirical mode decomposition

Objective. In the future, thoracic electrical impedance tomography (EIT) monitoring may include continuous and simultaneous tracking of both breathing and heart activity. However, an effective way to decompose an EIT image stream into physiological processes as ventilation-related and cardiac-related signals is missing.Approach. This study analyses the potential of Multi-dimensional Ensemble Empirical Mode Decomposition by application of the Complete Ensemble Empirical Mode Decomposition with Adaptive Noise and a novel frequency-based combination criterion for detrending, denoising and source separation of EIT image streams, collected from nine healthy male test subjects with similar age and constitution.Main results. In this paper, a novel approach to estimate the lung, the heart and the perfused regions of an EIT image is proposed, which is based on the Root Mean Square Error between the index of maximal respiratory and cardiac variation to their surroundings. The summation of the indexes of the respective regions reveals physiologically meaningful time signals, separated into the physiological bandwidths of ventilation and heart activity at rest. Moreover, the respective regions were compared with the relative thorax movement and photoplethysmogram (PPG) signal. In linear regression analysis and in the Bland–Altman plot, the beat-to-beat time course of both the ventilation-related signal and the cardiac-related signal showed a high similarity with the respective reference signal.Significance. Analysis of the data reveals a fair separation of ventilatory and cardiac activity realizing the aimed source separation, with optional detrending and denoising. For all performed analyses, a feasible correlation of 0.587 to 0.905 was found between the cardiac-related signal and the PPG signal.

Visual periodicity reveals distinct attentional signatures for face and non-face categories.

Observers can selectively deploy attention to regions of space, moments in time, specific visual features, individual objects, and even specific high-level categories-for example, when keeping an eye out for dogs while jogging. Here, we exploited visual periodicity to examine how category-based attention differentially modulates selective neural processing of face and non-face categories. We combined electroencephalography with a novel frequency-tagging paradigm capable of capturing selective neural responses for multiple visual categories contained within the same rapid image stream (faces/birds in Exp 1; houses/birds in Exp 2). We found that the pattern of attentional enhancement and suppression for face-selective processing is unique compared to other object categories: Where attending to non-face objects strongly enhances their selective neural signals during a later stage of processing (300-500ms), attentional enhancement of face-selective processing is both earlier and comparatively more modest. Moreover, only the selective neural response for faces appears to be actively suppressed by attending towards an alternate visual category. These results underscore the special status that faces hold within the human visual system, and highlight the utility of visual periodicity as a powerful tool for indexing selective neural processing of multiple visual categories contained within the same image sequence.

A time-correlated single photon counting SPAD array camera with a bespoke data-processing algorithm for lightsheet fluorescence lifetime imaging (FLIM) and FLIM videos

A wide-field microscope with epi-fluorescence and selective plane illumination was combined with a single-photon avalanche diode (SPAD) array camera to enable live-cell fluorescence lifetime imaging (FLIM) using time-correlated single-photon counting (TCSPC). The camera sensor comprised of 192×128\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ ext {192}\ imes \ ext {128}$$\\end{document} pixels, each integrating a single SPAD and a time-to-digital converter. Jointly, they produced a stream of single-photon images of photon arrival times with ≈38 ps\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\approx \ ext {38 ps}$$\\end{document} accuracy. The photon arrival times were subject to systematic delays and nonlinearities, which were corrected by a Monte-Carlo algorithm. The SPAD camera was then applied to FLIM where histogramming the resulting photon arrival times in each pixel resulted in decays compatible with common data processing pipelines for fluorescence lifetime analysis. The capabilities of the TCSPC camera-based FLIM microscope were demonstrated by imaging living unicellular photosynthetic algae and artificial lipid vesicles. Epi-fluorescence illumination enabled rapid fluorescence lifetime imaging of living cells and selective-plane illumination enabled 3-dimensional FLIM of stationary samples.

Various frameworks for integrating image and video streams for spatiotemporal information learning employing 2D–3D residual networks for human action recognition

Human action recognition has been identified as an important research topic in computer vision because it is an essential form of communication and interplay between computers and humans to assist computers in automatically recognizing human behaviors and accurately comprehending human intentions. Inspired by some keyframe extraction and multifeatured fusion research, this paper improved the accuracy of action recognition by utilizing keyframe features and fusing them with video features. In this article, we suggest a novel multi-stream approach architecture made up of two distinct models fused using different fusion techniques. The first model combines convolutional neural networks in two-dimensional (2D-CNN) with long-short term memory networks to glean long-term spatial and temporal features from video keyframe images for human action recognition. The second model is a three-dimensional convolutional neural network (3D-CNN) that gathers quick spatial–temporal features from video clips. Subsequently, two frameworks are put forth to explain how various fusion structures can improve the performance of action recognition. We investigate methods for video action recognition using early and late fusion. While the late-fusion framework addresses the decision fusion from the two models' choices for action recognition, the early-fusion framework examines the impact of early feature fusion of the two models for action recognition. The various fusion techniques investigate how much each spatial and temporal feature influences the recognition model's accuracy. The HMDB-51 and UCF-101 datasets are two important action recognition benchmarks used to evaluate our method. When applied to the HMDB-51 dataset and the UCF-101 dataset, the early-fusion strategy achieves an accuracy of 70.1 and 95.5%, respectively, while the late-fusion strategy achieves an accuracy of 77.7 and 97.5%, respectively.

Real-time analysis of large-scale neuronal imaging enables closed-loop investigation of neural dynamics.

Large-scale imaging of neuronal activities is crucial for understanding brain functions. However, it is challenging to analyze large-scale imaging data in real time, preventing closed-loop investigation of neural circuitry. Here we develop a real-time analysis system with a field programmable gate array-graphics processing unit design for an up to 500-megabyte-per-second image stream. Adapted to whole-brain imaging of awake larval zebrafish, the system timely extracts activity from up to 100,000 neurons and enables closed-loop perturbations of neural dynamics.

Implementing Privacy Homomorphism with Random Encoding and Computation Controlled by a Remote Secure Server

Remote IoT devices face significant security risks due to their inherent physical vulnerability. An adversarial actor with sufficient capability can monitor the devices or exfiltrate data to access sensitive information. Remotely deployed devices such as sensors need enhanced resilience against memory leakage if performing privileged tasks. To increase the security and trust of these devices we present a novel framework implementing a privacy homomorphism which creates sensor data directly in an encoded format. The sensor data is permuted at the time of creation in a manner which appears random to an observer. A separate secure server in communication with the device provides necessary information which allows the device to perform processing on the encoded data but does not allow decoding of the result. The device transmits the encoded results to the secure server which maintains the ability to interpret the results. In this paper we show how this framework works for an image sensor calculating differences between a stream of images, with initial results showing an overhead as low as only 266% in terms of throughput when compared to computing on standard unencoded numbers such as two’s complement. We further show 5,000x speedup over a recent homomorphic encryption ASIC.

Deep-Learning-Based Segmentation of Keyhole in In-Situ X-ray Imaging of Laser Powder Bed Fusion.

In laser powder bed fusion processes, keyholes are the gaseous cavities formed where laser interacts with metal, and their morphologies play an important role in defect formation and the final product quality. The in-situ X-ray imaging technique can monitor the keyhole dynamics from the side and capture keyhole shapes in the X-ray image stream. Keyhole shapes in X-ray images are then often labeled by humans for analysis, which increasingly involves attempting to correlate keyhole shapes with defects using machine learning. However, such labeling is tedious, time-consuming, error-prone, and cannot be scaled to large data sets. To use keyhole shapes more readily as the input to machine learning methods, an automatic tool to identify keyhole regions is desirable. In this paper, a deep-learning-based computer vision tool that can automatically segment keyhole shapes out of X-ray images is presented. The pipeline contains a filtering method and an implementation of the BASNet deep learning model to semantically segment the keyhole morphologies out of X-ray images. The presented tool shows promising average accuracy of 91.24% for keyhole area, and 92.81% for boundary shape, for a range of test dataset conditions in Al6061 (and one AliSi10Mg) alloys, with 300 training images/labels and 100 testing images for each trial. Prospective users may apply the presently trained tool or a retrained version following the approach used here to automatically label keyhole shapes in large image sets.

Multi-Log Grasping Using Reinforcement Learning and Virtual Visual Servoing

We explore multi-log grasping using reinforcement learning and virtual visual servoing for automated forwarding in a simulated environment. Automation of forest processes is a major challenge, and many techniques regarding robot control pose different challenges due to the unstructured and harsh outdoor environment. Grasping multiple logs involves various problems of dynamics and path planning, where understanding the interaction between the grapple, logs, terrain, and obstacles requires visual information. To address these challenges, we separate image segmentation from crane control and utilise a virtual camera to provide an image stream from reconstructed 3D data. We use Cartesian control to simplify domain transfer to real-world applications. Because log piles are static, visual servoing using a 3D reconstruction of the pile and its surroundings is equivalent to using real camera data until the point of grasping. This relaxes the limits on computational resources and time for the challenge of image segmentation, and allows for data collection in situations where the log piles are not occluded. The disadvantage is the lack of information during grasping. We demonstrate that this problem is manageable and present an agent that is 95% successful in picking one or several logs from challenging piles of 2–5 logs.

A Novel Efficient Dual-Gate Mixed Dilated Convolution Network for Multi-Scale Pedestrian Detection

With the increasing use of onboard high-speed computing systems, vehicle manufacturers are offering significant advanced features of driver assistance systems. Pedestrian detection is one of the major requirements of such systems, which commonly use cameras, radar, and ultrasonic sensors. Image recognition based on captured image streams is one of the powerful tools used for the detection of pedestrians, which exhibits similarities and distinguishing features compared to general object detection. Although pedestrian detection has advanced significantly along with deep learning, some issues still need to be addressed. Pedestrian detection is essential for several real-world applications and is an initial step in outdoor scene analysis. Typically, in a crowded situation, conventional detectors are unable to distinguish persons from each other successfully. This study presents a novel technique, based on the Dual Gate Mixed Dilated Convolution Network, to address this problem by adaptively filtering spatial areas where the patterns are still complicated and require further processing. The proposed technique manages obscured patterns while offering improved multiscale pedestrian recognition accuracy.

Validation of image stream hashing: A forensic method for content verification.

How do forensic examiners know if they have altered an image stream when converting a digital image from one codec or file container to another for analysis? Forensic standards and best practices recommend avoiding alteration or degradation of multimedia data during transcoding. An image stream hashing method was recently introduced to the forensic science community to answer the question above. This paper offers an initial validation study of image stream hashing method that may answer the question above. The first half of the study's experiments tested the image stream hashing method to measure fitness for use in forensic science while identifying errors and limitations. The study's second phase analyzed the systematic errors detected in initial tests to discover error causation. Causation analysis identified four method limitations subsequently used to develop proposed standard controls of method operations. The final study phase repeated the initial experiments used in the first phase while implementing the proposed standard controls of method operations. Initial test results indicated the method had significant error rates, limiting the effectiveness of the method to only three of the five file types used in the study. The final testing phase revealed that implementing proposed standard controls of method operations reduced the potential systematic errors to a negligible level when using the image stream hashing method for content verification. The validation study concluded that examiners could use the image stream hashing method for forensic science only by implementing error mitigation techniques that utilize the proposed standard controls of method operations.

Crafts in the Time of Coronavirus

Crafts in the Time of Coronavirus

Pendampingan Penerapan Keterampilan Menulis melalui Strategi Image Streaming pada Pembelajaran Bahasa Siswa Kelas VII SMPN 1 Sanrobone

Looking at the nature of the Image Streaming strategy, it is hoped that the Image Streaming strategy can be used as an effective and appropriate strategy when used in learning to write narratives in the form of short story texts. This service aims to provide continuous and continuous learning so that it is not just a matter of putting together a complete word but can be put into writing using the right strategy; improving students' skills by implementing appropriate learning strategies, namely the image streaming strategy in the learning process so that students can practice expressing their ideas in written form; find out the extent to which students' abilities have increased in writing skills to compose a complete sentence through the Image streaming strategy. In order to achieve this goal, the service team carried out planning by chronologically chronologically lining up its implementation starting from discussions, providing materials, providing assistance and implementing the Image Streaming strategy, as well as evaluating the results of student activities. Therefore, the final result of this community service is that it can provide motivation and useful information for students to be more active in the teaching and learning process in class.

IMMU-12. TUMOR EXPRESSION OF CD83 PROMOTES IMMUNE CLEARANCE OF GLIOMA

Abstract Glioblastoma (GBM) remains deadly with minimal therapeutic improvement over the past decades. A hallmark of GBM is an immunosuppressive microenvironment, which includes anergic and suppressive T-cells. Antigen presentation is a critical component to successful activation of anti-tumor T cell responses and are dysregulated in GBM. To better understand how GBM cells modulate antigen presenting processes, we examined expression of antigen presentation genes in our single cell RNA-sequencing (scRNA-seq) cohort consisting of 18 GBM samples from 12 different human patients. Unexpectedly, we found a rare population of tumor cells with high expression of CD83, a marker of activated antigen presenting cells (APCs) that stabilizes MHC-II and CD86 expression. In an immunocompetent GFP-labeled piggyBac in utero electroporation (pB-IUE) model of GBM, we validated the existence of dually labeled CD83+GFP+ tumor cells using FACS image streaming. In vivo overexpression of CD83 (CD83OE) in tumor cells prolonged survival compared to control and CD83 knockout (CD83KO) tumors. In vitro, tumor cell lines derived from CD83OE tumors increased expansion and activation of co-cultured CD8+ T-cells whereas CD83KO tumors had a large population of immunosuppressive CD8+ T-cells. ELISAs of media from tumor cell lines revealed that CD83OE tumor cells secrete higher levels of the T cell activating cytokine, IL-2 than controls and CD83KO. scRNA-seq of pB-IUE tumors confirmed that CD83OE have increased activating cytokines compared to CD83KO and control tumors, and that CD83KO tumors had greater expression of immunosuppressive T-regulatory profiles. Intriguingly, we found that despite equal proliferative capacity of tumor cells both in vitro and in vivo conditions, CD83OE tumors were smaller, suggesting that CD83 expression by tumor cells stabilizes CD8+ T-cells to promote immune clearance of tumors. Collectively, our data suggests that enhancing tumor cell expression of antigen presenting markers such as CD83 may encourage immune-mediated clearance of GBM.

Hierarchical AI enables global interpretation of culture plates in the era of digital microbiology

Full Laboratory Automation is revolutionizing work habits in an increasing number of clinical microbiology facilities worldwide, generating huge streams of digital images for interpretation. Contextually, deep learning architectures are leading to paradigm shifts in the way computers can assist with difficult visual interpretation tasks in several domains. At the crossroads of these epochal trends, we present a system able to tackle a core task in clinical microbiology, namely the global interpretation of diagnostic bacterial culture plates, including presumptive pathogen identification. This is achieved by decomposing the problem into a hierarchy of complex subtasks and addressing them with a multi-network architecture we call DeepColony. Working on a large stream of clinical data and a complete set of 32 pathogens, the proposed system is capable of effectively assist plate interpretation with a surprising degree of accuracy in the widespread and demanding framework of Urinary Tract Infections. Moreover, thanks to the rich species-related generated information, DeepColony can be used for developing trustworthy clinical decision support services in laboratory automation ecosystems from local to global scale.