Video Sensor Research Articles

MDAR: A Multiscale Features-Based Network for Remotely Measuring Human Heart Rate Utilizing Dual-Branch Architecture and Alternating Frame Shifts in Facial Videos.

Remote photoplethysmography (rPPG) refers to a non-contact technique that measures heart rate through analyzing the subtle signal changes of facial blood flow captured by video sensors. It is widely used in contactless medical monitoring, remote health management, and activity monitoring, providing a more convenient and non-invasive way to monitor heart health. However, factors such as ambient light variations, facial movements, and differences in light absorption and reflection pose challenges to deep learning-based methods. To solve these difficulties, we put forward a measurement network of heart rate based on multiscale features. In this study, we designed and implemented a dual-branch signal processing framework that combines static and dynamic features, proposing a novel and efficient method for feature fusion, enhancing the robustness and reliability of the signal. Furthermore, we proposed an alternate time-shift module to enhance the model's temporal depth. To integrate the features extracted at different scales, we utilized a multiscale feature fusion method, enabling the model to accurately capture subtle changes in blood flow. We conducted cross-validation on three public datasets: UBFC-rPPG, PURE, and MMPD. The results demonstrate that MDAR not only ensures fast inference speed but also significantly improves performance. The two main indicators, MAE and MAPE, achieved improvements of at least 30.6% and 30.2%, respectively, surpassing state-of-the-art methods. These conclusions highlight the potential advantages of MDAR for practical applications.

A method to estimate prey densityfrom single-camera images: A case study with chinstrap penguins and Antarctic krill.

Estimating the densities of marine prey observed in animal-borne video loggers when encountered by foraging predators represents an important challenge for understanding predator-prey interactions in the marine environment. We used video images collected during the foraging trip of one chinstrap penguin (Pygoscelis antarcticus) from Cape Shirreff, Livingston Island, Antarctica to develop a novel approach for estimating the density of Antarctic krill (Euphausia superba) encountered during foraging activities. Using the open-source Video and Image Analytics for a Marine Environment (VIAME), we trained a neural network model to identify video frames containing krill. Our image classifier has an overall accuracy of 73%, with a positive predictive value of 83% for prediction of frames containing krill. We then developed a method to estimate the volume of water imaged, thus the density (N·m-3) of krill, in the 2-dimensional images. The method is based on the maximum range from the camera where krill remain visibly resolvable and assumes that mean krill length is known, and that the distribution of orientation angles of krill is uniform. From 1,932 images identified as containing krill, we manually identified a subset of 124 images from across the video record that contained resolvable and unresolvable krill necessary to estimate the resolvable range and imaged volume for the video sensor. Krill swarm density encountered by the penguins ranged from 2 to 307 krill·m-3 and mean density of krill was 48 krill·m-3 (sd = 61 krill·m-3). Mean krill biomass density was 25 g·m-3. Our frame-level image classifier model and krill density estimation method provide a new approach to efficiently process video-logger data and estimate krill density from 2D imagery, providing key information on prey aggregations that may affect predator foraging performance. The approach should be directly applicable to other marine predators feeding on aggregations of prey.

Distributed video transmission reduction approach for energy saving in WMSNs

SummaryWireless Multimedia Sensor Networks (WMSNs) are composed of a large number of sensor nodes that are distributed in a region to collect and transmit data. Video transmission is one of the most important applications of WMSNs because it can provide critical information about monitored areas. WMSNs face challenges related to energy consumption, bandwidth usage, and network congestion related to huge amounts of data collected by sensors. To tackle this problem, this paper proposes the Distributed Video Transmission Reduction Approach for Energy Saving in WMSN (DiViTRA). The method involves two phases: sensing and transmission phases. DiViTRA achieves frame rate adaptation to reduce the number of captured video frames and save energy during the sensing phase. In the transmission phase, three effective techniques, ORB (Oriented FAST and Rotated BRIEF), Brute‐Force (BF) Matcher, and Grid‐based Motion Statistics (GMS) are applied to decide whether to transmit the current captured frame or remove it and adjust the frame capturing rate of the video sensor accordingly. In the case of frame transmission, the DiViTRA approach compresses the frame using two data reduction approaches: PCA (Principal Component Analysis) and Huffman encoding. Through simulations, DiViTRA demonstrates a 12% reduction in energy consumption, and 71% is a ratio of reduction in sent frames while preserving stream quality. The approach has been validated in scenarios involving critical events, showcasing its efficacy in maintaining data integrity during transmission.

Intelligent Unmanned Robot Using IOT for Military Applications

Most military organization now takes the help of robots to carry out many risky jobs that cannot be done by the soldier. These robots used in the military are usually employed with an integrated system, including video screens, sensors, grippers, and cameras. The military robots also have different shapes according to the purposes of each robot. Here the new system is proposed with the help of low low-power IOT wireless sensor network to trace out the intruders (unknown persons) and the robot will take the necessary action automatically. Thus the proposed system, an Intelligent Unmanned Robot (IUR) using IOT saves human lives and reduces manual error on the defense side. This is a specially designed robotic system to save human life and protect the country from enemies. Robots are specially designed for humans to make our lives easier. Robots are designed for various purposes like military purposes, industry, for home home-based applications. At the border, different tanks, missiles, guns, etc. are used by the enemy. This causes problems and harms our forces or soldiers. For this, a robot is designed and developed for military purposes application to protect our army. robots are used to detect obstacles that are found in their path. If it finds any obstacle in its path, then using a gun mechanism it will able to shoot that obstacle. To make it a multifunctional robot all the actions performed by the user same actions performed by a robot using the stretch sensor. All these mechanisms are embedded in the propeller.

Congestion and throughput optimization protocol for providing better quality of service and experience

<p>Multimedia traffic in Internet of Things applications is generated for various purposes and encompasses a wide range of multimedia data, including video streams, audio files, images, and sensor data. Network providers employ various strategies to handle multimedia traffic in IoT applications efficiently. But most of these methods have not considered optimizing the RTSP (Real-Time Streaming Protocol), RTP (Real-time Transport Protocol), and RTCP (Real-Time Control Protocol) to improve the throughput and QoS of the IoT applications. Hence, in this Congestion and Throughput Optimization Protocol (CTOP) work, we present a model which optimizes the RTSP, RTP, and RTCP protocol to improve the throughput and QoS. The CTOP model outperforms the Big Packet Protocol model in terms of average throughput, multimedia loss, delay, and energy consumption for both less and high-traffic scenarios. For less-level of traffic and high level of traffic, the CTOP model achieves a better average throughput, and average multimedia delay, reducing the average multimedia loss and average energy consumption in comparison to the existing BBP model. These results highlight the improved performance and efficiency of the CTOP model compared to the BBP model.</p>

HawkEar: A bird‐borne visual and acoustic platform for eavesdropping the behaviour of mobile animals

Abstract Unoccupied aerial vehicles (UAVs; drones) offer mobile platforms for ecological investigation, but can be impractical in some environments and the resulting noise can disturb wildlife. We developed a mobile alternative using a bird‐borne platform to record the behaviour of other animals in the field. This unit consists of a lightweight audio and video sensor that is carried by a trained Harris's hawk Parabuteo unicinctus. We tested the hypothesis that our bird‐borne platform is a viable option for collecting behavioural data from mobile animals. We recorded acoustic and video data as the hawk flew through a dense group of Brazilian free‐tailed bats Tadarida brasiliensis emerging from a cave, with a test case of investigating how echolocation calls change depending on spatial position in the bat group. The HawkEar platform is an alternative for collecting behavioural data when a mobile platform that is less noisy and restrictive than traditional UAVs is needed. The design and software are open source and can be modified to accommodate additional sensor needs.

Noncontact respiration monitoring techniques in young children: A scoping review.

Pediatric sleep-related breathing disorders, or sleep-disordered breathing (SDB), cover a range of conditions, including obstructive sleep apnea,central sleep apnea,sleep-related hypoventilation disorders, and sleep-related hypoxemia disorder. Pediatric SDB is often underdiagnosed, potentially due to difficulties associated with performing the gold standard polysomnographyin children. This scoping review aims to: (1) provide an overview of the studies reporting on safe, noncontact monitoring of respiration in young children,(2) describe the accuracy of these techniques, and (3) highlight their respective advantages and limitations. PubMed and EMBASE were searched for studies researching techniques in children <12 years old. Both quantitative data and the quality of the studies were analyzed. The evaluation of study quality was conducted using the QUADAS-2 tool. A total of 19 studies were included. Techniques could be grouped into bed-based methods, microwave radar, video, infrared (IR) cameras, and garment-embedded sensors. Most studies either measured respiratory rate (RR) or detected apneas; n = 2 aimed to do both. At present, bed-based approaches are at the forefront of research in noncontact RRmonitoring in children, boasting the most sophisticated algorithms in this field. Yet, despite extensive studies, there remains no consensus on a definitive method that outperforms the rest. The accuracies reported by these studies tend to cluster within a similar range, indicating that no single technique has emerged as markedly superior. Notably, all identified methods demonstrate capability in detecting body movements and RR, with reported safety for use in children across the board. Further research into contactless alternatives should focus on cost-effectiveness, ease-of-use, and widespread availability.

A Review and Comparative Study of Works that Care is Monitoring Detection and Therapy of Children with Autism Spectrum Disorder

Recognizing human activity from video sequences and sensor data is one of the major challenges in human-computer interaction and computer vision. Health care is a rapidly developing field of technology and services. The latest development in this field is remote patient monitoring, which has many advantages in a rapidly evolving world. With relatively simple applications for monitoring patients within hospital rooms, technology has advanced to the point where a patient can be allowed to carry out normal daily activities at home while still being monitored using modern communication technologies and sensors. These new technologies can monitor patients based on their disease or condition. The technology varies from sensors attached to the body to peripheral sensors connected to the environment, and innovations show contactless monitoring that only requires the patient to be within a few meters of the sensor. Nowadays, the Internet of Things, wearable devices, mobile technologies, and improved communication and computing capabilities have given rise to innovative mobile health solutions, and several research efforts have recently been made in the field of autism spectrum disorders (ASD). This technology may be particularly useful for some rapidly changing emotional states, especially people with ASD. Children with ASD have some disturbing activities, and usually cannot speak fluently. Instead, they use signs and words to establish rapport, so understanding their needs is one of the most challenging tasks for healthcare providers, but monitoring the disease can make it much easier. We study in this work more than 50 collected articles that have made a significant contribution to the field were selected. Indeed, the current paper reviews the literature to identify current trends, expectations, and potential gaps related to the latest portable, smart, and wearable technologies in the field of ASD. This study also provides a review of recent developments in health care and monitoring of people with autism.

The *.* Group’s Contribution to the Development of Technological Art in Brazil

Abstract The author describes three interactive installations presented by the *.* Group from the turn of the 1980s to the 1990s. The first installation, Fractal Art, explored the fractal theme through various mediums, including images, music, sculpture, video, and photoelectric sensors. The second installation, Foreseen Variations, featured a robot performing a choreography with a dancer in an environment equipped with a camera and TV sets, allowing the audience to be immersed in a virtual performance. The third installation, Aurora Beings, was a continuation of the previous one, where young people could program a choreography for the robot and act as both artists and engineers within the environment.

Optimizing spectrum efficiency in 6G multi-UAV networks through source correlation exploitation

In the context of 6G and 5G communication networks, particularly in disaster-stricken areas with a surging demand for connectivity and the rapid evolution of the internet of everything (IoE), the imperative for augmenting spectrum efficiency in unmanned aerial vehicles (UAVs) communication-enabled wireless networks becomes ever more pronounced. This work embarks on the mission of significantly enhancing spectrum efficiency by capitalizing on redundancies, such as correlations inherent in data sources like co-located video sensors. While the intuitive potential of exploiting redundancies for spectrum efficiency enhancement is apparent, our approach is systematically structured. We introduce a linear programming framework designed to meticulously allocate bandwidth to individual links based on their respective demands, while judiciously adhering to spatial spectrum reuse constraints. Subsequently, we employ this linear program to empirically quantify the improvements in spectrum efficiency engendered by source correlations. Furthermore, we elucidate various strategies for harnessing these correlations to maximize spectrum efficiency gains, while navigating the trade-off terrain between computational complexity and precision. Our findings resoundingly underscore the trans-formative power of identifying the precise set of source correlations, resulting in spectrum efficiency enhancements of up to two orders of magnitude. In terms of network performance, the judicious exploitation of source correlations grants admission to nearly 100% of delay-intolerant traffic, alongside substantial reductions in mean delay for delay-tolerant traffic.”

Attention-enhanced gated recurrent unit for action recognition in tennis.

Human Action Recognition (HAR) is an essential topic in computer vision and artificial intelligence, focused on the automatic identification and categorization of human actions or activities from video sequences or sensor data. The goal of HAR is to teach machines to comprehend and interpret human movements, gestures, and behaviors, allowing for a wide range of applications in areas such as surveillance, healthcare, sports analysis, and human-computer interaction. HAR systems utilize a variety of techniques, including deep learning, motion analysis, and feature extraction, to capture and analyze the spatiotemporal characteristics of human actions. These systems have the capacity to distinguish between various actions, whether they are simple actions like walking and waving or more complex activities such as playing a musical instrument or performing sports maneuvers. HAR continues to be an active area of research and development, with the potential to enhance numerous real-world applications by providing machines with the ability to understand and respond to human actions effectively. In our study, we developed a HAR system to recognize actions in tennis using an attention-based gated recurrent unit (GRU), a prevalent recurrent neural network. The combination of GRU architecture and attention mechanism showed a significant improvement in prediction power compared to two other deep learning models. Our models were trained on the THETIS dataset, one of the standard medium-sized datasets for fine-grained tennis actions. The effectiveness of the proposed model was confirmed by three different types of image encoders: InceptionV3, DenseNet, and EfficientNetB5. The models developed with InceptionV3, DenseNet, and EfficientNetB5 achieved average ROC-AUC values of 0.97, 0.98, and 0.81, respectively. While, the models obtained average PR-AUC values of 0.84, 0.87, and 0.49 for InceptionV3, DenseNet, and EfficientNetB5 features, respectively. The experimental results confirmed the applicability of our proposed method in recognizing action in tennis and may be applied to other HAR problems.

Satellite Video Remote Sensing for Flood Model Validation

AbstractSatellite‐based optical video sensors are poised as the next frontier in remote sensing. Satellite video offers the unique advantage of capturing the transient dynamics of floods with the potential to supply hitherto unavailable data for the assessment of hydraulic models. A prerequisite for the successful application of hydraulic models is their proper calibration and validation. In this investigation, we validate 2D flood model predictions using satellite video‐derived flood extents and velocities. Hydraulic simulations of a flood event with a 5‐year return period (discharge of 722 m3 s−1) were conducted using Hydrologic Engineering Center—River Analysis System 2D in the Darling River at Tilpa, Australia. To extract flood extents from satellite video of the studied flood event, we use a hybrid transformer‐encoder, convolutional neural network (CNN)‐decoder deep neural network. We evaluate the influence of test‐time augmentation (TTA)—the application of transformations on test satellite video image ensembles, during deep neural network inference. We employ Large Scale Particle Image Velocimetry (LSPIV) for non‐contact‐based river surface velocity estimation from sequential satellite video frames. When validating hydraulic model simulations using deep neural network segmented flood extents, critical success index peaked at 94% with an average relative improvement of 9.5% when TTA was implemented. We show that TTA offers significant value in deep neural network‐based image segmentation, compensating for aleatoric uncertainties. The correlations between model predictions and LSPIV velocities were reasonable and averaged 0.78. Overall, our investigation demonstrates the potential of optical space‐based video sensors for validating flood models and studying flood dynamics.

An Improved MPSO Coverage Enhancement Algorithm for Directional Video Sensor Networks

An Improved MPSO Coverage Enhancement Algorithm for Directional Video Sensor Networks

An improved coverage optimization method for video sensor networks based on whale optimization algorithm

An improved coverage optimization method for video sensor networks based on whale optimization algorithm

Roadway Weather Challenges Illuminate Real‐World Driving Biomarkers of Dementia Risk

AbstractBackgroundCognitive decline is a leading early predictor of dementia risk and may be evident in real‐world (RW) digital data even before patients ever present to clinic. This study examined the feasibility of using video and digital sensor profiles of real world (RW) driving in adverse weather as a marker of dementia risk.Method70 older drivers (mean age = 75.6 years) without dementia participated in a RW driving study for 2, 3‐month periods, one year apart (Wang, et al. 2021) . Self‐reported demographics and neuropsychological tests relevant to dementia and driving were collected at study start. Drivers were classed as cognitively neurotypical (N = 33) or impaired (N = 44) based on normed neuropsychological scores. Sensors installed in each driver’s vehicle collected video, speed, and GPS data across 405,104 miles driven. GIS data categorized roadway type (residential, commercial, interstate). A computer vision model (99% accuracy) we developed classed roads as clear or poor (rain, snow). Bayesian mixed‐effect logistic regressions assessed driver speed limit compliance (vehicle speed‐posted speed limit) across driver class, roadway, and weather.ResultDrivers were 8.01 times more likely to speed on interstates (95% CI: 7.61‐8.61) than residential roads (15‐25 mph). Impaired drivers were 15% more likely to drive slower than neurotypical drivers on clear roadways (95% C: 1.0413‐1.2829). This pattern reversed in poor weather with impaired drivers driving faster than neurotypical (b = 0.74, 95% CI: 0.57‐0.94)ConclusionDigital biomarkers from RW data show promise to screen for early dementia risk, advancing research and innovative clinical trials to prevent dementia. Driving challenges, like poor weather, discriminate cognitive decline before dementia. Impaired drivers showed self‐restriction by driving slower on clear roads and reduced self‐restriction by driving faster in poor weather despite risk. Big data strategies, like computer vision, complement driver performance data to identify RW environmental risk, paving the road for the car as a medical diagnostic device.

A smart e-health framework for monitoring the health of the elderly and disabled

The healthcare sector is experiencing a significant transformation due to the widespread adoption of IoT-based systems, especially in the care of elderly and disabled individuals who can now be monitored through portable and wearable devices. The widespread implementation of IoT-based systems in the healthcare sector holds immense potential for improving care and ensuring the well-being of vulnerable populations. In our paper, we introduce an e-health framework that utilizes real-time data from inertial, ECG, and video sensors to monitor the health and activities of these individuals. Our framework employs edge computing for efficient analysis and generates notifications based on data analysis while prioritizing privacy by activating multimedia sensors only when necessary. To achieve accurate results, we evaluated each component of our framework separately. Using the MHEALTH dataset, our proposed machine learning model achieves accuracies of 99.97% for inertial sensors’ human activity recognition (HAR) and 99.14% for ECG sensors. We also evaluated our first-level inconsistency detection-based anomaly proposal mechanism using one user’s data, demonstrating its proof-of-concept. Furthermore, our video-based HAR and fall detection module achieve an accuracy of 86.97% on the well-known DMLSmartActions dataset. We successfully deployed our proposed HAR model with inertial sensors in a controlled experimental environment, yielding promising results with an accuracy of 96% and an F1 score of 90.52%. These evaluations validate the effectiveness of our framework in monitoring the health and activities of elderly and disabled individuals.

Time budgets of dairy cows in a cow-calf contact system with automatic milking

Daily time budgets can be used to determine the amount of time cows allocate to different behaviors throughout the day and can be useful when evaluating housing and management systems. There is a growing interest in keeping dairy cows and calves together during the first months of lactation; however, it is not known how their time budgets are affected by the calf contact. The aim of this study was to investigate differences in time budgets between 2 groups of dairy cows housed in the same pen within a freestall system with automatic milking. One group of cows had access to their calves until 4 mo of lactation, whereas the other group had no direct contact with the calves. Using focal animal sampling by video analysis and sensor data available from the milking unit we determined the 24-h time budgets of 37 dairy cows. The sample consisted of 20 primi- and 17 multiparous cows of the breeds Swedish Red and Swedish Holstein. The cows were randomly assigned either cow-calf contact or no contact, with separation from the calf within 24 h postparturition. Data were collected when cows were on average 43, 75, and 107 d in milk. The results showed that the cow-calf cows spent less time eating silage, without reduced dry matter intake, less time socializing with other cows, and less time standing in cubicles. However, the duration spent in the waiting area in front of the milking unit was greater among the cow-calf cows. In conclusion, neither cow-calf contact nor no-contact cows seemed to be constrained in any of the included behaviors to a point of reduced welfare; hence, a cow-calf contact system in combination with automatic milking may be possible to set up on farms with maintained or improved animal welfare.

Machine-Learning-Based Approaches for Post-Traumatic Stress Disorder Diagnosis Using Video and EEG Sensors: A Review

Machine-Learning-Based Approaches for Post-Traumatic Stress Disorder Diagnosis Using Video and EEG Sensors: A Review

Stomach eversion and retraction by a tagged tiger shark at Ningaloo Reef, Western Australia

Stomach eversion is common in sharks captured during fishing activities, however, records of subsequent retraction of the stomach once animals are released are rare and limited by the logistics of recording post-release behaviors in the wild. Here, we report stomach eversion and retraction by a tiger shark captured and tagged at Ningaloo Reef, Western Australia. Initial eversion occurred during capture, and retraction was recorded by video and tri-axial acceleration sensors approximately five minutes after release. The ability to retract the stomach without apparent ill effect is consistent with the documented resilience of tiger sharks to stressful capture events, but remains to be confirmed for other species.

VAX

The use of audio and video modalities for Human Activity Recognition (HAR) is common, given the richness of the data and the availability of pre-trained ML models using a large corpus of labeled training data. However, audio and video sensors also lead to significant consumer privacy concerns. Researchers have thus explored alternate modalities that are less privacy-invasive such as mmWave doppler radars, IMUs, motion sensors. However, the key limitation of these approaches is that most of them do not readily generalize across environments and require significant in-situ training data. Recent work has proposed cross-modality transfer learning approaches to alleviate the lack of trained labeled data with some success. In this paper, we generalize this concept to create a novel system called VAX (Video/Audio to 'X'), where training labels acquired from existing Video/Audio ML models are used to train ML models for a wide range of 'X' privacy-sensitive sensors. Notably, in VAX, once the ML models for the privacy-sensitive sensors are trained, with little to no user involvement, the Audio/Video sensors can be removed altogether to protect the user's privacy better. We built and deployed VAX in ten participants' homes while they performed 17 common activities of daily living. Our evaluation results show that after training, VAX can use its onboard camera and microphone to detect approximately 15 out of 17 activities with an average accuracy of 90%. For these activities that can be detected using a camera and a microphone, VAX trains a per-home model for the privacy-preserving sensors. These models (average accuracy = 84%) require no in-situ user input. In addition, when VAX is augmented with just one labeled instance for the activities not detected by the VAX A/V pipeline (~2 out of 17), it can detect all 17 activities with an average accuracy of 84%. Our results show that VAX is significantly better than a baseline supervised-learning approach of using one labeled instance per activity in each home (average accuracy of 79%) since VAX reduces the user burden of providing activity labels by 8x (~2 labels vs. 17 labels).