Consumer-grade Hardware Research Articles

EyeLoop: An Open-Source System for High-Speed, Closed-Loop Eye-Tracking.

Eye-trackers are widely used to study nervous system dynamics and neuropathology. Despite this broad utility, eye-tracking remains expensive, hardware-intensive, and proprietary, limiting its use to high-resource facilities. It also does not easily allow for real-time analysis and closed-loop design to link eye movements to neural activity. To address these issues, we developed an open-source eye-tracker – EyeLoop – that uses a highly efficient vectorized pupil detection method to provide uninterrupted tracking and fast online analysis with high accuracy on par with popular eye tracking modules, such as DeepLabCut. This Python-based software easily integrates custom functions using code modules, tracks a multitude of eyes, including in rodents, humans, and non-human primates, and operates at more than 1,000 frames per second on consumer-grade hardware. In this paper, we demonstrate EyeLoop’s utility in an open-loop experiment and in biomedical disease identification, two common applications of eye-tracking. With a remarkably low cost and minimum setup steps, EyeLoop makes high-speed eye-tracking widely accessible.

Feasibility Analysis of Fifth-generation (5G) Mobile Networks for Transmission of Medical Imaging Data.

Next to higher data rates and lower latency, the upcoming fifth-generation mobile network standard will introduce a new service ecosystem. Concepts such as multi-access edge computing or network slicing will enable tailoring service level requirements to specific use-cases. In medical imaging, researchers and clinicians are currently working towards higher portability of scanners. This includes i) small scanners to be wheeled inside the hospital to the bedside and ii) conventional scanners provided via trucks to remote areas. Both use-cases introduce the need for mobile networks adhering to high safety standards and providing high data rates. These requirements could be met by fifth-generation mobile networks. In this work, we analyze the feasibility of transferring medical imaging data using the current state of development of fifth-generation mobile networks (3GPP Release 15). We demonstrate the potential of reaching 100Mbit/s upload rates using already available consumer-grade hardware. Furthermore, we show an effective average data throughput of 50Mbit/s when transferring medical images using out-of-the-box open-source software based on the Digital Imaging and Communications in Medicine (DICOM) standard. During transmissions, we sample the radio frequency bands to analyse the characteristics of the mobile radio network. Additionally, we discuss the potential of new features such as network slicing that will be introduced in forthcoming releases.

SMS-OPTIMIZED CONSUMER-GRADE TRACKING SYSTEM FOR LOW-COST MONITORING

Consumer grade tracking systems exist but are not optimally designed for low-cost operation in the Philippines setting. Tracking devices typically use Global Positioning System (GPS) and Global System for Mobile Communications (GSM) technologies. Commonly, for device management and real-time tracking, data are transmitted as General Packet Radio Service (GPRS) to a proprietary or third-party cloud service. This method is costly in the Philippines context as well as power consuming. This study was conducted to design a tracking system that is optimized for short messaging service (SMS) mode of data transmission. This study covered the design of a tracking device using consumer-grade hardware components, development of system interface to enable remote operation using commands sent as short message(s), development of a simple desktop client monitoring service and actual field-testing using land and water vehicles. Tests results showed a 100% tracking data delivery efficiency and small difference between Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) for each route indicating that there is a small variance in distance errors. The tracking device also performed well in a sea route which is comparable to a commercial standard GPS data logger. Test results also showed that the device can be used for tracking applications i.e., vehicle security, inland fleet management, monitoring of tourist boats, and research or field surveys, where GPS and GSM are available.

Assembly of Biomolecular Gigastructures and Visualization with the Vulkan Graphics API.

Building and displaying all-atom models of biomolecular structures with millions or billions of atoms, like virus particles or cells, remain a challenge due to the sheer size of the data, the required levels of automated building, and the visualization limits of today’s graphics hardware. Based on concepts introduced with the CellPack program, we report new algorithms to create such large-scale models using an intermediate coarse-grained “pet representation” of biomolecules with 1/10th the normal size. Pet atoms are placed such that they optimally trace the surface of the original molecule with just ∼1/50th the original atom number and are joined with covalent bonds. Molecular dynamics simulations of pet molecules allow for efficient packing optimization, as well as the generation of realistic DNA/RNA conformations. This pet world can be expanded back to the all-atom representation to be explored and visualized with full details. Essential for the efficient interactive visualization of gigastructures is the use of multiple levels of detail (LODs), where distant molecules are drawn with a heavily reduced polygon count. We present a grid-based algorithm to create such LODs for all common molecular graphics styles (including ball-and-sticks, ribbons, and cartoons) that do not require monochrome molecules to hide LOD transitions. As a practical application, we built all-atom models of SARS-CoV-2, HIV, and an entire presynaptic bouton with 1 μm diameter and 3.6 billion atoms, using modular building blocks to significantly reduce GPU memory requirements through instancing. We employ the Vulkan graphics API to maximize performance on consumer grade hardware and describe how to use the mmCIF format to efficiently store such giant models. An implementation is available as part of the YASARA molecular modeling and simulation program from www.YASARA.org. The free YASARA View program can be used to explore the presented models, which can be downloaded from www.YASARA.org/petworld, a Creative Commons platform for sharing giant biomolecular structures.

University Implementation of TEACHActive – An Automated Classroom Feedback System and Dashboard

Computer vision has the potential to play a significant role in capacity building for classroom instructors via automated feedback. This paper describes the implementation of an automated sensing and feedback system, TEACHActive. The results of this paper can enable other campuses to replicate a similar system using open-source software and consumer-grade hardware. Some of the challenges discussed include faculty recruitment, IRB procedures, camera-based classroom footage privacy, hardware setup, software setup, and IT support. The design and implementation of the TEACHActive system is being carried out at Iowa State University and is being tested with faculty in classrooms pilots. Preliminary interviews with instructors show a desire to include more active learning methods in their classrooms and overall interest in a system that can perform automated feedback. The primary results of this paper include lessons learned from the institutional implementation process.

Performance and Pitfalls of 60 GHz WLANs Based on Consumer-Grade Hardware

Wireless networks operating in the 60 GHz band have the potential to provide very high throughput but face a number of challenges (e.g., high attenuation, beam training, and coping with mobility) which are widely accepted but often not well understood in practice. Understanding these challenges, and especially their actual impact on consumer-grade hardware is fundamental to fully exploit the high physical layer rates in the 60 GHz band. To this end, we perform an extensive measurement campaign using two commercial off-the-shelf 60 GHz routers in real-world environments. Our results allow us to revisit a range of issues and provide much deeper insights into the reasons for specific performance compared to prior work on performance characterization. Further, our study goes beyond basic link characterization and explores for the first time practical considerations such as coverage and access point deployment. While some of our observations are expected, we also obtain highly surprising insights that challenge the prevailing wisdom in the community. We derive the shortcomings of current commercial 60 GHz devices, and the fundamental problems that remain open on the way to fast and efficient 60 GHz networking.

Emerging Hearing Assessment Technologies for Patient Care

Emerging Hearing Assessment Technologies for Patient Care

Fast variational alignment of non-flat 1D displacements for applications in neuroimaging

BackgroundIn the context of signal analysis and pattern matching, alignment of 1D signals for the comparison of signal morphologies is an important problem. For image processing and computer vision, 2D optical flow (OF) methods find wide application for motion analysis and image registration and variational OF methods have been continuously improved over the past decades. New methodWe propose a variational method for the alignment and displacement estimation of 1D signals. We pose the estimation of non-flat displacements as an optimization problem with a similarity and smoothness term similar to variational OF estimation. To this end, we can make use of efficient optimization strategies that allow real-time applications on consumer grade hardware. ResultsWe apply our method to two applications from functional neuroimaging: The alignment of 2-photon imaging line scan recordings and the denoising of evoked and event-related potentials in single trial matrices. We can report state of the art results in terms of alignment quality and computing speeds. Existing methodsExisting methods for 1D alignment target mostly constant displacements, do not allow native subsample precision or precise control over regularization or are slower than the proposed method. ConclusionsOur method is implemented as a MATLAB toolbox and is online available. It is suitable for 1D alignment problems, where high accuracy and high speed is needed and non-constant displacements occur.

Portable Automated Rapid Testing (PART) for auditory assessment: Validation in a young adult normal-hearing population.

This study aims to determine the degree to which Portable Automated Rapid Testing (PART), a freely available program running on a tablet computer, is capable of reproducing standard laboratory results. Undergraduate students were assigned to one of three within-subject conditions that examined repeatability of performance on a battery of psychoacoustical tests of temporal fine structure processing, spectro-temporal amplitude modulation, and targets in competition. The repeatability condition examined test/retest with the same system, the headphones condition examined the effects of varying headphones (passive and active noise-attenuating), and the noise condition examined repeatability in the presence of recorded cafeteria noise. In general, performance on the test battery showed high repeatability, even across manipulated conditions, and was similar to that reported in the literature. These data serve as validation that suprathreshold psychoacoustical tests can be made accessible to run on consumer-grade hardware and perform in less controlled settings. This dataset also provides a distribution of thresholds that can be used as a normative baseline against which auditory dysfunction can be identified in future work.

Long-Term Tracking of Group-Housed Livestock Using Keypoint Detection and MAP Estimation for Individual Animal Identification.

Tracking individual animals in a group setting is a exigent task for computer vision and animal science researchers. When the objective is months of uninterrupted tracking and the targeted animals lack discernible differences in their physical characteristics, this task introduces significant challenges. To address these challenges, a probabilistic tracking-by-detection method is proposed. The tracking method uses, as input, visible keypoints of individual animals provided by a fully-convolutional detector. Individual animals are also equipped with ear tags that are used by a classification network to assign unique identification to instances. The fixed cardinality of the targets is leveraged to create a continuous set of tracks and the forward-backward algorithm is used to assign ear-tag identification probabilities to each detected instance. Tracking achieves real-time performance on consumer-grade hardware, in part because it does not rely on complex, costly, graph-based optimizations. A publicly available, human-annotated dataset is introduced to evaluate tracking performance. This dataset contains 15 half-hour long videos of pigs with various ages/sizes, facility environments, and activity levels. Results demonstrate that the proposed method achieves an average precision and recall greater than 95% across the entire dataset. Analysis of the error events reveals environmental conditions and social interactions that are most likely to cause errors in real-world deployments.

Feasibility and Validity of a Low-Cost Racing Simulator in Driving Assessment after Stroke.

There is a myriad of methodologies to assess driving performance after a stroke. These include psychometric tests, driving simulation, questionnaires, and/or road tests. Research-based driving simulators have emerged as a safe, convenient way to assess driving performance after a stroke. Such traditional research simulators are useful in recreating street traffic scenarios, but are often expensive, with limited physics models and graphics rendering. In contrast, racing simulators developed for motorsport professionals and enthusiasts offer high levels of realism, run on consumer-grade hardware, and can provide rich telemetric data. However, most offer limited simulation of traffic scenarios. This pilot study compares the feasibility of research simulation and racing simulation in a sample with minor stroke. We determine that the racing simulator is tolerated well in subjects with a minor stroke. There were correlations between research and racing simulator outcomes with psychometric tests associated with driving performance, such as the Trails Making Test Part A, Snellgrove Maze Task, and the Motricity Index. We found correlations between measures of driving speed on a complex research simulator scenario and racing simulator lap time and maximum tires off track. Finally, we present two models, using outcomes from either the research or racing simulator, predicting road test failure as linked to a previously published fitness-to-drive calculator that uses psychometric screening.

Low-light image enhancement of high-speed endoscopic videos using a convolutional neural network.

Laryngeal endoscopy is one of the primary diagnostic tools for laryngeal disorders. The main techniques are videostroboscopy and lately high-speed video endoscopy. Unfortunately, due to the restricting anatomy of the larynx and technical limitations of the recording equipment, many videos suffer from insufficient illumination, which complicates clinical examination and analysis. This work presents an approach to enhance low-light images from high-speed video endoscopy using a convolutional neural network. We introduce a new technique to generate realistically darkened training samples using Perlin noise. Extensive data augmentation is employed to cope with the limited training data allowing training with just 55 videos. The approach is compared against four state-of-the-art low-light enhancement methods and statistically significantly outperforms each on a no-reference (NIQE) and two full-reference (PSNR, SSIM) image quality metrics. The presented approach can be run on consumer-grade hardware and is thereby directly applicable in a clinical context. It is likely transferable to similar techniques such as videostroboscopy. Graphical Abstract The basic setup for training and employing an improved fully convolutional U-Net neural network to predict a brightness map used to enhance the lighting of ill-lit endoscopic high-speed videos - Artificially darkened training data are created using Perlin noise to allow region-specific darkening.

SLAMCast: Large-Scale, Real-Time 3D Reconstruction and Streaming for Immersive Multi-Client Live Telepresence

Real-time 3D scene reconstruction from RGB-D sensor data, as well as the exploration of such data in VR/AR settings, has seen tremendous progress in recent years. The combination of both these components into telepresence systems, however, comes with significant technical challenges. All approaches proposed so far are extremely demanding on input and output devices, compute resources and transmission bandwidth, and they do not reach the level of immediacy required for applications such as remote collaboration. Here, we introduce what we believe is the first practical client-server system for real-time capture and many-user exploration of static 3D scenes. Our system is based on the observation that interactive frame rates are sufficient for capturing and reconstruction, and real-time performance is only required on the client site to achieve lag-free view updates when rendering the 3D model. Starting from this insight, we extend previous voxel block hashing frameworks by introducing a novel thread-safe GPU hash map data structure that is robust under massively concurrent retrieval, insertion and removal of entries on a thread level. We further propose a novel transmission scheme for volume data that is specifically targeted to Marching Cubes geometry reconstruction and enables a 90% reduction in bandwidth between server and exploration clients. The resulting system poses very moderate requirements on network bandwidth, latency and client-side computation, which enables it to rely entirely on consumer-grade hardware, including mobile devices. We demonstrate that our technique achieves state-of-the-art representation accuracy while providing, for any number of clients, an immersive and fluid lag-free viewing experience even during network outages.

Automated detection and sorting of microencapsulation via machine learning.

Microfluidic-based microencapsulation requires significant oversight to prevent material and quality loss due to sporadic disruptions in fluid flow that routinely arise. State-of-the-art microcapsule production is laborious and relies on experts to monitor the process, e.g. through a microscope. Unnoticed defects diminish the quality of collected material and/or may cause irreversible clogging. To address these issues, we developed an automated monitoring and sorting system that operates on consumer-grade hardware in real-time. Using human-labeled microscope images acquired during typical operation, we train a convolutional neural network that assesses microencapsulation. Based on output from the machine learning algorithm, an integrated valving system collects desirable microcapsules or diverts waste material accordingly. Although the system notifies operators to make necessary adjustments to restore microencapsulation, we can extend the system to automate corrections. Since microfluidic-based production platforms customarily collect image and sensor data, machine learning can help to scale up and improve microfluidic techniques beyond microencapsulation.

Pedestrian Heading Estimation Based on Spatial Transformer Networks and Hierarchical LSTM

Accurate heading estimation is the foundation of numerous applications, including augmented reality, pedestrian dead reckoning, and human-computer interactions. While magnetometer is a key source of heading information, the poor accuracy of consumer-grade hardware coupled with the pervasive magnetic disturbances makes accurate heading estimation a challenging issue. Heading error is one of the main error sources of pedestrian dead reckoning. To reduce the heading error and enhance robustness, we proposed a novel heading estimation method based on Spatial Transformer Networks (STNs) and Long Short-Term Memory (LSTM), termed DeepHeading, which uses sensors embedded in a smartphone without any historical training data or dedicated infrastructure. We automatically annotate heading data based on map matching, and augment heading data based on device attitude. We leverage the STNs to align the device coordinate system and the navigation coordinate system, allow an unconstrained use of smartphones. Based on the characteristics of pedestrian heading continuity, we designed a hierarchical LSTM-basedSeq2Seq model to estimate the walking heading of the pedestrian. We conducted well-designed experiments to evaluate the performance of deepheading and compared it with the state-of-the-art heading estimation algorithms. The experimental results on real-world demonstrated that deepheading outperformed the compared heading estimation algorithms and achieved promising estimation accuracy with a median heading error of 4.52°, mean heading error of 6.07° and heading error of 9.18° at the confidence of 80% when a pedestrian is walking in indoor environments with magnetic field disturbances. The proposed method is high-efficiency and easy to integrate with various mobile applications.

Bend-it

Elastically deforming wire structures are lightweight, durable, and can be bent within minutes using CNC bending machines. We present a computational technique for the design of kinetic wire characters, tailored for fabrication on consumer-grade hardware. Our technique takes as input a network of curves or a skeletal animation, then estimates a cable-driven, compliant wire structure which matches user-selected targets or keyframes as closely as possible. To enable large localized deformations, we shape wire into functional spring-like entities at a discrete set of locations. We first detect regions where changes to local stiffness properties are needed, then insert bendable entities of varying shape and size. To avoid a discrete optimization, we first optimize stiffness properties of generic, non-fabricable entities which capture well the behavior of our bendable designs. To co-optimize stiffness properties and cable forces, we formulate an equilibrium-constrained minimization problem, safeguarding against inelastic deformations. We demonstrate our method on six fabricated examples, showcasing rich behavior including large deformations and complex, spatial motion.

Auto-Calibration for Dynamic Multi-Projection Mapping on Arbitrary Surfaces.

The quality of every dynamic multi-projection mapping system is limited by the quality of the projector to tracking device calibration. Common problems with poor calibration result in noticeable artifacts for the user, such as ghosting and seams. In this work we introduce a new, fully automated calibration algorithm that is tailored to reduce these artifacts, based on consumer-grade hardware. We achieve this goal by repurposing a structured-light scanning setup. A structured-light scanner can generate 3D geometry based on a known intrinsic and extrinsic calibration of its components (projector and RGB camera). We revert this process by providing the resulting 3D model to determine the intrinsic and extrinsic parameters of our setup (including those of a variety of tracking systems). Our system matches features and solves for all parameters in a single pass while respecting the lower quality of our sensory input.

Collaborative Large-Scale Dense 3D Reconstruction with Online Inter-Agent Pose Optimisation.

Reconstructing dense, volumetric models of real-world 3D scenes is important for many tasks, but capturing large scenes can take significant time, and the risk of transient changes to the scene goes up as the capture time increases. These are good reasons to want instead to capture several smaller sub-scenes that can be joined to make the whole scene. Achieving this has traditionally been difficult: joining sub-scenes that may never have been viewed from the same angle requires a high-quality camera relocaliser that can cope with novel poses, and tracking drift in each sub-scene can prevent them from being joined to make a consistent overall scene. Recent advances, however, have significantly improved our ability to capture medium-sized sub-scenes with little to no tracking drift: real-time globally consistent reconstruction systems can close loops and re-integrate the scene surface on the fly, whilst new visual-inertial odometry approaches can significantly reduce tracking drift during live reconstruction. Moreover, high-quality regression forest-based relocalisers have recently been made more practical by the introduction of a method to allow them to be trained and used online. In this paper, we leverage these advances to present what to our knowledge is the first system to allow multiple users to collaborate interactively to reconstruct dense, voxel-based models of whole buildings using only consumer-grade hardware, a task that has traditionally been both time-consuming and dependent on the availability of specialised hardware. Using our system, an entire house or lab can be reconstructed in under half an hour and at a far lower cost than was previously possible.

A Review of Commercial and Medical-Grade Physiological Monitoring Devices for Biofeedback-Assisted Quality of Life Improvement Studies.

With the rise in wearable technology and "health culture", we are seeing an increasing interest and affordances in studying how to not only prolong life expectancy but also in how to improve individuals' quality of life. On the one hand, this attempts to give meaning to the increasing life expectancy, as living above a certain threshold of pain and lack of autonomy or mobility is both degrading and unfair. On the other hand, it lowers the cost of continuous care, as individuals with high quality of life indexes tend to have lower hospital readmissions or secondary complications, not to mention higher physical and mental health. In this paper, we evaluate the current state of the art in physiological therapy (biofeedback) along with the existing medical grade and consumer grade hardware for physiological research. We provide a quick primer on the most commonly monitored physiologic metrics, as well as a brief discussion on the current state of the art in biofeedback-assisted medical applications. We then go on to present a comparative analysis between medical and consumer grade biofeedback devices and discuss the hardware specifications and potential practical applications of each consumer grade device in terms of functionality and adaptability for controlled (laboratory) and uncontrolled (field) studies. We end this article with some empirical observations based on our study so that readers might use take them into consideration when arranging a laboratory or real-world experience, thus avoiding costly time delays and material expenditures.

FaceForge: Markerless Non-Rigid Face Multi-Projection Mapping.

Recent publications and art performances demonstrate amazing results using projection mapping. To our knowledge, there exists no multi-projection system that can project onto non-rigid target geometries. This constrains the applicability and quality for live performances with multiple spectators. Given the cost and complexity of current systems, we present a low-cost easy-to-use markerless non-rigid face multi-projection system. It is based on a non-rigid, dense face tracker and a real-time multi-projection solver adapted to imprecise tracking, geometry and calibration. Using this novel system we produce compelling results with only consumer-grade hardware.