Vision Applications Research Articles

3D Gamut Morphing for Non-Rectangular Multi-Projector Displays.

In a spatially augmented reality system, multiple projectors are tiled on a complex shaped surface to create a seamless display on it. This has several applications in visualization, gaming, education and entertainment. The main challenges in creating seamless and undistorted imagery on such complex shaped surfaces are geometric registration and color correction. Prior methods that provide solutions for the spatial color variation in multi-projector displays assume rectangular overlap regions across the projectors that is possible only on flat surfaces with extremely constrained projector placement. In this article, we present a novel and fully automated method for removing color variations in a multi-projector display on arbitrary shaped smooth surfaces using a general color gamut morphing algorithm that can handle any arbitrarily shaped overlap between the projectors and assures imperceptible color variations across the display surface.

ShinySperm: navigating the sperm proteome landscape.

Context Integrated omics studies hold a crucial role in improving our understanding of reproductive biology. However, the complex datasets so generated are often only accessible via supplementary data files, which lack the capacity for interactive features to allow users to readily interrogate and visualise data of interest. Aims The intent of this technical note was to develop an interactive web-based application that enables detailed interrogation of a representative sperm proteome, facilitating a deeper understanding of the proteins identified, their relative abundance, classifications, functions, and associated phenotypes. Methods We developed a Shiny web application, ShinySperm (https://reproproteomics.shinyapps.io/ShinySperm/ ), utilising R and several complementary libraries for data manipulation (dplyr), interactive tables (DT), and visualisation (ggplot2, plotly). ShinySperm features a responsive user interface, dynamic filtering options, interactive charts, and data export capabilities. Key results ShinySperm allows users to interactively search, filter, and visualise sperm proteomics data based on key features (e.g. protein classification, sperm cell domain, presence, or absence at different maturation stages). This application responds live to filtering options and enables the generation of interactive plots and tables, thus enhancing user engagement and understanding of the data. Conclusions ShinySperm provides a robust platform for the dynamic exploration of epididymal sperm proteome data. It significantly improves accessibility and interpretability of complex datasets, allowing for effective data-driven insights. Implications This technical note highlights the potential of interactive web applications in reproductive biology and provides a plug and play script for the field to produce applications for meaningful researcher interaction with complex omic datasets.

Feasibility of in-home monitoring for people with glaucoma: the I-TRAC mixed-methods study.

Glaucoma is a chronic disease of the optic nerve and a leading cause of severe visual loss in the UK. Once patients have been diagnosed, they need regular monitoring at hospital eye services. Recent advances in technology mean patients with glaucoma can now monitor their disease at home. This could be more convenient for patients and potentially reduce costs and increase capacity for the NHS. However, it is uncertain whether self-monitoring would be acceptable or possible for patients with glaucoma. The objectives were to: identify which patients are most appropriate for home monitoring; understand views of key stakeholders (patients, clinicians, researchers) on whether home glaucoma monitoring is feasible and acceptable; develop a conceptual framework for the economic evaluation of home glaucoma monitoring; and explore the need for and provide evidence on the design of a future study to evaluate the clinical and cost-effectiveness of digital technologies for home monitoring of glaucoma. In-home Tracking of glaucoma: Reliability, Acceptability, and Cost (I-TRAC) was a multiphase mixed-methods feasibility study with key components informed by theoretical and conceptual frameworks. Expert glaucoma specialists in the UK recruited through professional glaucoma societies; study site staff and patient participants recruited through three UK hospital eye services (England, Scotland, Northern Ireland); and UK research teams recruited though existing networks. Home tonometer that measures intraocular pressure and a tablet computer with a visual function application. Patients were asked to use the technology weekly for 12 weeks. Forty-two patients were recruited. Retention and completion of follow-up procedures was successful, with 95% (n = 40) completing the 3-month follow-up clinic visits. Adherence to the interventions was generally high [adherence to both devices (i.e. ≥80% adherence) was 55%]. Overall, patients and healthcare professionals were cautiously optimistic about the acceptability of digital technologies for home monitoring of patients with glaucoma. While most clinicians were supportive of the potential advantages glaucoma home monitoring could offer, concerns about the technologies (e.g. reliability and potential to miss disease progression) and how they would fit into routine care need to be addressed. Additionally, clarity is required on defining the ideal population for this intervention. Plans for how to evaluate value for money in a future study were also identified. However, the study also highlighted several unknowns relating to core components of a future evaluative study that require addressing before progression to a definitive effectiveness trial. The main limitation relates to our sample and its generalisability, for example, the over-representation of educated persons of white ethnicity who were generally experienced with technology and research motivated. The In-home Tracking of glaucoma: Reliability, Acceptability, and Cost study has demonstrated 'cautious optimism' when considering patients' and healthcare professionals' views on the acceptability of digital technologies for home monitoring of patients with glaucoma. However, the study also highlighted several unknowns relating to the research question and design of a future evaluative study that require addressing before progression to a randomised controlled trial. Further research is required to determine the appropriate population (i.e. low vs. high risk of progression) and further refine the intervention components and delivery for planning of future evaluation studies. This study is registered as Research Registry #6213. This award was funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme (NIHR award ref: NIHR129248) and is published in full in Health Technology Assessment; Vol. 28, No. 44. See the NIHR Funding and Awards website for further award information.

Open-Ended Online Learning for Autonomous Visual Perception.

The visual perception systems aim to autonomously collect consecutive visual data and perceive the relevant information online like human beings. In comparison with the classical static visual systems focusing on fixed tasks (e.g., face recognition for visual surveillance), the real-world visual systems (e.g., the robot visual system) often need to handle unpredicted tasks and dynamically changed environments, which need to imitate human-like intelligence with open-ended online learning ability. Therefore, we provide a comprehensive analysis of open-ended online learning problems for autonomous visual perception in this survey. Based on "what to online learn" among visual perception scenarios, we classify the open-ended online learning methods into five categories: instance incremental learning to handle data attributes changing, feature evolution learning for incremental and decremental features with the feature dimension changed dynamically, class incremental learning and task incremental learning aiming at online adding new coming classes/tasks, and parallel and distributed learning for large-scale data to reveal the computational and storage advantages. We discuss the characteristic of each method and introduce several representative works as well. Finally, we introduce some representative visual perception applications to show the enhanced performance when using various open-ended online learning models, followed by a discussion of several future directions.

AMSVT: audio Mel-spectrogram vision transformer for spoken Arabic digit recognition

This work presents a novel model to recognize spoken digits in the Arabic language. Due to the transformer-based models' tremendous success in natural language processing (NLP), several attempts have been made to extend transformer-based designs to other domains, such as vision and audio. However, our approach consists of extracting and inputting Mel-spectrogram features into our model of the proposed audio Mel-spectrogram vision transformer (AMSVT) for training. The signal processing community has been interested in these models due to the successful use of vision transformers (ViT) in several computer vision applications. This is because signals are frequently recorded as spectrograms (using the Mel-spectrogram, for example), which may be given directly as input to vision transformers. Our model outperformed a group of models in terms of accuracy and time, such as convolutional neural network (CNN)-based and recurrent neural network (RNN)-based.

Promote computer vision applications in pig farming scenarios: high-quality dataset, fundamental models, and comparable performance1

Computer vision is widely recognized as an influential technology in the field of precision management of animals. Emerging studies have demonstrated the potential to improve pig health and welfare through animal surveillance systems and computer vision (CV) algorithms. However, the lack of benchmark datasets and robust fundamental algorithms restrict CV applications for the commercial use. This study aims to bridge the gap between technology development and commercial applications in pig farming scenarios by introducing a general-purpose dataset (PigLife), comparing benchmark performances of foundational CV algorithms and model development workflows. The PigLife dataset contains video clips and images (38 short video clips, 2K image frames, 22K pig instances) across most pig production phases in a typical commercial pig farm: Breeding and Gestation, Farrow to Wean, Weaning & Nursery, and Growth to Finish. Three detection algorithms (Faster R-CNN, RetinaNet, TridentNet) and three segmentation algorithms (Mask R-CNN, MViTv2, Point-Rend) were trained on the PigLife dataset from scratch. Fine-tuning of pre-trained models (YOLO8-m, Faster-RCNN-r50) and no-training from zero-shot models (CLIP-SAM, Grouddino-HQSAM) were also evaluated to suggest faster CV development workflows for commercial applications in pig farming. This study emphasizes the necessity of a benchmark dataset for evaluating the robustness of algorithms and identifying the remaining difficulties and challenges across various algorithms. Furthermore, developing CV models from pre-trained algorithms or zero-shot models showed better performance and a faster process, which could reduce barriers when developing high-performance CV products in pig production industry.

Performance analysis of deep unified model for facial expression recognition using convolution neural network

Facial expression recognition has gathered substantial attention in computer vision applications, with the need for robust and accurate models that can decipher human emotions from facial images. Performance analysis of a novel hybrid model combines the strengths of residual network (ResNet) and dense network (DenseNet) architectures after applying preprocessing for facial expression recognition. The proposed hybrid model capitalizes on the complementary characteristics of ResNet's and DenseNet's densely connected blocks to enhance the model's capacity to extract discriminative features from facial images. This research evaluates the hybrid model performance and conducts a comprehensive benchmark against established facial expression recognition convolution neural network (CNN) models. The analysis encompasses key aspects of model performance, including classification accuracy, and adaptability with the LFW dataset for facial expressions such as Anger, Fear, Happy, Disgust, Sad, Surprise, along Neutral. The research observes that the proposed hybrid model is more accurate and efficient computationally than existing models consistently. This performance analysis eliminates information on the hybrid model's perspective to further facial expression recognition research.

Stellar Evolution in Real Time. II. R Hydrae and an Open-Source Grid of >3000 Seismic TP-AGB Models Computed with MESA

Abstract We present a comprehensive characterization of the evolved thermally pulsing asymptotic giant branch (TP-AGB) star R Hydrae (R Hya), building on the techniques applied in Stellar Evolution in Real Time I (Molnár et al.) to T Ursae Minoris. We compute over 3000 theoretical TP-AGB pulse spectra using MESA and the corresponding oscillation spectra with GYRE. We combine these with classical observational constraints and nearly 400 years of measurements of R Hya’s period evolution to fit R Hya’s evolutionary and asteroseismic features. Two hypotheses for the mode driving R Hya’s period are considered. Solutions that identify this as the fundamental mode (FM) as well as the first overtone are consistent with observations. Using a variety of statistical tests, we find that R Hya is most likely driven by the FM and currently occupies the “power-down” phase of an intermediate pulse (TP ∼ 9–16). We predict that its pulsation period will continue to shorten for millennia. Supported by calculations from the Monash stellar evolution code, we find that R Hya has most likely undergone third dredge-up in its most recent pulse. The MESA + GYRE model grid used in this analysis includes exact solutions to the linear, adiabatic equations of stellar oscillation for the first 10 radial-order pressure modes for every time step in every evolutionary track. The grid is fully open source and packaged with a data visualization application. This is the first publicly available grid of TP-AGB models with seismology produced with MESA.

Evaluation of Brachial Plexus Nerve Variations in Axilla by Ultrasonography

The use of ultrasound in peripheral blocks has now become the gold standard. Ultrasound is a method that is easy to apply and most importantly does not carry any risk, and its only disadvantage is based on the skill and knowledge of the practitioner. Injury to vascular structures, which is the most common occurrence in peripheral block applications, has been significantly reduced by the use of ultrasound. The aim of this study is to determine the location of nerve branches and to determine the most common anthropometric parameters in the axillary fossa. In this way, the common anatomy of the axillary BP will be determined and will guide the practitioners while performing the axillary plexus block. Observational Clinical Study. The patients were positioned with forearm abducted 90 degrees and elbow flexed 90 degrees. A high-frequency linear ultrasound probe was placed on the lateral border of the pectoralis major muscle in the transverse plane. Pulsation of the axillary artery was visualized and shifted slowly to view the nerves around the artery. The axillary vein was also visualized to facilitate the movement of the transducer and to find the nerve localization more easily. The regions on the prepared wheel were marked. At the same time, demographic information such as gender, age, weight, and height of the patients were also recorded. The sample was 248 patients, 61.3% female and 38.7% male. Our results showed that only 59% were compatible with the most common nerve locations in cadaveric dissections and the locations described in anatomy textbooks. Since there are many anatomical variations, validation of nerves with a nerve stimulator as well as simultaneous visual application under ultrasound guidance will increase the success chance of axillary brachial plexus block and protect it from unwanted complications.

Application of 3-dimensional visualization and image fusion technology in liver cancer with portal vein tumor thrombus surgery.

Liver cancer with portal vein tumor thrombus (PVTT) is a frequent finding and is related to poor prognosis. Surgical resection provides a more promising prognosis in selected patients. The purpose of this study was to explore the application of 3D (3-dimensional) visualization and image fusion technology in liver cancer with PVTT surgery. 12 patients were treated with surgery between March 2019 and August 2022. The preoperative standard liver volume (SLV), estimated future liver remnant (FLR), FLR/SLV, 3D visualization models, PVTT classification, operation programs, surgical results, and prognosis were collected and analyzed. Twelve patients who had complete data of 3D visualization and underwent hemihepatectomy combined with portal vein tumor thrombectomy. The operation plan was formulated by 3D visualization and was highly consistent with the actual surgery. The SLV was 1208.33 ± 63.22 mL, FLR was 734.00 mL and FLR/SLV was 61.62 ± 19.38%. The accuracy of classification of PVTT by 3D visualization was 100%, Cheng type Ⅱa (4 cases), Ⅱb (2 cases), Ⅲa (4 cases), and Ⅲb (2 cases). The 3D visualization model was a perfect fusion with the intraoperative live scene and precise guidance for hepatectomy. No patient was suffering from postoperative liver failure and without procedure‑associated death. 6 patients died of tumor recurrence, and 2 patients died of other reasons. The 12-month cumulative survival rate was 25.9%. 3D visualization and image fusion technology could be used for precise assessment of FLR, classification of PVTT, surgery navigation, and which was helpful in improving the safety of hepatectomy.

Symmetry group detection of point clouds in 3D via a decomposition method

Analyzing the symmetries present in point clouds, which represent sets of 3D coordinates, is important for understanding their underlying structure and facilitating various applications. In this paper, we propose a novel decomposition-based method for detecting the entire symmetry group of 3D point clouds. Our approach decomposes the point cloud into simpler shapes whose symmetry groups are easier to find. The exact symmetry group of the original point cloud is then derived from the symmetries of these individual components. The method presented in this paper is a direct extension of the approach recently formulated in Bizzarri et al. (2022a) for discrete curves in plane. The method can be easily modified also for perturbed data. This work contributes to the advancement of symmetry analysis in point clouds, providing a foundation for further research and enhancing applications in computer vision, robotics, and augmented reality.

An energy transfer strategy towards versatile sensing applications from Cr3+ and Yb3+ codoped near infrared phosphors

Near-infrared (NIR) phosphors have garnered notable attention for their extensive applications in anti-counterfeiting, optical imaging, non-invasive medical diagnosis and detection. Nevertheless, the quest for NIR phosphors with outstanding detection and sensing capabilities remains a challenge. In this study, we have successfully prepared Ga1.6Mg0.2Ge0.2O3:Cr3+,Yb3+ NIR phosphors by codoping and substitutation strategies. When incorporated into polydimethylsiloxane films, these phosphors demonstrated a direct correlation between luminescence intensity ratio and applied stress under 450 nm excitation. Moreover, variations in luminescence spectra occurred with different compositions and concentrations of liquids coated on the film, facilitating both force measurement and composition analysis. Furthermore, a NIR phosphor-converted light-emitting diode (pc-LED) was fabricated by integrating phosphor with a blue LED chip, exhibiting excellent performance. Utilizing the prepared NIR pc-LED as a light source, we facilitated non-destructive imaging, night vision, and anti-counterfeiting applications with an infrared camera. These findings suggest that the synthesized NIR phosphors possess hold promise for diverse detection and sensing applications.

Electronic Visualization Laboratory's 50th Anniversary Retrospective: Look to the Future, Build on the Past

Abstract September 2023 marks the 50th anniversary of the Electronic Visualization Laboratory (EVL) at University of Illinois Chicago (UIC). EVL's introduction of the CAVE Automatic Virtual Environment in 1992, the first widely replicated, projection-based, walk-in, virtual-reality (VR) system in the world, put EVL at the forefront of collaborative, immersive data exploration and analytics. However, the journey did not begin then. Since its founding in 1973, EVL has been developing tools and techniques for real-time, interactive visualizations—pillars of VR. But EVL's culture is also relevant to its successes, as it has always been an interdisciplinary lab that fosters teamwork, where each person's expertise contributes to the development of the necessary tools, hardware, system software, applications, and human interface models to solve problems. Over the years, as multidisciplinary collaborations evolved and advanced scientific instruments and data resources were distributed globally, the need to access and share data and visualizations while working with colleagues, local and remote, synchronous and asynchronous, also became important fields of study. This paper is a retrospective of EVL's past 50 years that surveys the many networked, immersive, collaborative visualization and VR systems and applications it developed and deployed, as well as lessons learned and future plans.

Trame: an Open Source Framework for Efficiently Building Interactive Visualization and Analysis Applications

trame: an Open Source Framework for Efficiently Building Interactive Visualization and Analysis Applications

Game Engines for Immersive Visualization: Using Unreal Engine Beyond Entertainment

Abstract One core aspect of immersive visualization labs is to develop and provide powerful tools and applications that allow for efficient analysis and exploration of scientific data. As the requirements for such applications are often diverse and complex, the same applies to the development process. This has led to a myriad of different tools, frameworks, and approaches that grew and developed over time. The steady advance of commercial off-the-shelf game engines such as Unreal Engine has made them a valuable option for development in immersive visualization labs. In this work, we share our experience of migrating to Unreal Engine as a primary developing environment for immersive visualization applications. We share our considerations on requirements, present use cases developed in our lab to communicate advantages and challenges experienced, discuss implications on our research and development environments, and aim to provide guidance for others within our community facing similar challenges.

TTFDNet: Precise Depth Estimation from Single-Frame Fringe Patterns.

This work presents TTFDNet, a transformer-based and transfer learning network for end-to-end depth estimation from single-frame fringe patterns in fringe projection profilometry. TTFDNet features a precise contour and coarse depth (PCCD) pre-processor, a global multi-dimensional fusion (GMDF) module and a progressive depth extractor (PDE). It utilizes transfer learning through fringe structure consistency evaluation (FSCE) to leverage the transformer's benefits even on a small dataset. Tested on 208 scenes, the model achieved a mean absolute error (MAE) of 0.00372 mm, outperforming Unet (0.03458 mm) models, PDE (0.01063 mm) and PCTNet (0.00518 mm). It demonstrated precise measurement capabilities with deviations of ~90 μm for a 25.4 mm radius ball and ~6 μm for a 20 mm thick metal part. Additionally, TTFDNet showed excellent generalization and robustness in dynamic reconstruction and varied imaging conditions, making it appropriate for practical applications in manufacturing, automation and computer vision.

Applications of RIM-Based Flow Visualization in Fluid-Solid Interaction Problems: A Review of Formulations and Prospects

Over the past three decades, optical visualization measurements based on the Refractive Index Match (RIM) method have played a significant role in the experimental studies of fluid-solid interaction. The RIM method, which coordinates the refractive indices of the liquid and solid materials in the experiment, dramatically reduces the observation error due to optical refraction. However, the existing literature on RIM has not systematically reviewed the various applications of this technique. This review aims to fill this gap by providing a comprehensive overview of the RIM technique, examining its role in material selection for fluid-solid interaction studies, and scrutinizing its applications across various engineering disciplines. The paper begins with a brief introduction to the RIM technique and then turns to material selection and its various applications in fluid-solid interaction. It also enumerates and analyzes specific RIM-based optical measurement techniques such as Laser Doppler Velocimetry (LDA), Particle Tracking Velocimetry (PTV), and Particle Image Velocimetry (PIV) from various research perspectives in previous studies. In addition, it summarizes RIM formulations categorized by different applications in liquid-solid interaction fields. RIM-based measurement techniques generally offer intuitive, non-intrusive, cost-effective, and convenient advantages over traditional methods. The paper also critically evaluates the strengths and limitations of different materials used in RIM experiments and suggests directions for future research, emphasizing the need to develop environmentally friendly and cost-effective RIM materials.

Color analysis of ancient Egyptian paintings and its applications in modern digital visualization

Color analysis of ancient Egyptian paintings and its applications in modern digital visualization

Identification of Glaucoma from Retinal Fundus Images using Deep Learning Model, MobileNet

Glaucoma is one of the leading causes of permanent vision impairment and blindness everywhere in the world due to high intraocular stress inside the eyes. So, early and accurate detection is crucial for preventing irreversible vision loss. Manual recognition of Glaucoma is a difficult task that requires proficiency and a highly experienced person. Computer Aided Detection (CAD) techniques assist ophthalmologists in the detection of such ophthalmologic diseases by analyzing the retinal fundus images. Deep learning(DL) algorithms have accomplished exceptionally in various computer vision applications by giving them an apparent advantage over more traditional techniques by analyzing the retinal fundus images. In this paper, a pre-trained Convolutional Neural Network, MobileNet, is used to detect Glaucoma in retinal fundus images by automatically extracting the features rather than manually classifying fundus images. Due to efficiency, speed of execution, accuracy, customizability, transfer learning, real-time applications and size of the architecture, MobileNet is proposed here. The DrishtiGS, EyePACS AIROGS-Light, BEH, REFUGE, sjchoi86- HRF, CRFO-v4, G1020, FIVES, and PAPILA datasets were utilized in this study. The suggested model is evaluated by significant factors such as accuracy, precision, sensitivity, specificity, F1 score, and Confusion Matrix to examine the model's efficacy. These studies reveal the capability of the DL approach to classify Glaucoma from retinal fundus images and recommend that the suggested approach can assist ophthalmologists in a quick, correct, and dependable diagnosis of Glaucoma.

Implicit calibration method for underwater stereo cameras.

Underwater stereo cameras can effectively capture intricate environments with restricted accessibility, offering an appealing solution for precise perception. Stereo imaging is however susceptible to distortions caused by the refraction of incoming rays. These distortions are nonlinear and challenge the standard single viewpoint projection assumption. In this paper, we propose a data-driven implicit calibration method for underwater stereo cameras. To address the imaging characteristics and aberration distributions across different coordinates of underwater stereo cameras, we have developed the corresponding coordinates regression network and fusion strategy, thereby converting the calibration process into network-based learning. Secondly, we designed an underwater self-luminous calibration target system and the underwater corner point extraction strategy for sample dataset acquisition. We evaluated the proposed method comprehensively in terms of measurement, camera posture estimation, and 3D reconstruction, and compared it with other explicit calibration methods. The experimental results show that the proposed implicit calibration method is superior to other explicit calibration. We demonstrate with real experiments that our method enables efficient camera calibration for underwater vision applications.