Input Image Research Articles

Enhanced Cross-stage-attention U-Net for esophageal target volume segmentation

PurposeThe segmentation of target volume and organs at risk (OAR) was a significant part of radiotherapy. Specifically, determining the location and scale of the esophagus in simulated computed tomography images was difficult and time-consuming primarily due to its complex structure and low contrast with the surrounding tissues. In this study, an Enhanced Cross-stage-attention U-Net was proposed to solve the segmentation problem for the esophageal gross tumor volume (GTV) and clinical tumor volume (CTV) in CT images.MethodsFirst, a module based on principal component analysis theory was constructed to pre-extract the features of the input image. Then, a cross-stage based feature fusion model was designed to replace the skip concatenation of original UNet, which was composed of Wide Range Attention unit, Small-kernel Local Attention unit, and Inverted Bottleneck unit. WRA was employed to capture global attention, whose large convolution kernel was further decomposed to simplify the calculation. SLA was used to complement the local attention to WRA. IBN was structed to fuse the extracted features, where a global frequency response layer was built to redistribute the frequency response of the fused feature maps.ResultsThe proposed method was compared with relevant published esophageal segmentation methods. The prediction of the proposed network was MSD = 2.83(1.62, 4.76)mm, HD = 11.79 ± 6.02 mm, DC = 72.45 ± 19.18% in GTV; MSD = 5.26(2.18, 8.82)mm, HD = 16.22 ± 10.01 mm, DC = 71.06 ± 17.72% in CTV.ConclusionThe reconstruction of the skip concatenation in UNet showed an improvement of performance for esophageal segmentation. The results showed the proposed network had better effect on esophageal GTV and CTV segmentation.

Cross-Field Road Markings Detection Based on Inverse Perspective Mapping

With the rapid development of the autonomous vehicles industry, there has been a dramatic proliferation of research concerned with related works, where road markings detection is an important issue. When there is no public open data in a field, we must collect road markings data and label them by ourselves manually, which is huge labor work and takes lots of time. Moreover, object detection often encounters the problem of small object detection. The detection accuracy often decreases when the detection distance increases. This is primarily because distant objects on the road take up few pixels in the image and object scales vary depending on different distances and perspectives. For the sake of solving the issues mentioned above, this paper utilizes a virtual dataset and open dataset to train the object detection model and cross-field testing in the field of Taiwan roads. In order to make the model more robust and stable, the data augmentation method is employed to generate more data. Therefore, the data are increased through the data augmentation method and homography transformation of images in the limited dataset. Additionally, Inverse Perspective Mapping is performed on the input images to transform them into the bird’s eye view, which solves the “small objects at far distance” problem and the “perspective distortion of objects” problem so that the model can clearly recognize the objects on the road. The model testing on the front-view images and bird’s eye view images also shows a remarkable improvement of accuracy by 18.62%.

Agent Enhancement using Deep Reinforcement Learning Algorithms for Multiplayer game (Slither.io)

Developing self learning model for a game is challenging as the environment keeps changing all the time and therefore require highly intelligent models which can make decisions depending on the environment in real time. The agent has to learn the environment and takes action based on the inference. Based on the action, a positive or negative reward is given to the agent. The agent again learns from the reward and enhances / trains itself to behave better in the environment. This work aims to train an agent using deep reinforcement learning algorithms to play a multiplayer online game like SLITHER.IO. We use an OpenAI Universe environment to collect raw image inputs from sample gaming as training data. Agent learns the current state of the environment and the position of the other players (snakes). Then it takes action in the form of direction of its movement. To compare our model to other existing systems and random policy, we propose to use deep Q-learning and other actor critic approaches such as Proximal Policy Optimisation (PPO) with reward shaping and replay buffer. Out of all these algorithms the PPO agent shows significant improvement in the score over a range of episodes. PPO agent learns quickly and its reward progression is higher when compared to other techniques.

UNIFIED MULTIMODAL BIOMETRICS FUSION USING DEEP LEARNING FOR SECURING IOT

Advancements in multimodal biometrics, which amalgamate multiple biometric traits, hold promise for augmenting the accuracy and robustness of biometric identification systems. The focal point of this innovative study is the enhancement of multimodal biometrics identification, using face and iris images as the key biometric traits. This work taps into the expansive collection of face and iris images present in the WVU-Multimodal dataset for evaluation purposes. Our proposed approach employs “Convolutional Neural Network (CNN)” architectures, notable for their efficacy in computer vision tasks, to extract potent discriminative features from the input images. This work specifically incorporates three popular CNN architectures: ResNet-50, InceptionNet, XceptionNet, and fine-tuned CNN. To amalgamate the extracted features, investigate various fusion techniques in the security-centric industry: early fusion, and score-level fusion. Early fusion is an approach that merges the raw images of both face and iris at the input level to a single CNN model. Use the Gabor approach to enhance the image's quality and make the face and iris information more visible. This technique modifies the histogram equalization process for local regions, thus enabling better visibility and subsequent feature extraction. Our experimental evaluation employs performance metrics like accuracy, “Equal Error Rate”, and “Receiver Operating Characteristic” curves. In this work undertakes a comparative analysis to appraise the performance of the different CNN architectures and fusion techniques under scrutiny.

LesionScanNet: dual-path convolutional neural network for acute appendicitis diagnosis.

Acute appendicitis is an abrupt inflammation of the appendix, which causes symptoms such as abdominal pain, vomiting, and fever. Computed tomography (CT) is a useful tool in accurate diagnosis of acute appendicitis; however, it causes challenges due to factors such as the anatomical structure of the colon and localization of the appendix in CT images. In this paper, a novel Convolutional Neural Network model, namely, LesionScanNet for the computer-aided detection of acute appendicitis has been proposed. For this purpose, a dataset of 2400 CT scan images was collected by the Department of General Surgery at Kanuni Sultan Süleyman Research and Training Hospital, Istanbul, Turkey. LesionScanNet is a lightweight model with 765K parameters and includes multiple DualKernel blocks, where each block contains a convolution, expansion, separable convolution layers, and skip connections. The DualKernel blocks work with two paths of input image processing, one of which uses 3 × 3 filters, and the other path encompasses 1 × 1 filters. It has been demonstrated that the LesionScanNet model has an accuracy score of 99% on the test set, a value that is greater than the performance of the benchmark deep learning models. In addition, the generalization ability of the LesionScanNet model has been demonstrated on a chest X-ray image dataset for pneumonia and COVID-19 detection. In conclusion, LesionScanNet is a lightweight and robust network achieving superior performance with smaller number of parameters and its usage can be extended to other medical application domains.

Exploring Image Decolorization: Methods, Implementations, and Performance Assessment

Decolorization is an image processing technique that converts a color input image into a grayscale image. This paper discusses the decolorization process and provides an overview of the methods based on the different principles used: basic conversion from RGB to YUV format using ITU Recommendations 601, 709, and 2020; basic conversion from RGB to LAB color space; the method using cumulative distribution function of color channels; one global decolorization method; and one based on deep learning. The grayscale images produced by these methods were evaluated using four objective metrics, allowing for a thorough analysis and comparison of the decolorization results. Additionally, the execution speed of the algorithms was assessed, providing insight into their performance efficiency. The results demonstrate that different metrics evaluate the decolorization methods differently, highlighting the importance of selecting an appropriate metric that aligns with the subsequent image processing tasks following decolorization. Furthermore, it was shown that the decolorization methods depend on the content of the images, performing better on natural images than on artificially generated ones. The decolorization methods were also examined in the context of object segmentation and edge detection. The results from segmentation and edge detection were aligned with the decolorization results, revealing that certain objective metrics for evaluating decolorization more effectively assessed the properties of the decolorized images, which are crucial for successful object segmentation and edge detection.

Optimized dual-tree complex wavelet transform aided multimodal image fusion with adaptive weighted average fusion strategy

Image fusion is generally utilized for retrieving significant data from a set of input images to provide useful informative data. Image fusion enhances the applicability and quality of data. Hence, the analysis of multimodal image fusion is a new to the research topic, which is designed by combining the images of multimodal into single image in order to preserveexact details. On the other hand, the existing approaches face challenges in the precise interpretation of source images, and also it have only captured local information without considering the wide range of information. To consider these weaknesses, a multimodal image fusion model is planned to develop according to the multi-resolution transform along with the optimization strategy. At first, the images are effectively analyzed from standard public datasets and further, the images given into the Optimized Dual-Tree Complex Wavelet Transform (ODTCWT) to acquire low frequency and high frequency coefficients. Here, certain parameters in DTCWT get tuned with the hybridized heuristic strategy with the Probability of Fitness-based Honey Badger Squirrel Search Optimization (PF-HBSSO) to enhance the decomposition quality. Then, the fusion of high-frequency coefficients is performed using adaptive weighted average fusion technique, whereas the weights are optimized using PF-HBSSOto achieve the optimal fused results. Similarly, the low-frequency coefficients are combined by average fusion. Finally, the fused images undergo image reconstruction using the inverse ODTCWT. The experimental evaluation of the designed multimodal image fusion illustratessuperioritythat distinguishes this work from others.

An Unbiased Feature Estimation Network for Few-Shot Fine-Grained Image Classification.

Few-shot fine-grained image classification (FSFGIC) aims to classify subspecies with similar appearances under conditions of very limited data. In this paper, we observe an interesting phenomenon: different types of image data augmentation techniques have varying effects on the performance of FSFGIC methods. This indicates that there may be biases in the features extracted from the input images. The bias of the acquired feature may cause deviation in the calculation of similarity, which is particularly detrimental to FSFGIC tasks characterized by low inter-class variation and high intra-class variation, thus affecting the classification accuracy. To address the problems mentioned, we propose an unbiased feature estimation network. The designed network has the capability to significantly optimize the quality of the obtained feature representations and effectively reduce the feature bias from input images. Furthermore, our proposed architecture can be easily integrated into any contextual training mechanism. Extensive experiments on the FSFGIC tasks demonstrate the effectiveness of the proposed algorithm, showing a notable improvement in classification accuracy.

Multimodal Contrastive Learning for Remote Sensing Image Feature Extraction Based on Relaxed Positive Samples

Traditional multimodal contrastive learning brings text and its corresponding image closer together as a positive pair, where the text typically consists of fixed sentence structures or specific descriptive statements, and the image features are generally global features (with some fine-grained work using local features). Similar to unimodal self-supervised contrastive learning, this approach can be seen as enforcing a strict identity constraint in a multimodal context. However, due to the inherent complexity of remote sensing images, which cannot be easily described in a single sentence, and the fact that remote sensing images contain rich ancillary information beyond just object features, this strict identity constraint may be insufficient. To fully leverage the characteristics of remote sensing images, we propose a multimodal contrastive learning method for remote sensing image feature extraction, based on positive sample tripartite relaxation, where the model is relaxed in three aspects. The first aspect of relaxation involves both the text and image inputs. By introducing learnable parameters in the language and image branches, instead of relying on fixed sentence structures and fixed image features, the network can achieve a more flexible description of remote sensing images in text and extract ancillary information from the image features, thereby relaxing the input constraints. Second relaxation is achieved through multimodal alignment of various features. By aligning semantic information with the corresponding semantic regions in the images, the method allows for the relaxation of local image features under semantic constraints. This approach addresses the issue of selecting image patches in unimodal settings, where there is no semantic constraint. The proposed method for remote sensing image feature extraction has been validated on four datasets. On the PatternNet dataset, it achieved a 91.1% accuracy with just one-shot.

Large Language Models with Vision on Diagnostic Radiology Board Exam Style Questions

Rationale and ObjectivesThe expansion of large language models to process images offers new avenues for application in radiology. This study aims to assess the multimodal capabilities of contemporary large language models, which allow analysis of image inputs in addition to textual data, on radiology board style examination questions with images. Materials and Methods280 questions were retrospectively selected from the AuntMinnie public test bank. The test questions were converted into three formats of prompts; 1) Multimodal, 2) Image only, and 3) Text only input. Three models, GPT-4V, Gemini 1.5 Pro, and Claude 3.5 Sonnet, were evaluated using these prompts. The Cochran Q test and pairwise McNemar test were used to compare performances between prompt formats and models. ResultsNo difference was found for the performance in terms of % correct answers between the text, image, and multimodal prompt formats for GPT-4V (54%, 52%, and 57%, respectively; p=.31) and Gemini 1.5 Pro (53%, 54%, and 57%, respectively; p=.53). For Claude 3.5 Sonnet, the image input (48%) significantly underperformed compared to the text input (63%, p<.001) and the multimodal input (66%, p<.001), but no difference was found between the text and multimodal inputs (p=.29). Claude significantly outperformed GPT and Gemini in the text and multimodal formats (p<.01). ConclusionVision capable large language models cannot effectively use images to increase performance on radiology board-style examination questions. When using textual data alone, Claude 3.5 Sonnet outperforms GPT-4V and Gemini 1.5 Pro, highlighting the advancements in the field and its potential for the use in further research.

Ground-Based Cloud Image Segmentation Method Based on Improved U-Net

Cloud image segmentation is a technique that divides images captured by meteorological satellites or ground-based observations into different regions or categories. By extracting the distribution, shape, and dynamic features of clouds, it provides precise data support for the meteorological and environmental fields, significantly influencing photovoltaic (PV) power generation forecasting, astronomical telescope observatory site selection, and weather forecasting. A ground-based cloud image segmentation model based on an improved U-Net is proposed, which adopts an overall encoder–decoder structure. In the encoder phase, this paper constructs a dilated convolution–atrous spatial pyramid pooling (ASPP)–dilated convolution structure to enhance early cloud feature extraction. Dilated convolution is a novel type of convolution that expands the receptive field by inserting holes into standard convolution, thereby capturing a larger range of contextual information. ASPP maintains high resolution while paying attention to both local details and global structures of the image. In the decoder stage, the bicubic interpolation method is used for up-sampling to restore the feature map resolution and improve the clarity of the segmented image. The bicubic interpolation method refers to the use of cubic polynomial functions to interpolate the pixel values of the input image. In addition, this paper designs a novel skip connection layer structure between the encoder and decoder, composed of a depthwise separable path (DS path) and an improved channel spatial attention module (Im-CSAM) connected in sequence. The DS path combines depthwise separable convolutions and residual structures to facilitate information exchange between high-level and low-level features. The Im-CSAM is a modular attention mechanism that focuses on important cloud features in spatial and channel dimensions to enhance segmentation accuracy. Experiments show that compared to the traditional U-Net, the accuracy, precision, and MIoU of this model improved by 2.2%, 4.1%, and 5.0%, respectively, in the SWINySEG dataset, and by 3.2%, 3.6%, and 5.8%, respectively, in the TCDD dataset, proving that the improved method has a better generalization ability and segmentation performance.

DAC: Differentiable Auto-Cropping in Deep Learning

Auto-cropping, the process of automatically adjusting the boundaries of an image to focus on the region of interest, is crucial to improving the diagnostic quality of dental panoramic radiographs. Its importance lies in its ability to standardize the size of different input images with minimal loss of information, thus ensuring consistency and improving the performance of subsequent image-processing tasks. Despite the widespread use of CNNs in many studies, research on auto-cropping for different-sized images remains limited. This study aims to explore the potential of differentiable auto-cropping in dental panoramic radiographs. A unique dataset of 20,973 dental panoramic radiographs, mostly with a resolution of 2836×1536 or close, divided into five classes by 3 dentists, was used, which is the same dataset from the previous study (Top et al. 2023). ResNet-101 model, which was the most successful network for the dataset (Top et al. 2023), was used for the evaluation. To reduce variance, the model was evaluated using 10-fold cross-validation for both non-auto-cropped and auto-cropped trainings. Data augmentation was also used to produce more accurate and robust results. For auto-cropped training, it was adjusted to be much less effective than the non-auto-cropped one. Accuracy was improved by 1.8%, from 92.7% to 94.5%, thanks to the proposed auto-crop optimization developed to reduce dataset-related issues. Its macro-average AUC was also raised from 0.989 to 0.993. The proposed auto-crop optimization can be implemented as a trainable network layer in an end-to-end CNN and can be used for other problems as well. Increasing the accuracy from 92.7% to 94.5% is a very challenging task due to diminishing returns, as there is little room for improvement. The results show the potential of the proposed differentiable auto-crop algorithm and encourages its use in different fields.

Hybrid deep learning-based skin cancer classification with RPO-SegNet for skin lesion segmentation

ABSTRACT Skin melanin lesions are typically identified as tiny patches on the skin, which are impacted by melanocyte cell overgrowth. The number of people with skin cancer is increasing worldwide. Accurate and timely skin cancer identification is critical to reduce the mortality rates. An incorrect diagnosis can be fatal to the patient. To tackle these issues, this article proposes the Recurrent Prototypical Object Segmentation Network (RPO-SegNet) for the segmentation of skin lesions and a hybrid Deep Learning (DL) – based skin cancer classification. The RPO-SegNet is formed by integrating the Recurrent Prototypical Networks (RP-Net), and Object Segmentation Networks (O-SegNet). At first, the input image is taken from a database and forwarded to image pre-processing. Then, the segmentation of skin lesions is accomplished using the proposed RPO-SegNet. After the segmentation, feature extraction is accomplished. Finally, skin cancer classification and detection are accomplished by employing the Fuzzy-based Shepard Convolutional Maxout Network (FSCMN) by combining the Deep Maxout Network (DMN), and Shepard Convolutional Neural Network (ShCNN). The established RPO-SegNet+FSCMN attained improved accuracy, True Negative Rate (TNR), True Positive Rate (TPR), dice coefficient, Jaccard coefficient, and segmentation analysis of 91.985%, 92.735%, 93.485%, 90.902%, 90.164%, and 91.734%.

An optimized bidirectional vision transformer based colorectal cancer detection using histopathological images

The high death rate of colorectal cancer has a long-term impact on human life around the world. Early detection of colorectal cancer leads to an increase in the survival rate of the patients. Overfitting issues occurred due to the presence of imbalanced datasets. To resolve the challenges, a novel deep learning-based approach based on an effective optimization strategy with a bidirectional vision transformer model for colorectal cancer detection. Disease classification is performed using two datasets: colorectal histology images and the NCT-CRC-HE-100 K dataset. Initially, the histopathological images from the dataset are trained and pre-processed using a Trimmed Pixel density-based median filter (TPDMF), which removes noise from the input images and enhances the quality of the images. The features of the pre-processed image are extracted using the Capsule Assisted Res2Net (CAR2N) model, and an optimized Bidirectional Vision Transformer (OBVT) model is presented to identify colorectal cancer. Here, the Bi- Long short-term memory (Bi-LSTM) model is used in the Multi-layer perception module of the vision transformer to reduce the complexity of the detection process. The proposed methodology uses a chaotic sequence-based Snake optimization algorithm (Ch-SOA) to tune the hyperparameters in the proposed model. The proposed model can be analyzed using different performance metrics like accuracy, precision, sensitivity, specificity, False negative rate (FNR), and False positive rate (FPR). The proposed model can obtain an accuracy of 97.11458 % for the colorectal histology images dataset and 97.01235 % for the NCT-CRC-HE-100 K dataset. From this analysis, the proposed model can obtain better results than other existing models.

Designing a novel framework of email spam detection using an improved heuristic algorithm and dual-scale feature fusion-based adaptive convolution neural network

ABSTRACT Nowadays, e-mail is becoming more prevalent among individuals. In recent days, it has been declared to be the least expensive and quickest mode of communicating. In recent years, e-mail spam has become a big problem, so the number of e-mail spam has also increased, and they are used for unethical and illegal conduct, scams, and theft. As a result, the main objective of the study is to detect e-mail spam using an adaptive deep-learning model with feature fusion. Initially, the input image and text are collected from benchmark datasets. Further, the raw text undergoes the pre-processing stage, and it is then followed by extracting the features using Bidirectional Encoder Representations from Transformers (BERT). Similarly, the Vision Transformer (ViT) is employed for extracting the features from raw images. Finally, these features are given in Dual Scale Feature Fusion-based Adaptive Convolutional Neural Network (DFF-ACNNet), and it is then fused for performing the classification in Convolutional Neural Network (CNN). Here, the hyper-parameters are tuned by using Modified Squid Game Optimizer (MSGO). Finally, the evaluation is done with various metrics to estimate the performance of the model and the accuracy of the developed system is 98.46, which is higher than the other conventional approaches.

Online Multi-object Tracking with YOLOv9 and DeepSORT Optimized by Optical Flow

To ensure reliable environmental perception in the realm of autonomous driving, precise and robust multi-object tracking proves imperative. This study proposes an innovative approach to multi-object tracking by combining YOLOv9's sophisticated detection capabilities with an enhanced DeepSORT tracking algorithm, enriched through the integration of optical flow. In the proposed method, the YOLOv9 detector acutely identifies objects in input images, and these detected entities are subsequently transmitted to the optimized DeepSORT tracking algorithm. The principal contribution of this study lies in improving the Kalman filter measurement model within DeepSORT by incorporating robust local optical flow, thus adding a velocity dimension to the filter's update vector. This novel approach significantly improves tracking resilience in the face of occlusions, rapid movements, and appearance changes. Evaluations on MOT17 and KITTI show substantial improvement gains of 2.42%, 2.85%, and 1.84% for HOTA, MOTA, and IDF1, respectively, on MOT17, and 1.94% in MOTA and 2.09% in HOTA on KITTI. The proposed method particularly excels in managing scenarios involving dense traffic and light variations, which are recurrent problems in dynamic urban environments. This enhanced performance positions the proposed solution as an essential component of future perception architectures for autonomous vehicles, promising safer and more efficient navigation in the complex real world.

Enabling Perspective-Aware Ai with Contextual Scene Graph Generation

This paper advances contextual image understanding within perspective-aware Ai (PAi), an emerging paradigm in human–computer interaction that enables users to perceive and interact through each other’s perspectives. While PAi relies on multimodal data—such as text, audio, and images—challenges in data collection, alignment, and privacy have led us to focus on enabling the contextual understanding of images. To achieve this, we developed perspective-aware scene graph generation with LLM post-processing (PASGG-LM). This framework extends traditional scene graph generation (SGG) by incorporating large language models (LLMs) to enhance contextual understanding. PASGG-LM integrates classical scene graph outputs with LLM post-processing to infer richer contextual information, such as emotions, activities, and social contexts. To test PASGG-LM, we introduce the context-aware scene graph generation task, where the goal is to generate a context-aware situation graph describing the input image. We evaluated PASGG-LM pipelines using state-of-the-art SGG models, including Motifs, Motifs-TDE, and RelTR, and showed that fine-tuning LLMs, particularly GPT-4o-mini and Llama-3.1-8B, improves performance in terms of R@K, mR@K, and mAP. Our method is capable of generating scene graphs that capture complex contextual aspects, advancing human–machine interaction by enhancing the representation of diverse perspectives. Future directions include refining contextual scene graph models and expanding multi-modal data integration for PAi applications in domains such as healthcare, education, and social robotics.

Comparing Large Language Model and Human Reader Accuracy with New England Journal of Medicine Image Challenge Case Image Inputs.

Background Application of multimodal large language models (LLMs) with both textual and visual capabilities has been steadily increasing, but their ability to interpret radiologic images is still doubted. Purpose To evaluate the accuracy of LLMs and compare it with that of human readers with varying levels of experience and to assess the factors affecting LLM accuracy in answering New England Journal of Medicine Image Challenge cases. Materials and Methods Radiologic images of cases from October 13, 2005, to April 18, 2024, were retrospectively reviewed. Using text and image inputs, LLMs (Open AI's GPT-4 Turbo with Vision [GPT-4V] and GPT-4 Omni [GPT-4o], Google's DeepMind Gemini 1.5 Pro, and Anthropic's Claude 3) provided answers. Human readers (seven junior faculty radiologists, two clinicians, one in-training radiologist, and one medical student), blinded to the published answers, also answered. LLM accuracy with and without image inputs and short (cases from 2005 to 2015) versus long text inputs (from 2016 to 2024) was evaluated in subgroup analysis to determine the effect of these factors. Factor analysis was assessed using multivariable logistic regression. Accuracy was compared with generalized estimating equations, with multiple comparisons adjusted by using Bonferroni correction. Results A total of 272 cases were included. GPT-4o achieved the highest overall accuracy among LLMs (59.6%; 162 of 272), outperforming a medical student (47.1%; 128 of 272; P < .001) but not junior faculty (80.9%; 220 of 272; P < .001) or the in-training radiologist (70.2%; 191 of 272; P = .003). GPT-4o exhibited similar accuracy regardless of image inputs (without images vs with images, 54.0% [147 of 272] vs 59.6% [162 of 272], respectively; P = .59). Human reader accuracy was unaffected by text length, whereas LLMs demonstrated higher accuracy with long text inputs (all P < .001). Text input length affected LLM accuracy (odds ratio range, 3.2 [95% CI: 1.9, 5.5] to 6.6 [95% CI: 3.7, 12.0]). Conclusion LLMs demonstrated substantial accuracy with text and image inputs, outperforming a medical student. However, their accuracy decreased with shorter text lengths, regardless of image input. © RSNA, 2024 Supplemental material is available for this article.

R 3LIVE++: A Robust, Real-Time, Radiance Reconstruction Package With a Tightly-Coupled LiDAR-Inertial-Visual State Estimator.

This work proposed a LiDAR-inertial-visual fusion framework termed R 3LIVE++ to achieve robust and accurate state estimation while simultaneously reconstructing the radiance map on the fly. R 3LIVE++ consists of a LiDAR-inertial odometry (LIO) and a visual-inertial odometry (VIO), both running in real-time. The LIO subsystem utilizes the measurements from a LiDAR for reconstructing the geometric structure, while the VIO subsystem simultaneously recovers the radiance information of the geometric structure from the input images. R 3LIVE++ is developed based on R 3LIVE and further improves the accuracy in localization and mapping by accounting for the camera photometric calibration and the online estimation of camera exposure time. We conduct more extensive experiments on public and self-collected datasets to compare our proposed system against other state-of-the-art SLAM systems. Quantitative and qualitative results show that R 3LIVE++ has significant improvements over others in both accuracy and robustness. Moreover, to demonstrate the extendability of R 3LIVE++, we developed several applications based on our reconstructed maps, such as high dynamic range (HDR) imaging, virtual environment exploration, and 3D video gaming.

Deep indoor illumination estimation based on spherical gaussian representation with scene prior knowledge

High dynamic range illumination estimation from a single low dynamic range image is a critical task in the fields of computer vision, graphics and augmented reality. However, directly learning the full HDR environment map or parametric lighting information from a single image is extremely difficult and inaccurate. As a result, we propose a two-stage network approach for illumination estimation that integrates spherical gaussian (SG) representation with scene prior knowledge. In the first stage, a convolutional neural network is utilized to generate material and geometric information about the scene, which serves as prior knowledge for lighting prediction. In the second stage, we model indoor environment illumination using 128 SG functions with fixed center direction and bandwidth, allowing only the amplitude to vary. Subsequently, a Transformer-based lighting parameter regressor is employed to capture the complex relationship between the input images with scene prior information and its SG illumination. Additionally, we introduce a hybrid loss function, which combines a masked loss for high-frequency illumination with a rendering loss for improving the visual quality. By training and evaluating the lighting model on the created SG illumination dataset, the proposed method achieves competitive results in both quantitative metrics and visual quality, outperforming state-of-the-art methods.