Open Image Research Articles

Gaze-Dependent Image Re-Ranking Technique for Enhancing Content-Based Image Retrieval

Content-based image retrieval (CBIR) aims to find desired images similar to the image input by the user, and it is extensively used in the real world. Conventional CBIR methods do not consider user preferences since they only determine retrieval results by referring to the degree of resemblance or likeness between the query and potential candidate images. Because of the above reason, a “semantic gap” appears, as the model may not accurately understand the potential intention that a user has included in the query image. In this article, we propose a re-ranking method for CBIR that considers a user’s gaze trace as interactive information to help the model predict the user’s inherent attention. The proposed method uses the user’s gaze trace corresponding to the image obtained from the initial retrieval as the user’s preference information. We introduce image captioning to effectively express the relationship between images and gaze information by generating image captions based on the gaze trace. As a result, we can transform the coordinate data into a text format and explicitly express the semantic information of the images. Finally, image retrieval is performed again using the generated gaze-dependent image captions to obtain images that align more accurately with the user’s preferences or interests. The experimental results on an open image dataset with corresponding gaze traces and human-generated descriptions demonstrate the efficacy or efficiency of the proposed method. Our method considers visual information as the user’s feedback to achieve user-oriented image retrieval.

GND-PCA Method for Identification of Gene Functions Involved in Asymmetric Division of C. elegans

Due to the rapid development of imaging technology, a large number of biological images have been obtained with three-dimensional (3D) spatial information, time information, and spectral information. Compared with the case of two-dimensional images, the framework for analyzing multidimensional bioimages has not been completely established yet. WDDD is an open biological image database. It dynamically records 3D developmental images of 186 samples of nematodes C. elegans. In this study, based on WDDD, we constructed a framework to analyze the multidimensional dataset, which includes image segmentation, image registration, size registration by the length of main axes, time registration by extracting key time points, and finally, using generalized N-dimensional principal component analysis (GND-PCA) to analyze the phenotypes of bioimages. As a data-driven technique, GND-PCA can automatically extract the important factors involved in the development of P1 and AB in C. elegans. A 3D bioimage can be regarded as a third-order tensor. Therefore, GND-PCA was applied to the set of third-order tensors, and a set of third-order tensor bases was iteratively learned to linearly approximate the set. For each tensor base, a corresponding characteristic image is built to reveal its geometric meaning. The results show that different bases can be used to express different vital factors in development, such as the asymmetric division within the two-cell stage of C. elegans. Based on selected bases, statistical models were built by 50 wild-type (WT) embryos in WDDD, and were applied to RNA interference (RNAi) embryos. The results of statistical testing demonstrated the effectiveness of this method.

An underwater image enhancement model for domain adaptation

Underwater imaging has been suffering from color imbalance, low contrast, and low-light environment due to strong spectral attenuation of light in the water. Owing to its complex physical imaging mechanism, enhancing the underwater imaging quality based on the deep learning method has been well-developed recently. However, individual studies use different underwater image datasets, leading to low generalization ability in other water conditions. To solve this domain adaptation problem, this paper proposes an underwater image enhancement scheme that combines individually degraded images and publicly available datasets for domain adaptation. Firstly, an underwater dataset fitting model (UDFM) is proposed to merge the individual localized and publicly available degraded datasets into a combined degraded one. Then an underwater image enhancement model (UIEM) is developed base on the combined degraded and open available clear image pairs dataset. The experiment proves that clear images can be recovered by only collecting the degraded images at some specific sea area. Thus, by use of the scheme in this study, the domain adaptation problem could be solved with the increase of underwater images collected at various sea areas. Also, the generalization ability of the underwater image enhancement model is supposed to become more robust. The code is available at https://github.com/fanren5599/UIEM.

Bee Movie

Abstract Bee Movie is a short film that invites questions for conversations on visual ethics. The viewer is invited into a world of circular, tragic and absurd questions concerning what a filmmaker, an abstract writer, a journal editor, and a film viewer ought to do when observing the apparent reality of a bee’s circular attempts to escape a pond. As a filmmaker and abstract writer, one does not want to tell the viewer and reader how one feels about this bee, bees, insects, ponds, water, life, death and circles. And one feels obliged not to explain the context through which the film came to life as the bee was engaged in efforts that might be narrated as lifesaving but also as another complex of efforts entirely. As an open image, without our textual dissection (or, at least, with a dampening of that dissection to an abstract and with a few questions and challenges), we regard Bee Movie as inviting questions about the ethics of the use of images as pedagogy. Whatever we thought about whether and how to share this film, we find ourselves in 359 other relationships, and always back again to what we assumed might be the starting position.

A comparative study on image forgery-facial retouching

Forgery with the digital images is being very easy now days due to the very advanced and open source image editing tools, software and devises which supports a high quality of resolutions. Tempering with digital documents for changing identity or sometimes for fun is increasing day by day as the era is of digital world. Detecting clues of tampering and verifying the authenticity of images is an important issues now-a-days and growing research field. The existing research in the area of digital image forgery identification is discussed here. Different types of image forgery attacks along with its detection and classification are discussed too. A general review of facial image authentication is studied and discussed here along with the current methods and techniques for the detection and classification. A comparative analysis is presented here for the researchers as a mind mapping to work in the same field.

Efficient Hashing Method Using 2D-2D PCA for Image Copy Detection

Image copy detection is an important technology of copyright protection. This paper proposes an efficient hashing method for image copy detection using 2D-2D (two-directional two-dimensional) PCA (Principal Component Analysis). The key is the discovery of the translation invariance of 2D-2D PCA. With the property of translation invariance, a novel model of extracting rotation-invariant low-dimensional features is designed by combining PCT (Polar Coordinate Transformation) and 2D-2D PCA. The PCT can convert an input rotated image to a translation matrix. Since the 2D-2D PCA is invariant to translation, the low-dimensional features learned from the translation matrix are rotation-invariant. Moreover, vector distances of low-dimensional features are stable to common digital operations and thus hash construction with the vector distances is of robustness and compactness. Three open image datasets are exploited to conduct various experiments for validating efficiencies of the proposed method. The results demonstrate that the proposed method is much better than some representative hashing methods in the performances of classification and copy detection.

Performance Analysis of You Only Look Once Object Detection Models on Custom Datasets

Object detection, localization, and classification are now thought to be common study areas. The advancement of Deep Learning has aided researchers working in the field of object detection. By decreasing the model’s loss functi on, the accuracy of object detection may be boosted. Training and executing this model on a suitable GPUenabled machine can also cut down on total time. In this article, we employed the YOLO family models, namely YOLO v3, YOLO v4, and YOLO v5, and trained them using the transfer learning approach on custom datasets. These cutting- edge performers outperformed the competition on the Custom Dataset of Open Image Dataset V6 for various classes, achieving mean average precision mAP @0.5 97 %, 89 %, and 88 %, respectively. This is better than the initial trained models, which had mAP @0.5 values of 57.9 %, 43.5 %, and 33 % for YOLO v3, YOLO v4, and YOLO v5, respectively

Feasibility of Tc-99 m sestamibi uptake quantification with few-projection emission tomography.

Molecular breast imaging uses Tc-99 m sestamibi to obtain functional images of the breast. Determining the Tc-99 m sestamibi uptake in volumes of interest in the breast may be useful in assessing the response to neoadjuvant chemotherapy or for the purposes of breast cancer risk assessment. To determine, using Monte Carlo simulation, if emission tomography can be used to quantify the uptake of Tc-99 m sestamibi in molecular breast imaging and if so, to determine the accuracy as a function of the number of projections used in the reconstruction process. In this study, two voxelized breast models are implemented with different ratios of fibroglandular to fatty tissue and tumoral masses of varying dimensions. Monte Carlo simulation is used to calculate sets of projections, which assumes that each tumoral mass contains a given Tc-99 m activity. Projections are also calculated for a calibration phantom in order to correlate the known activity with the image pixel value. For each case, the total number of calculated projections is 36 and the reconstruction is carried out for 36, 18, 9, 7 and 5 projections, respectively, using an open source image reconstruction toolbox. Study data show that determination of Tc-99 m sestamibi uptake with and average error of 7% can be carried out with as little as 7 projections. Molecular breast emission tomography enables to accurately determine the Tc-99 m sestamibi tumoral mass uptake with the number of projections very close to the number of images currently acquired in clinical practice.

Skew Class-Balanced Re-Weighting for Unbiased Scene Graph Generation

An unbiased scene graph generation (SGG) algorithm referred to as Skew Class-Balanced Re-Weighting (SCR) is proposed for considering the unbiased predicate prediction caused by the long-tailed distribution. The prior works focus mainly on alleviating the deteriorating performances of the minority predicate predictions, showing drastic dropping recall scores, i.e., losing the majority predicate performances. It has not yet correctly analyzed the trade-off between majority and minority predicate performances in the limited SGG datasets. In this paper, to alleviate the issue, the Skew Class-Balanced Re-Weighting (SCR) loss function is considered for the unbiased SGG models. Leveraged by the skewness of biased predicate predictions, the SCR estimates the target predicate weight coefficient and then re-weights more to the biased predicates for better trading-off between the majority predicates and the minority ones. Extensive experiments conducted on the standard Visual Genome dataset and Open Image V4 and V6 show the performances and generality of the SCR with the traditional SGG models.

Validation of the open biological negative image set for a Portuguese population: Comparing Japanese and Portuguese samples and an exploration of low-order visual properties of the stimuli.

Recently, Shirai and Watanabe Royal Society Open Science, 9(1), 211128 (2022) developed OBNIS (Open Biological Negative Image Set), a comprehensive database containing images (primarily animals but also fruits, mushrooms, and vegetables) that visually elicit disgust, fear, or neither. OBNIS was initially validated for a Japanese population. In this article, we validated the color version of OBNIS for a Portuguese population. In study 1, the methodology of the original article was used. This allowed direct comparisons between the Portuguese and Japanese populations. Aside from a few emotional classification mismatches between disgust, fear, or neither-related images, we found that arousal and valence relate distinctively in both populations. In contrast to the Japanese sample, the Portuguese reported increased arousal for more positive valenced stimuli, suggesting that OBNIS images elicit positive emotions in the Portuguese population. These results showed important cross-cultural differences regarding OBNIS. In study 2, a methodological change was introduced: instead of the three classification options used originally (fear, disgust, or neither), six basic emotions were used (fear, disgust, sadness, surprise, anger, happiness), and a "neither" option, to confirm whether some of the originally "neither-related" images are associated with positive emotions (happiness). Additionally, the low-order visual properties of images (luminosity, contrast, chromatic complexity, and spatial frequency distribution) were explored due to their important role in emotion-related research. A fourth image group associated with happiness was found in the Portuguese sample. Moreover, image groups present differences regarding the low-order visual characteristics, which are correlated with arousal and valence ratings, highlighting the importance of controlling such characteristics in emotion-related research.

Robust Data-Aware Sensing Based on Half Quadratic Hashing for the Visual Internet of Things

In the Visual Internet of Things (VIoT) such as drones over flying ad-hoc networks (FANETs), terminal nodes need to preliminarily identify sensing data. Thus, data can be rapidly dispatched to dedicated edge nodes for further processing. Nonetheless, terminals may collect partially observed visual data, which cause biased labeling. In view of such, this study proposes half quadratic supervised discrete hashing (HQSDH) that can conquer the biased similarity estimation for partially observed images, especially when large continuous occlusion, missing values, or sample-specific outliers exist. The proposed method can automatically and adaptively adjust itself via HQ auxiliary variables to avoid oversensitivity to large errors caused by partially observed data. In this study, to solve HQ discrete optimization problems in HQSDH, HQ discrete cyclic coordinate descent (HQDCCD) is developed. Second, partially observed images may cause inconsistent binary codes and increase the Hamming distances. Thus, this study devises HQ balancing decorrelation constraints along with normalization regularization to accommodate the problems. Third, a systematic and generic solution that allows various HQ functions to jointly work in the same framework is modeled to provide versatility. Experiments on open image data sets were carried out for evaluation. The results showed that the proposed method provided more resilience against continuous occlusion, missing values, and sample-specific outliers than the baselines when dealing with partially observed images.

First person – Sara Rolfe

ABSTRACT First Person is a series of interviews with the first authors of a selection of papers published in Biology Open, helping researchers promote themselves alongside their papers. Sara Rolfe is first author on ‘ Deep learning enabled multi-organ segmentation of mouse embryos’, published in BiO. Sara is a Research Scientist in the lab of Murat Maga at Seattle Children's Research Institute, investigating quantifying morphology from 3D images, automated phenotyping, and open image science.

A Comparative Study of YOLOv5 and YOLOv7 Object Detection Algorithms

This paper presents a comparative analysis of the widely accepted YOLOv5 and the latest version of YOLO which is YOLOv7. Experiments were carried out by training a custom model with both YOLOv5 and YOLOv7 independently in order to consider which one of the two performs better in terms of precision, recall, mAP@0.5 and mAP@0.5:0.95. The dataset used in the experiment is a custom dataset for Remote Weapon Station which consists of 9,779 images containing 21,561 annotations of four classes gotten from Google Open Images Dataset, Roboflow Public Dataset and locally sourced dataset. The four classes are Persons, Handguns, Rifles and Knives. The experimental results of YOLOv7 were precision score of 52.8%, recall value of 56.4%, mAP@0.5 of 51.5% and mAP@0.5:0.95 of 31.5% while that of YOLOv5 were precision score of 62.6%, recall value of 53.4%, mAP@0.5 of 55.3% and mAP@0.5:0.95 of 34.2%. It was observed from the experiment conducted that YOLOv5 gave a better result than YOLOv7 in terms of precision, mAP@0.5 and mAP@0.5:0.95 overall while YOLOv7 has a higher recall value during testing than YOLOv5. YOLOv5 records 4.0% increase in accuracy compared to YOLOv7.

Intraoperative Assessment of Surgical Margins in Cancer Resection Surgery Via the Sub-Diffuse Optical Tomography "S-DOT"

In Head and Neck Squamous Cell Carcinomas (HNSCC), positive margins after surgical resection dominate the clinical outcome of patients. Between 15-30 precent of all HNSCC surgeries result in a positive margin (>5mm) that requires postoperative chemo-radiation, radiotherapy, and/or revision surgeries. To minimize local recurrence, margin assessment is typically performed through surgical pathology departments; however, this technique often takes more than 24 hours to process and only a small fraction of the total surgical margin is evaluated. A second, intraoperativeapproach used by surgeons to minimize local recurrence is frozen section analysis (FSA), which in time intensive and suffers heavily from sampling errors. To effect change, we have hypothesized a sub-diffuse optical tomography modality that can identify the differences in depth sensitivity of the fluorescence, Cetuximab-IRDye800CW conjugate (ICON), by the fluorescent photons collected from sub-diffused media as a function of the photon exit angle from the excited tissue. Based on promising preliminary simulations using Monte Carlo MATLAB, closed vs. open aperture fluorescence imaging in biological tissue demonstrated an enhanced true depth of fluorescence resolution up to 6mm, well beyond the insufficient 5mm mark. By identifying the deep margins from the fluorescent peaks within 5 minutes, we hope to improve complete oncological surgery by providing surgeons an exact region of interest on the specimen where the tumor is closest to the edge, so they may continue surgery by providing sugeons an exact region of interest on the specimen where the tumor is closest to the edge, so they may continue surgery on the spot to minimize the probability of cancer being left behind.

A Novel Geometric Parameter Self-Calibration Method for Portable CBCT Systems

In outdoor environments or environments with space restrictions, it is difficult to transport and use conventional computed tomography (CT) systems. Therefore, there is an urgent need for rapid reconstruction of portable cone-beam CT (CBCT) systems. However, owing to its portability and the characteristics of temporary construction environments, high precision spatial location is difficult to achieve with portable CBCT systems. To overcome these limitations, we propose an iterative self-calibration improvement method with a self-calculated initial value based on the projection relationship and image features. The CT value of an open field image was used as the weight value of the projection data in the subsequent experiments to reduce the nonlinear attenuation of the projection intensity. Subsequently, an initial value was obtained based on the invariance of the rotation axis. Finally, self-calibration was realized iteratively using the reconstructed image. This method overcomes the main problem of the rotation axis invariance calibration algorithm—high similarity between the adjacent positions of symmetrical homogeneous materials. The proposed method not only improves the precision of self-calibration based on the projection relationship, but also reduces the performance cost and solution time of the self-calibration algorithm based on the image features. Thus, it satisfies the precision requirements for self-calibration of portable CBCT systems.

Enhancement of Marine Lantern’s Visibility under High Haze Using AI Camera and Sensor-Based Control System

This thesis describes research to prevent maritime safety accidents by notifying navigational signs when sea fog and haze occur in the marine environment. Artificial intelligence, a camera sensor, an embedded board, and an LED marine lantern were used to conduct the research. A deep learning-based dehaze model was learned by collecting real marine environment and open haze image data sets. By applying this learned model to the original hazy images, we obtained clear dehaze images. Comparing those two images, the concentration level of sea fog was derived into the PSNR and SSIM values. The brightness of the marine lantern was controlled through serial communication with the derived PSNR and SSIM values in a realized sea fog environment. As a result, it was possible to autonomously control the brightness of the marine lantern according to the concentration of sea fog, unlike the current marine lanterns, which adjust their brightness manually. This novel-developed lantern can efficiently utilize power consumption while enhancing its visibility. This method can be used for other fog concentration estimation systems at the embedded board level, so that applicable for local weather expectations, UAM navigation, and autonomous driving for marine ships.

A novel approach for yoga pose estimation based on in-depth analysis of human body joint detection accuracy.

Virtual motion and pose from images and video can be estimated by detecting body joints and their interconnection. The human body has diverse and complicated poses in yoga, making its classification challenging. This study estimates yoga poses from the images using a neural network. Five different yoga poses, viz. downdog, tree, plank, warrior2, and goddess in the form of RGB images are used as the target inputs. The BlazePose model was used to localize the body joints of the yoga poses. It detected a maximum of 33 body joints, referred to as keypoints, covering almost all the body parts. Keypoints achieved from the model are considered as predicted joint locations. True keypoints, as the ground truth body joint for individual yoga poses, are identified manually using the open source image annotation tool named Makesense AI. A detailed analysis of the body joint detection accuracy is proposed in the form of percentage of corrected keypoints (PCK) and percentage of detected joints (PDJ) for individual body parts and individual body joints, respectively. An algorithm is designed to measure PCK and PDJ in which the distance between the predicted joint location and true joint location is calculated. The experiment evaluation suggests that the adopted model obtained 93.9% PCK for the goddess pose. The maximum PCK achieved for the goddess pose, i.e., 93.9%, PDJ evaluation was carried out in the staggering mode where maximum PDJ is obtained as 90% to 100% for almost all the body joints.

π-spaces and their open images

We study spaces that can be mapped onto the Baire space (i.e. the countable power of the countable discrete space) by a continuous quasi-open bijection. We give a characterization of such spaces in terms of Souslin schemes and call these spaces π-spaces. We show that every space that has a Lusin π-base is a π-space and that every second-countable π-space has a Lusin π-base. The main result of this paper is a characterization of continuous open images of π-space.

Examining the utility of a photorealistic virtual ear in otologic education

BackgroundOtolaryngology–head and neck surgical (OHNS) trainees’ operating exposure is supplemented by a combination of didactic teaching, textbook reading, and cadaveric dissections. Conventional teaching, however, may not adequately equip trainees with an understanding of complex visuospatial relationships of the middle ear. Both face and content validation were assessed of a novel three-dimensional (3D) photorealistic virtual ear simulation tool underwent face and content validation as an educational tool for OHNS trainees.MethodsA three-dimensional mesh reconstruction of open access imaging was generated using geometric modeling, which underwent global illumination, subsurface scattering, and texturing to create photorealistic virtual reality (VR) ear models were created from open access imaging and comiled into a educational platform. This was compiled into an educational VR platform which was explored to validate the face and content validity questionnaires in a prospective manner. OHNS post-graduate trainees were recruited from University of Toronto and University of Calgary OHNS programs. Participation was on a voluntary basis.ResultsTotal of 23 OHNS post-graduate trainees from the two universities were included in this study. The mean comfort level of otologic anatomy was rated 4.8 (± 2.2) out of 10. Senior residents possessed more otologic surgical experience (P < 0.001) and higher average comfort when compared to junior residents [6.7 (± 0.7) vs. 3.6 (± 1.9); P = 0.001]. Face and content validities were achieved in all respective domains with no significant difference between the two groups. Overall, respondents believed OtoVIS was a useful tool to learn otologic anatomy with a median score of 10.0 (8.3–10.0) and strongly agreed that OtoVIS should be added to OHNS training with a score of 10.0 (9.3–10.0).ConclusionsOtoVIS achieved both face and content validity as a photorealistic VR otologic simulator for teaching otologic anatomy in the postgraduate setting. As an immersive learning tool, it may supplement trainees’ understanding and residents endorsed its use.Graphical

Quaternion Relation Embedding for Scene Graph Generation

As an important visual understanding task, scene graph generation has been drawing widespread attention and could boost a broad range of downstream vision applications. Traditional scene graph generation methods based on different context refinements are trained with probabilistic chain rule, which treats objects and relationships as independent entities. Despite their surprisingly great progress, such a plain formulation unconsciously ignores the latent geometric structure of entities and relationships. To address this issue, we move beyond the traditional real-valued representations and use <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Quat</u> ernion <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R</u> elation <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">E</u> mbedding (QuatRE) to generate scene graphs with more expressive hypercomplex representations. More specifically, we introduce the concept of quaternion representations, hyper-complex valued with three imaginary components for objects entities, then formulate the relation triplets with Hamilton product. Benefiting from explicitly modeling the latent inter-dependencies among all imaginary components and strong expressive capacity, our proposed QuatRE method could better capture the interactions between entities. More importantly, our novel QuatRE method can be treated as a plug-in and well generalized into other methods for performance improvement as it involves no additional layers. Finally, extensive comparisons of our proposed method against the state-of-the-art methods on two large-scale and widely-used datasets, i.e. Visual Genome and Open Images, demonstrated our superiority and generalization capability on various metrics for biased or unbiased inference.