Object Retrieval Research Articles

Large-Scale Coarse-to-Fine Object Retrieval Ontology and Deep Local Multitask Learning.

Object retrieval plays an increasingly important role in video surveillance, digital marketing, e-commerce, etc. It is facing challenges such as large-scale datasets, imbalanced data, viewpoint, cluster background, and fine-grained details (attributes). This paper has proposed a model to integrate object ontology, a local multitask deep neural network (local MDNN), and an imbalanced data solver to take advantages and overcome the shortcomings of deep learning network models to improve the performance of the large-scale object retrieval system from the coarse-grained level (categories) to the fine-grained level (attributes). Our proposed coarse-to-fine object retrieval (CFOR) system can be robust and resistant to the challenges listed above. To the best of our knowledge, the new main point of our CFOR system is the power of mutual support of object ontology, a local MDNN, and an imbalanced data solver in a unified system. Object ontology supports the exploitation of the inner-group correlations to improve the system performance in category classification, attribute classification, and conducting training flow and retrieval flow to save computational costs in the training stage and retrieval stage on large-scale datasets, respectively. A local MDNN supports linking object ontology to the raw data, and an imbalanced data solver based on Matthews' correlation coefficient (MCC) addresses that the imbalance of data has contributed effectively to increasing the quality of object ontology realization without adjusting network architecture and data augmentation. In order to evaluate the performance of the CFOR system, we experimented on the DeepFashion dataset. This paper has shown that our local MDNN framework based on the pretrained NASNet architecture has achieved better performance (14.2% higher in recall rate) compared to single-task learning (STL) in the attribute learning task; it has also shown that our model with an imbalanced data solver has achieved better performance (5.14% higher in recall rate for fewer data attributes) compared to models that do not take this into account. Moreover, MAP@30 hovers 0.815 in retrieval on an average of 35 imbalanced fashion attributes.

DeepCCFV: Camera Constraint-Free Multi-View Convolutional Neural Network for 3D Object Retrieval

3D object retrieval has a compelling demand in the field of computer vision with the rapid development of 3D vision technology and increasing applications of 3D objects. 3D objects can be described in different ways such as voxel, point cloud, and multi-view. Among them, multi-view based approaches proposed in recent years show promising results. Most of them require a fixed predefined camera position setting which provides a complete and uniform sampling of views for objects in the training stage. However, this causes heavy over-fitting problems which make the models failed to generalize well in free camera setting applications, particularly when insufficient views are provided. Experiments show the performance drastically drops when the number of views reduces, hindering these methods from practical applications. In this paper, we investigate the over-fitting issue and remove the constraint of the camera setting. First, two basic feature augmentation strategies Dropout and Dropview are introduced to solve the over-fitting issue, and a more precise and more efficient method named DropMax is proposed after analyzing the drawback of the basic ones. Then, by reducing the over-fitting issue, a camera constraint-free multi-view convolutional neural network named DeepCCFV is constructed. Extensive experiments on both single-modal and cross-modal cases demonstrate the effectiveness of the proposed method in free camera settings comparing with existing state-of-theart 3D object retrieval methods.

Angular Triplet-Center Loss for Multi-View 3D Shape Retrieval

How to obtain the desirable representation of a 3D shape, which is discriminative across categories and polymerized within classes, is a significant challenge in 3D shape retrieval. Most existing 3D shape retrieval methods focus on capturing strong discriminative shape representation with softmax loss for the classification task, while the shape feature learning with metric loss is neglected for 3D shape retrieval. In this paper, we address this problem based on the intuition that the cosine distance of shape embeddings should be close enough within the same class and far away across categories. Since most of 3D shape retrieval tasks use cosine distance of shape features for measuring shape similarity, we propose a novel metric loss named angular triplet-center loss, which directly optimizes the cosine distances between the features. It inherits the triplet-center loss property to achieve larger inter-class distance and smaller intra-class distance simultaneously. Unlike previous metric loss utilized in 3D shape retrieval methods, where Euclidean distance is adopted and the margin design is difficult, the proposed method is more convenient to train feature embeddings and more suitable for 3D shape retrieval. Moreover, the angle margin is adopted to replace the cosine margin in order to provide more explicit discriminative constraints on an embedding space. Extensive experimental results on two popular 3D object retrieval benchmarks, ModelNet40 and ShapeNetCore 55, demonstrate the effectiveness of our proposed loss, and our method has achieved state-ofthe-art results on various 3D shape datasets.

Blinded Evaluation of Endoscopic Skill and Instructability After Implementation of an Endoscopic Simulation Experience.

Background Interest in endoscopic simulation is increasing. Past studies have used virtual reality or nonhuman models or residents with varying experience. Our aim was to evaluate the effect of simulation on procedural and psychomotor competence of medical students—surgical novices—performing endoscopic tasks on human cadavers and assess student perceptions. Methods Students (n = 22) completed a baseline sinus model skill evaluation graded by 2 blinded Rhinology fellows. Intervention and control groups with equal baselines were assigned. Intervention students practiced endoscopic tasks on the model for 45 minutes minimum over 2 weeks. All students reviewed sinus anatomy/disease and sinus surgery materials. The final cadaver evaluation was similar to the baseline. Fellows graded students on anatomy identification (sinuses, turbinates), psychomotor (navigation, camera alignment, instrument handling), and timed procedural (sinus object retrieval) skills, confidence, and instructability via fellow-guided frontal balloon placement. Results Participants included 16 males (72.7%) and 6 females (27.3%). Intervention and control groups contained 10 (45.4%) and 12 (54.6%) students, respectively. Intervention group final “Total Psychomotor” scores were higher (10.1/15 vs 7.8/15, P = .0231). “Surgical confidence” was 3.3/5 versus 2.5/5, and “Instructability” was 3.9/5 versus 3.4/5 in intervention versus control groups, respectively ( P < .050). Multivariate regression analysis demonstrated superior psychomotor skills, navigation, and confidence in the intervention group ( P < .036). Activity perception scores were higher in intervention students versus controls, 26.13 versus 18.36/40 ( P = .022). Conclusion In surgical novices, endoscopic simulation leads to superior endoscopic navigation and task performance in cadavers. This simulation presents a novel method for incorporating Otolaryngology simulation in medical student education.

Multi-view-based siamese convolutional neural network for 3D object retrieval

Representing three-dimensional (3D) objects by multiple views has become a common solution to the problem of 3D object retrieval. It has gained excellent achievements because of its remarkable adaptability and flexibility. In this paper, we develop a multi-view-based Siamese convolutional neural network for 3D object retrieval. It consists of two sub-networks which have the same architecture and also share the same set of weights. First, we generate a set of RGB images and binary images for each 3D object to capture local and global features. Second, the two sub-networks take corresponding images as input and avoid camera constraint by using average fusion layers. The final compact descriptors are learned by integrating features of sub-networks and then used for retrieval. Experimental results on two benchmarks, the PSB dataset and ETH dataset, testify that our proposed method receives superior retrieval performance compared to state-of-the-art methods.

Detection and Content Retrieval of Object in an Image using YOLO

Detection and Content Retrieval of Object in an Image using YOLO

Co-weighting semantic convolutional features for object retrieval

Deep feature aggregation, which refers to aggregating a set of local convolutional features into a global image-level vector, has attracted increasing attention in object instance retrieval. In this manuscript, we propose an unsupervised framework that aggregates feature maps by an adaptive selection and two weighting strategies. Particularly, the selection process finds the foreground contour by explaining the semantic structure implicated in the feature maps, while two weighting process including an adaptive Gaussian filter that highlights semantic features and an element-value sensitive channel vector that activates feature channels corresponding to sparse yet distinctive image patterns. Experimental results on benchmark image retrieval datasets validate that the selection and two weighting schemes are complementary in improving the discriminative power of image vectors. With the same experimental settings, the proposed approach outperforms state-of-the-art aggregation approaches by a considerable margin.

Phase retrieval exact solution based on structured window modulation without direct reference waves

Arising in coherent diffraction imaging (CDI), the phase retrieval problem is to reconstruct a complex object by measuring only diffraction intensity patterns. Existing approaches, such as iterative Fourier algorithm (IFA), transport intensity equation (TIE) methods and matrix optimisation, result in near-optimal solutions rather than exact solutions. This paper proposes an analytic method to get the exact phase retrieval solution without relying on direct reference waves. Based on structured template windows, the object far-field phase is solved analytically from the recorded Fourier intensities, from which the exact object reconstruction can be achieved. Sufficient conditions for obtaining exact phase retrieval solution are analysed rigorously and verified by numerical simulations, which also show the robustness of the method through simulations under noises, dynamic range, nonlinear response and bad alignment. A simple experimental result is provided to demonstrate the practical feasibility of the method. It achieves analytic phase retrieval for arbitrary complex-valued objects and may be applicable to quick phase imaging.

End-to-end semantic-aware object retrieval based on region-wise attention

Image representations based on pre-trained Convolutional Neural Networks (CNNs) have achieved the new state of the art in computer vision tasks such as object retrieval. Such methods usually encode the activations of convolutional layers to produce highly competitive global or local representations, as they contain the spatial information of the input image. In this work, we propose the region-wise attention mechanism to generate a semantic-aware encoding of convolutional features by two different methods. One is to re-weight the convolutional features according to the pixel-wise label from the semantic segmentation CNNs, and the other is to design a spatial attention block that adaptively recalibrates region-wise weights by explicitly modelling interdependencies between channels. We further build an end-to-end semantic-aware object retrieval pipeline based on off-the-shelf models and assess the performance of our proposed approach on the public available datasets Oxford5k and Paris6k, including large-scale datasets Oxford105k and Paris106k. As a result, we significantly improve the current state of the art.

Hyper-Clique Graph Matching and Applications

This paper proposes a method for hyper-clique graph (HCG) generation, which can be considered an extension of classical graphs and hyper-graphs in which the node is replaced with the clique (a set of neighboring nodes in a specific feature space) and the hyper-edge linking multiple nodes is replaced with the hyper-edge linking multiple cliques. In addition, we propose the HCG matching method by preserving global and local structures. Specifically, we embed the clique relations of arbitrary orders in a high-order similarity tensor in a recursive manner. Then, we formulate the objective function of HCG matching with respect to two latent variables: the latent clique structure information in the original graph and the similarity measure of clique sets from pairwise HCGs. Since the objective function is not jointly convex with respect to both latent variables, we decompose it into two consecutive measurements for optimization: 1) a clique-to-clique similarity measurement by preserving local unary and pairwise correspondences and 2) a graph-to-graph similarity measurement by preserving global clique-to-clique correspondence. We suitably adopt the affinity-preserving reweighted random walks to optimize the objective function. We extensively evaluate the HCG matching performance on multiple applications: 1) we evaluate the robustness of HCG with respect to the deformation noise, the number of outliers, and the edge density on synthetic data and explore the effects of both the clique order and hyper-edge order on performance; 2) we explore HCG matching for feature point matching on multiple image data sets (CMU house sequence, Caltech+MSRC, and Car+Motor); and 3) we explore HCG matching for multi-view object retrieval, which is a much more challenging task since multi-view objects contain significant variations of illumination, viewpoint, and so on, using popular data sets (MV-RED and NTU). A comparison against the state-of-the-art methods demonstrates the superior performance of the proposed method.

Effect of the average number of reference speckles in speckle imaging using off-axis speckle holography.

The propagation of a coherent beam of light through a random scattering medium results in the generation of a speckle pattern. Although the intensity distribution of the speckle pattern is random in nature, the information of the object hidden behind a scattering medium is scrambled into it. The scrambled object information can be retrieved using the off-axis speckle holographic technique, where the object retrieval is made from a recorded interferogram, formed by the superposition of the object speckles and a tilted reference speckle pattern. In the present paper, the effect of the average number of reference speckles on the signal-to-noise ratio of the retrieved object is investigated in two different random domains of the reference speckles, which are defined from the study of the Shannon entropy of the reference speckle patterns. The observed results can be useful in tuning the visibility and sharpness of the object, retrieved by employing the off-axis speckle holographic technique.

An improved phase-coding method for absolute phase retrieval based on the path-following algorithm

Although the traditional phase-coding method for absolute phase retrieval possesses strong robustness, the method leads to unwrapping artifacts as a mass of codewords required. To solve this problem, a novel improved method is proposed in this paper. The incorrect points in the recovered phase via the traditional method are located using a computational framework, and the path-following algorithm is adopted as an aid to correct the unwrapping artifacts. The effectiveness of the proposed method is experimentally verified by an established measurement system. The reconstructed geometry of standard sphere has sufficient accuracy and the absolute phase retrieval of a complex star object has high quality.

Infants plan prehension while pivoting.

Skilled object retrieval requires coordination of the perceptual and motor systems. Coordination is especially challenging when body position is changing and visual search is required to locate the target. In three experiments, we used a "pivot paradigm" to induce changes in body position: Participants were passively pivoted 180° toward a target placed at varied locations to the left and right of the center of a reaching board. Experiment 1 showed that 6- to 15-month-old infants (n=41) plan prehension so quickly that they retrieve targets mid-turn and scale their reaches to target location relative to turn direction. Experiment 2 characterized planning mid-turn reaching in 6- to 8-month-olds (n=5) wearing a head-mounted eye tracker. Reach planning depended on when the target appeared in the field of view-not on target fixation. Experiment 3 used head-mounted eye tracking and motion tracking to assess perceptual-motor coordination in adults (n=13). Adults displayed more mid-turn reaching than infants. But like infants, adults scaled reaching to target location relative to turn direction, and contact time depended on when the target came into view-not on target fixation. Findings show that fast, efficient perceptual-motor coordination supports flexibility in infant prehension, and constraints on coordination are similar across the lifespan.

Chronic phencyclidine treatment impairs spatial working memory in rhesus monkeys.

Phencyclidine (PCP) could induce schizophrenia (Sz) like behavior in both humans and animals, therefore, has been widely utilized to establish Sz animal models. It induced cognitive deficits, the core symptom of Sz, mainly through influencing frontal dopaminergic function. Nonhuman primate (NHP) studies demonstrated impaired object retrieval detour (ORD) and spatial delayed response (SDR) task performance by acute or chronic PCP treatment. However, NHP investigations, continually monitoring SDR performance before, during and after PCP treatment, are lacking. Present study investigated the long-term influence of chronic PCP treatment on SDR performance and the possible increase of SDR deficit severity and duration by the incremental dosing procedure in rhesus monkeys. SDR task was performed repeatedly up to eight weeks after constant dosing procedure (i.m., 0.3 mg/kg, day 12-25), during which drug effects on locomotor activity and blood cortisol concentration were assessed. Incremental dosing procedure (starting dose 0.3 mg/kg, day 6-19) began five months later. Constant dosing procedure induced differential level of hyperactivity across testing days, without significant influence on blood cortisol concentration. It reduced SDR performance, until occurrence of the first and worst impairment on day 15 and 23 respectively. The impaired performance recovered to pretreatment level over one week after drug cessation. In contrast, incremental dosing procedure impaired SDR performance on the first treatment day, which recovered within treatment period. Results suggested increase of SDR deficit severity by repeated PCP administrations, whereas the incremental dosing procedure did not increase SDR deficit severity and duration.

3D Object Retrieval Based on Multi-View Latent Variable Model

View-based 3D object retrieval, in which multiple views are used for representation and retrieval, has attracted increasing attention due to its great flexibility. In this paper, we propose a discriminative multi-view latent variable model (MVLVM) for this task. Specifically, we design MVLVM to have an undirected graph structure in which the view set of a given 3D object is treated as the observations from which to discover the latent visual and spatial contexts. Then, we detail the learning and inference process of MVLVM for view-based 3D object retrieval. The proposed MVLVM has the following beneficial features: 1) it jointly learns visual and spatial contexts for 3D object modelling and 2) it avoids the difficulty of representative view extraction for model representation. Consequently, it can support flexible 3D model retrieval for real applications by avoiding camera array constraints, which severely constrain traditional methods. We report extensive experiments conducted on single-modal datasets (the NTU and ITI datasets) and a multi-modal dataset (MVRED-RGB and MVRED-Depth). These comparative experiments demonstrate the superiority of the proposed method.

Improving Object Retrieval Quality by Integration of Similarity Propagation and Query Expansion

Re-ranking is an essential step for accurate image retrieval, due to its well-known power in performance improvement. Although numerous works have been proposed for re-ranking, many of them are only customized for a certain image representation model. In contrast to most existing techniques, we develop generalized re-ranking algorithms that are applicable to different kinds of image encodings in this paper. We first employ a quite successful theory of similarity propagation to reconstruct vectors of a query and its top ranked images and, subsequently, get a re-ranked list by comparing the new image vectors. Furthermore, considering that the just mentioned strategy is directly compatible with query expansion and, thus, in order to leverage advantages of this milestone, we then propose integrating them into a unified framework for maximizing re-ranking benefits. Our re-ranking algorithms are memory and computation efficient, and experimental results on benchmark datasets demonstrate that they compare favorably with the state of the art. Our code is available at https://github.com/MaJinWakeUp/rerank .

Graph-based particular object discovery

Severe background clutter is challenging in many computer vision tasks, including large-scale image retrieval. Global descriptors, which are popular due to their memory and search efficiency, are especially prone to corruption by such a clutter. Eliminating the impact of the clutter on the image descriptor increases the chance of retrieving relevant images and prevents topic drift due to actually retrieving the clutter in the case of query expansion. In this work, we propose a novel salient region detection method. It captures, in an unsupervised manner, patterns that are both discriminative and common in the dataset. Saliency is based on a centrality measure of a nearest neighbor graph constructed from regional CNN representations of dataset images. The proposed method exploits recent CNN architectures trained for object retrieval to construct the image representation from the salient regions. We improve particular object retrieval on challenging datasets containing small objects.

End-to-End Visual Domain Adaptation Network for Cross-Domain 3D CPS Data Retrieval

3D CPS (Cyber Physical System) data has been widely generated and utilized for multiple applications, e.g. autonomous driving, unmanned aerial vehicle and so on. For large-scale 3D CPS data analysis, 3D object retrieval plays a significant role for urban perception. In this paper, we propose an end-to-end domain adaptation framework for cross-domain 3D objects retrieval (C3DOR-Net), which learns a joint embedding space for 3D objects from different domains in an end-to-end manner. Specifically, we focus on the unsupervised case when 3D objects in the target domain are unlabeled. To better encode a 3D object, the proposed method learns multi-view visual features in a data-driven manner for 3D object representation. Then, the domain adaptation strategy is implemented to benefit both domain alignment and final classification. Specifically, an center-based discriminative feature learning method enables the domain invariant features with better intra-class compactness and inter-class separability. C3DOR-Net can achieve remarkable retrieval performances by maximizing the inter-class divergence and minimizing the intra-class divergence. We evaluate our method on two cross-domain protocols: 1) CAD-to-CAD object retrieval on two popular 3D datasets (NTU and PSB) in three designed cross-domain scenarios; 2) SHREC’19 monocular image based 3D object retrieval. Experimental results demonstrate that our method can significantly boost the cross-domain retrieval performances.

Accidental diagnosis of a foreign body embedded in maxillary anterior tooth

Accidental ingestion or aspiration of a variety of foreign bodies is a common pediatric emergency. A foreign body in the tooth, however, is rare. Retrieval of foreign objects from the teeth in children is a challenging aspect of pediatric dental practice. In most of the situations, the foreign objects are diagnosed in routine radiographs, which play a vital role in analyzing the location, size, and type of the foreign object. Removal of the foreign body is not very difficult if it is confined to the pulp chamber or root canal. This article describes two cases of foreign object embedded in the teeth, the possible etiology, and treatment.

Multiple foreign body ingestion in pica patient

Appetite for substances that are widely non-nutritive is termed Pica and commonly seen in patients with psychiatric disorders. Their clinical presentation may be delayed and include multiple foreign bodies. Most of such ingested bodies pass through gastrointestinal tract uneventfully and only on rare instances cause obstruction or perforation. The length and sharpness of an object govern its risk of perforation. Physicians are dependent upon imaging to detect a foreign body, especially if the history of ingestion is doubtful. We present a case of pica with multiple foreign bodies diagnosed on radiography and ultrasonography with the findings correlating with surgical retrieval of objects.