Large-scale Image Database Research Articles

MIL-ViT: A multiple instance vision transformer for fundus image classification

Despite the great success of deep learning approaches, retinal disease classification is still challenging as the early-stage pathological regions of retinal diseases may be extremely tiny and subtle, which are difficult for networks to detect. The feature representations learnt by deep learning models focusing more on the local view may lead to indiscriminative semantic-level representation. On the contrary, if they focus more on the global semantic-level, they may ignore the discerning subtle local pathological regions. To address this issue, in this paper, we propose a hybrid framework, combining the strong global semantic representation learning capability of the vision Transformer (ViT) and the excellent capacity of local representation extraction from the conventional multiple instance learning (MIL). Particularly, a multiple instance vision Transformer (MIL-ViT) is implemented, where the vanilla ViT branch and the MIL branch generate semantic probability distributions separately, and a bag consistency loss is proposed to minimize the difference between them. Moreover, a calibrated attention mechanism is developed to embed the instance representation into the bag representation in our MIL-ViT. To further improve the feature representation capability for fundus images, we pre-train the vanilla ViT on a large-scale fundus image database. The experimental results validate that the generalization capability of the model using our pre-trained weights for fundus disease diagnosis is better than the one using ImageNet pre-trained weights. Extensive experiments on four publicly available benchmarks demonstrate that our proposed MIL-ViT outperforms latest fundus image classification methods, including various deep learning models and deep MIL methods. All our source code and pre-trained models are publicly available at https://github.com/greentreeys/MIL-VT.

A Method on Classification and Recognition of Noisy Plant Images Based on Visual Domain Perception

At present, some achievements have been made in the research of plant leaf classification such as the introduction of artificial intelligence algorithm. But there are still some problems. First, the existing achievements do not consider the subjective perception mechanism and role of human visual system in leaf classification data labels. Second, the implementation of the deep learning algorithm completely depends on the computing power level of the high-cost machine hardware and the large-scale image database. Finally, these research results rarely consider the noise pollution of leaf image samples. In order to solve the above problems, the paper fully considered the subjective perception principle and characteristics of human vision system (HVS), and proposed a lightweight classification method of noisy plant leaves (LCM-NPLs) based on visual domain perception. First, the most suitable HVS front-end perception characteristics were applied to the physical visual processing of leaves. Then the plant leaves were denoised through the information processing mechanism of HVS back-end. The visual effect of regular and orderly plant leaves is obtained. Finally, the classification is realized by principal component analysis (PCA) and third-order nearest neighbor algorithm. The results of ablation contrast experiments show that the classification accuracy of the method in this paper is 82.50% for plant leaves in the presence of serious noise interference with PSNR of 10.2421, more than 90% for plant leaves with general noise pollution transmission with PSNR of more than 15.3759, and 98.33% for plant leaves of light pollution with PSNR of 20.5659. The proposed method has achieved very good results. The proposed method can not only accurately classify plant leaves in different growth periods, but also maintain a high classification accuracy rate in the presence of serious noise interference.

Learning binary codes for fast image retrieval with sparse discriminant analysis and deep autoencoders

Image retrieval with relevant feedback on large and high-dimensional image databases is a challenging task. In this paper, we propose an image retrieval method, called BCFIR (Binary Codes for Fast Image Retrieval). BCFIR utilizes sparse discriminant analysis to select the most important original feature set, and solve the small class problem in the relevance feedback. Besides, to increase the retrieval performance on large-scale image databases, in addition to BCFIR mapping real-valued features to short binary codes, it also applies a bagging learning strategy to improve the ability general capabilities of autoencoders. In addition, our proposed method also takes advantage of both labeled and unlabeled samples to improve the retrieval precision. The experimental results on three databases demonstrate that the proposed method obtains competitive precision compared with other state-of-the-art image retrieval methods.

Faster and Lighter Meteorological Satellite Image Classification by a Lightweight Channel-Dilation-Concatenation Net

With the development of satellite photography, meteorologists are inclined to rely on methods for the automatic and efficient classification of weather images. However, many popular networks require numerous parameters and a lengthy inference time, making them unsuitable for real-time classification tasks. To solve these problems, a lightweight convolutional network termed the Channel-Dilation-Concatenation network (CDC-net) is constructed for meteorological satellite image classification. When extracting features, CDC-net utilizes depth-wise convolution rather than standard convolution. Additionally, a FeatureCopy operation was employed instead of a half-convolution operation. CDC-net extracts high-dimensional features and contains a local importance-based pooling layer, reducing the network's depth, the number of network parameters and inference time. Based on these techniques, the CDC-net achieves an accuracy of 93.56% on the Large-Scale Satellite Cloud Image Database for Meteorological Research, with a GPU inference time of 3.261 ms and 1.12 million parameters. Because many weather images reveal multiple weather patterns, multiple labels are necessary. Therefore, we propose a prediction method and conduct experiments on multi-label data. Experiments on single-label and multi-label meteorological satellite image datasets demonstrate the superiority of the CDC-net over other structures. Thus, the proposed CDC-net can provide a faster and lighter solution in meteorological satellite image classification.

Guest Editorial Large-Scale Medical Image and Video Analytics for Clinical Decision Support

The papers in this special section focus on large-scale medical imaging and video analytics for clinical decision support systems. Biomedical images and videos are ubiquitous and overwhelming in volume, amounting to a database that can be measured in zettabytes.With increased access to open image and video datasets and the recent development of effective image and video analysis systems, there is a unique opportunity for the development of artificial intelligence (AI) systems that can be trained and tested on large-scale biomedical image and video databases. The special issue summarizes emerging methods associated with the development of computer-aided diagnostic systems. More specifically, the special issue discusses the development of methods for dealing with small or large or creating new datasets, biomedical image segmentation, and image classification. The development of new dataset methods allows us to develop methods for specific diseases, employ meta-learning for training on small datasets, or develop methods for reducing larger video datasets. Biomedical image segmentation is a primary focus of the special issue.

A large-scale image database for benchmarking mobile camera quality and NR-IQA algorithms

Camera-oriented quality assessment (CQA) differs from traditional image quality assessment (IQA) in that “distorted” images are straight out of real devices instead of various types and levels of artificial operations. However, despite its value for both customers and manufacturers, academic and industrial fields, there are few CQA databases created years ago. To reflect recent mobile imaging advancements, we present a new massive Phone Camera Benchmarking database (PCB2021) in this paper. In PCB2021, 40 modern phone units featuring photography are simultaneously compared in 182 scenes for a total of 7280 images, which can be classified into six categories (sub-datasets) based on different focal lengths and user cases: main camera (107), ultra-wide (20), 2×, 3×, 5× zoom lenses (26, 8, 10) and night-mode (11). The shooting process begins from 7:00 am to 11:00 pm and lasts more than a month. In the subjective study, to overcome the high-resolution-induced overall quality evaluation difficulty, five image attributes: exposure/contrast, color, sharpness, graininess, artifacts are assessed separately on each dataset. To reduce ranking complexity for large-scale cameras, a dynamic anchor ruler method is proposed to obtain quality orders efficiently. With the constructed PCB2021, we further evaluate 15 mainstream no-reference (NR) IQA algorithms. The finding is that for zoomed images, sharpness metrics can achieve Spearman correlation coefficients above 0.8, while for the subtle main camera and night-mode images, performances of all fifteen algorithms drop down quickly, i.e. 0.1-0.2 for the former and ∼0.5 for the latter. The entire database, expert rankings and algorithm performance reports will be freely available on request.

LSCIDMR: Large-Scale Satellite Cloud Image Database for Meteorological Research.

People can infer the weather from clouds. Various weather phenomena are linked inextricably to clouds, which can be observed by meteorological satellites. Thus, cloud images obtained by meteorological satellites can be used to identify different weather phenomena to provide meteorological status and future projections. How to classify and recognize cloud images automatically, especially with deep learning, is an interesting topic. Generally speaking, large-scale training data are essential for deep learning. However, there is no such cloud images database to date. Thus, we propose a large-scale cloud image database for meteorological research (LSCIDMR). To the best of our knowledge, it is the first publicly available satellite cloud image benchmark database for meteorological research, in which weather systems are linked directly with the cloud images. LSCIDMR contains 104 390 high-resolution images, covering 11 classes with two different annotation methods: 1) single-label annotation and 2) multiple-label annotation, called LSCIDMR-S and LSCIDMR-M, respectively. The labels are annotated manually, and we obtain a total of 414 221 multiple labels and 40 625 single labels. Several representative deep learning methods are evaluated on the proposed LSCIDMR, and the results can serve as useful baselines for future research. Furthermore, experimental results demonstrate that it is possible to learn effective deep learning models from a sufficiently large image database for the cloud image classification.

Learning rate of students detecting and annotating pediatric wrist fractures in supervised artificial intelligence dataset preparations.

The use of artificial intelligence (AI) in image analysis is an intensively debated topic in the radiology community these days. AI computer vision algorithms typically rely on large-scale image databases, annotated by specialists. Developing and maintaining them is time-consuming, thus, the involvement of non-experts into the workflow of annotation should be considered. We assessed the learning rate of inexperienced evaluators regarding correct labeling of pediatric wrist fractures on digital radiographs. Students with and without a medical background labeled wrist fractures with bounding boxes in 7,000 radiographs over ten days. Pediatric radiologists regularly discussed their mistakes. We found F1 scores—as a measure for detection rate—to increase substantially under specialist feedback (mean 0.61±0.19 at day 1 to 0.97±0.02 at day 10, p<0.001), but not the Intersection over Union as a parameter for labeling precision (mean 0.27±0.29 at day 1 to 0.53±0.25 at day 10, p<0.001). The times needed to correct the students decreased significantly (mean 22.7±6.3 seconds per image at day 1 to 8.9±1.2 seconds at day 10, p<0.001) and were substantially lower as annotated by the radiologists alone. In conclusion our data showed, that the involvement of undergraduated students into annotation of pediatric wrist radiographs enables a substantial time saving for specialists, therefore, it should be considered.

An Automated Framework Based on Deep Learning for Shark Recognition

The recent progress in deep learning has given rise to a non-invasive and effective approach for animal biometrics. These modern techniques allow researchers to track animal individuals on a large-scale image database. Typical approaches are suited to a closed-set recognition problem, which is to identify images of known objects only. However, such approaches are not scalable because they mis-classify images of unknown objects. To recognize the images of unknown objects as ‘unknown’, a framework should be able to deal with the open set recognition scenario. This paper proposes a fully automatic, vision-based identification framework capable of recognizing shark individuals including those that are unknown. The framework first detects and extracts the shark from the original image. After that, we develop a deep network to transform the extracted image to an embedding vector in latent space. The proposed network consists of the Visual Geometry Group-UNet (VGG-UNet) and a modified Visual Geometry Group-16 (VGG-16) network. The VGG-UNet is utilized to detect shark bodies, and the modified VGG-16 is used to learn embeddings of shark individuals. For the recognition task, our framework learns a decision boundary using a one-class support vector machine (OSVM) for each shark included in the training phase using a few embedding vectors belonging to them, then it determines whether a new shark image is recognized as belonging to a known shark individual. Our proposed network can recognize shark individuals with high accuracy and can effectively deal with the open set recognition problem with shark images.

Human EEG recordings for 1,854 concepts presented in rapid serial visual presentation streams

The neural basis of object recognition and semantic knowledge has been extensively studied but the high dimensionality of object space makes it challenging to develop overarching theories on how the brain organises object knowledge. To help understand how the brain allows us to recognise, categorise, and represent objects and object categories, there is a growing interest in using large-scale image databases for neuroimaging experiments. In the current paper, we present THINGS-EEG, a dataset containing human electroencephalography responses from 50 subjects to 1,854 object concepts and 22,248 images in the THINGS stimulus set, a manually curated and high-quality image database that was specifically designed for studying human vision. The THINGS-EEG dataset provides neuroimaging recordings to a systematic collection of objects and concepts and can therefore support a wide array of research to understand visual object processing in the human brain.

Learning-Based Quality Assessment for Image Super-Resolution

Image Super-Resolution (SR) techniques improve visual quality by enhancing the spatial resolution of images. Quality evaluation metrics play a critical role in comparing and optimizing SR algorithms, but current metrics achieve only limited success, largely due to the lack of large-scale quality databases, which are essential for learning accurate and robust SR quality metrics. In this work, we first build a large-scale SR image database using a novel semi-automatic labeling approach, which allows us to label a large number of images with manageable human workload. The resulting SR Image quality database with Semi-Automatic Ratings (SISAR), so far the largest of SR-IQA database, contains 12 600 images of 100 natural scenes. We train an end-to-end Deep Image SR Quality (DISQ) model by employing two-stream Deep Neural Networks (DNNs) for feature extraction, followed by a feature fusion network for quality prediction. Experimental results demonstrate that the proposed method outperforms state-of-the-art metrics and achieves promising generalization performance in cross-database tests. The SISAR database and DISQ model will be made publicly available to facilitate reproducible research.

A novel image retrieval strategy based on transfer learning and hand-crafted features for wool fabric

The application of image retrieval techniques in industrial settings aims at rapid and accurate retrieval of the same or similar products from an archive to facilitate the production process. Content-based image retrieval and its applications on fabrics focused on the appearance differences and ignored the delicate differences, thus the retrieval fineness cannot meet the demands of industrial applications. In this study, a novel image retrieval strategy was proposed to discriminate appearance differences by the classification model based on transfer learning and realize further discrimination delicate differences by hand-crafted features. The pre-trained model based on ImageNet was fine-tuned to extract features for automatic classification. The images in the wool fabric image database were classified into different categories. Meanwhile, the distribution probabilities were used to build the retrieval strategy. To form the 'feature database' in each category, oriented FAST and rotated BRIEF (ORB) was utilized for feature extractions and the vector of locally aggregated descriptors was adopted for feature aggregation. Ball tree was implemented to search the nearest neighbors for the final results. Based on the above methods, the query image was first classified automatically, and then the images were retrieved based on the retrieval strategy. A large-scale image database of wool fabrics with 82,073 images was built as the benchmark to evaluate the efficacy of the proposed method. Experiments indicated that the proposed strategy is effective for delicate image retrieval. The combination of transfer learning and hand-crafted features can discriminate the appearance and delicate differences, being superior to the existing methods on representing the delicate fabric differences. The proposed method can provide referential assistance for the production crew and reducing manual labor in the factory.

THINGS-fMRI/MEG: A large-scale multimodal neuroimaging dataset of responses to natural object images

A detailed understanding of visual object representations in brain and behavior is fundamentally limited by the number of stimuli that can be presented in any one experiment. Ideally, the space of objects should be sampled in a representative manner, with (1) maximal breadth of the stimulus material and (2) minimal bias in the object categories. Such a dataset would allow the detailed study of object representations and provide a basis for testing and comparing computational models of vision and semantics. Towards this end, we recently developed the large-scale object image database THINGS of more than 26,000 images of 1,854 object concepts sampled representatively from the American English language (Hebart et al., 2019). Here we introduce THINGS-fMRI and THINGS-MEG, two large-scale brain imaging datasets using functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG). Over the course of 12 scanning sessions, 7 participants (fMRI: n = 3, MEG: n = 4) were presented with images from the THINGS database (fMRI: 8,740 images of 720 concepts, MEG: 22,448 images of 1,854 concepts) while they carried out an oddball detection task. To reduce noise, participants’ heads were stabilized and repositioned between sessions using custom head casts. To facilitate the use by other researchers, the data were converted to the Brain Imaging Data Structure format (BIDS; Gorgolewski et al., 2016) and preprocessed with fMRIPrep (Esteban et al., 2018). Estimates of the noise ceiling and general quality control demonstrate overall high data quality, with only small overall displacement between sessions. By carrying out a broad and representative multimodal sampling of object representations in humans, we hope this dataset to be of use for visual neuroscience and computational vision research alike.

Fast and Accurate Ophthalmic Medication Bottle Identification Using Deep Learning on a Smartphone Device

To assess the accuracy and efficacy of deep learning models, specifically convolutional neural networks (CNNs), to identify glaucoma medication bottles. Algorithm development for predicting ophthalmic medication bottles using a large mobile image-based dataset. A total of 3750 mobile images of 5 ophthalmic medication bottles were included: brimonidine tartrate, dorzolamide-timolol, latanoprost, prednisolone acetate, and moxifloxacin. Seven CNN models were initially pretrained on a large-scale image database and subsequently retrained to classify 5 commonly prescribed topical ophthalmic medications using a training dataset of 2250 mobile-phone captured images. The retrained CNN models' accuracies were compared using k-fold cross-validation (k= 10). The top 2 performing CNN models were then embedded into separate iOS apps and evaluated using 1500 mobile images not included in the training dataset. Prediction accuracy, image processing time. Of the 7 CNN architectures, MobileNet v2 yielded the highest k-fold cross-validation accuracy of 0.974 (95% confidence interval [CI], 0.966-0.980) and the shortest average image processing time at 3.45 (95% CI, 3.13-3.77) sec/image. ResNet V2 had the second highest accuracy of 0.961 (95% CI, 0.952-0.969). When the 2 app-embedded CNNs were compared, in terms of accuracy, MobileNet V2, with an image prediction accuracy of 0.86 (95% CI, 0.84-0.88), was significantly greater than ResNet V2, 0.68 (95% CI, 0.66-0.71) (Table1). Sensitivities and specificities varied between medications (Table1). There was no significant difference inaverage imaging processing time, 0.32 (95% CI, 0.28-0.36) sec/image and 0.31 (95% CI, 0.29-0.33) sec/image for MobileNet V2 and ResNet V2, respectively. Information on beta-testing of the iOS app can be found here: https://lin.hs.uci.edu/research/. We have retrained MobileNet V2 to accurately identify ophthalmic medication bottles and demonstrated that this neural network can operate in a smartphone environment. This work serves as a proof-of-concept for the production of a CNN-based smartphone application to empower patients by decreasing risk for error.

Classification of large-scale image database of various skin diseases using deep learning.

The purpose of this study was to develop a deep learning-based computer-aided diagnosis system for skin disease classification using photographic images of patients. The targets are 59 skin diseases, including localized and diffuse diseases captured by photographic cameras, resulting in highly diverse images in terms of the appearance of the diseases or photographic conditions. ResNet-18 is used as a baseline model for classification and is reinforced by metric learning to boost generalization in classification by avoiding the overfitting of the training data and increasing the reliability of CADx for dermatologists. Patient-wise classification is performed by aggregating the inference vectors of all the input patient images. The experiment using 70,196 images of 13,038 patients demonstrated that classification accuracy was significantly improved by both metric learning and aggregation, resulting in patient accuracies of 0.579 for Top-1, 0.793 for Top-3, and 0.863 for Top-5. The McNemar test showed that the improvements achieved by the proposed method were statistically significant. This study presents a deep learning-based classification of 59 skin diseases using multiple photographic images of a patient. The experimental results demonstrated that the proposed classification reinforced by metric learning and aggregation of multiple input images was effective in the classification of patients with diverse skin diseases and imaging conditions.

Research on Industry Question Answering System Based on Near Neighbor Vector Search

In the field of natural language processing, language-based question answering has been widely studied and made great achievements. As an important form of search engine, question answering system can further improve the search quality compared with traditional search engines. Traditional tree-based indexing methods are inefficient in high-dimensional space. This paper focuses on the application of k nearest neighbor search in large-scale image databases, and researches on high-dimensional indexing technology based on vector approximation method. Fast-Hessian detector is used to detect feature points and generate SURF feature description vector; Then, the initial matching point pair is obtained by fast approximate nearest neighbor search algorithm, and then the unidirectional matching results are matched bidirectionally; at the same time, it makes a detailed analysis of the situation that the events in the sentences are verbs of “abstract change”, so as to realize the automatic question and answer after the change of abstract relations between entities.

Faster dense deformable image registration by utilizing both CPU and GPU.

.Purpose: Image registration is an important aspect of medical image analysis and a key component in many analysis concepts. Applications include fusion of multimodal images, multi-atlas segmentation, and whole-body analysis. Deformable image registration is often computationally expensive, and the need for efficient registration methods is highlighted by the emergence of large-scale image databases, e.g., the UK Biobank, providing imaging from 100,000 participants.Approach: We present a heterogeneous computing approach, utilizing both the CPU and the graphics processing unit (GPU), to accelerate a previously proposed image registration method. The parallelizable task of computing the matching criterion is offloaded to the GPU, where it can be computed efficiently, while the more complex optimization task is performed on the CPU. To lessen the impact of data synchronization between the CPU and GPU, we propose a pipeline model, effectively overlapping computational tasks with data synchronization. The performance is evaluated on a brain labeling task and compared with a CPU implementation of the same method and the popular advanced normalization tools (ANTs) software.Results: The proposed method presents a speed-up by factors of 4 and 8 against the CPU implementation and the ANTs software, respectively. A significant improvement in labeling quality was also observed, with measured mean Dice overlaps of 0.712 and 0.701 for our method and ANTs, respectively.Conclusions: We showed that the proposed method compares favorably to the ANTs software yielding both a significant speed-up and an improvement in labeling quality. The registration method together with the proposed parallelization strategy is implemented as an open-source software package, deform.

Fashion Style Classification by Weakly Supervised Learning Using Large-Scale Clothing Image Database

Fashion Style Classification by Weakly Supervised Learning Using Large-Scale Clothing Image Database

CASIA-Face-Africa: A Large-Scale African Face Image Database

Face recognition is a popular and well-studied area with wide applications in our society. However, racial bias had been proven to be inherent in most State Of The Art (SOTA) face recognition systems. Many investigative studies on face recognition algorithms have reported higher false positive rates of African subjects cohorts than the other cohorts. Lack of large-scale African face image databases in public domain is one of the main restrictions in studying the racial bias problem of face recognition. To this end, we collect a face image database namely CASIA-Face-Africa which contains 38,546 images of 1,183 African subjects. Multi-spectral cameras are utilized to capture the face images under various illumination settings. Demographic attributes and facial expressions of the subjects are also carefully recorded. For landmark detection, each face image in the database is manually labeled with 68 facial keypoints. A group of evaluation protocols are constructed according to different applications, tasks, partitions and scenarios. The performances of SOTA face recognition algorithms without re-training are reported as baselines. The proposed database along with its face landmark annotations, evaluation protocols and preliminary results form a good benchmark to study the essential aspects of face biometrics for African subjects, especially face image preprocessing, face feature analysis and matching, facial expression recognition, sex/age estimation, ethnic classification, face image generation, etc. The database can be downloaded from our website.

Re-Visiting Discriminator for Blind Free-Viewpoint Image Quality Assessment

Accurate measurement of perceptual quality is important for various immersive multimedia, which demand real-time quality control or quality-based bench-marking for relevant algorithms. For instance, virtual views rendering in Free-Viewpoint (FV) navigation scenarios is a typical case that introduces challenging distortions, particularly the ones around dis-occluded regions. Existing quality metrics, most of which are targeting for impairments caused by compression or network condition, fail to quantify such non-uniform structure-related distortions. Moreover, the lack of quality databases for such distortions makes it even more challenging to develop robust quality metrics. In this work, a Generative Adversarial Networks based No-Reference (NR) quality Metric, namely GANs-NRM, is proposed. We first present an approach to create masks mimicking dis-occlusions/textureless regions, which is applicable on large-scale 2D image databases publicly available in the computer vision domain. Using these synthetic data, we then train a GANs-based context renderer with the capability of rendering those masked regions. Since the naturalness of the rendered dis-occluded regions strongly relates to the perceptual quality, we assume that the discriminator of the trained GANs has an intrinsic ability for quality assessment. We thus use the features extracted from the discriminator to learn a Bag-of-Distortion-Word (BDW) codebook. We show that a quality predictor can be then well trained using only a small amount of subjective quality data for the FV views rendering. Moreover, in the proposed framework, the discriminator is also adapted as a distortion-detector to locate possible distorted regions. According to the experimental results, the proposed model outperforms significantly the state-of-the-art quality metrics. The corresponding context renderer also shows appealing visualized results over other rendering algorithms.