Google Cloud Vision API Research Articles

Web-based Cooking Recipe Recommender System based on Stocked Groceries

Due to the severity of the recent PM 2.5 and COVID-19 pandemic situations, the work-from-home lifestyle has been widely adopted as a new normal. Consequently, it’s necessarily preferable to cook at home instead of dining out as usual. However, a common problem is that there are unplanned and overstocked grocery items which are usually unrecognized and improperly managed. To address this issue, the "What To Cook" web application was developed with the theoretical application of Term Frequency-Inverted Document Frequency (TF-IDF) calculations to search for recipes based on the stocked groceries and then applied Cosine Similarity to calculate the similarity between each recipe and the stocked grocery items. Users can input the list of own stocked grocery items into the application and then apply the content-based filtering system to recommend recipes that can utilize the stocked grocery items. Additionally, the application supports the image capturing using Google Cloud Vision API. The application also stores the user's cooking history and saves the under interested recipes for future reference. After testing the application in real-world scenarios, it was found to be easy to use with satisfiable results.

Dynamic loss yielding more transferable targeted adversarial examples

Adversarial examples are known to have the property of transferability; as a result, deep neural networks can be compromised by transfer-based attacks in black-box scenarios. Recent studies have shown that targeted adversarial examples are far more difficult to transfer than untargeted adversarial examples. We find that the widely used pattern for targeted attacks (i.e., minimizing the loss with respect to the target label) is insufficient for the transfer scenario. To address this issue, we propose a dynamic loss to yield more transferable targeted adversarial examples. In each iteration of attack optimization, in addition to minimizing the loss with respect to the target label, we dynamically select the top k labels (excluding the target label) with high classification probability to penalize (i.e., maximize) their corresponding loss. As a result, the probability of adversarial examples being classified as target labels will be significantly higher than that of being classified as other labels. Extensive experiments on ImageNet and the Google Cloud Vision API show that the dynamic loss significantly outperforms the traditional loss. For example, in the ensemble-based transfer scenario, the dynamic Cross-Entropy (CE) loss can improve the transferability of targeted adversarial examples by 2∼6 times relative to the traditional CE loss. Code is available at https://github.com/mingcheung/Targeted-Transfer-with-Dynamic-Loss.

Identifying Interesting Sceneries and Objects in A Tropic Forest Through Visitors Employed Photography

Tropical Forest is a unique forest landscape character composed of vegetation with various altitude strata combined with other biotic and abiotic landscape elements. The combination of landscape objects produces a landscape character that is visually attractive to visitors. There are few references to forest sceneries and objects of interest to forest visitors, even though this information is important for forest tourism. This study identified tropical forest sceneries and objects of interest to forest visitors through photographs taken by visitors. The method used is Visitors' Employed Photography (VEP) combined with GPS tracking and artificial intelligence Google Clouds Vision API. In this VEP method, we invited 41 respondents to walk in the "Situ Gede Forest " and directly photograph interesting objects in the field. We obtained 1,206 photos from respondents containing the scenery and objects of the "Situ Gede Forest" and the location of the photo-taking points. The photos were labeled via Google Cloud's Vision API and clustered based on the Photo's label into 15 photo clusters using R Statistics software. The grouping results identified three sceneries and objects that interested the respondents: "View to the standing of trees inside the forest, woodland, or jungle," "Forest scenery with recreational activities and zoo," and "Body of water, edge of the body of water, waterway, and trees." Information on the three characteristics of tropical forest landscapes is beneficial in tropical forest management activities, where landscapes with these three types must be conserved and preserved in the interest of sustainable tourism forest.

ADDA: An Adversarial Direction-Guided Decision-Based Attack via Multiple Surrogate Models

Over the past decade, Convolutional Neural Networks (CNNs) have been extensively deployed in security-critical areas; however, the security of CNN models is threatened by adversarial attacks. Decision-based adversarial attacks, wherein an attacker relies solely on the final output label of the target model to craft adversarial examples, are the most challenging yet practical adversarial attacks. However, existing decision-based adversarial attacks generally suffer from poor query efficiency or low attack success rate, especially for targeted attacks. To address these issues, we propose a query-efficient Adversarial Direction-guided Decision-based Attack (ADDA), which exploits the advantages of transfer-based priors and the benefits of a single query. The transfer-based priors provided by the gradients of multiple different surrogate models can be utilized to suggest the most promising search directions for generating adversarial examples. The query consumption during the ADDA attack is mainly derived from a single query evaluation of the candidate adversarial samples, which significantly saves the number of queries. Experimental results on several ImageNet classifiers, including l∞ and l2 threat models, demonstrate that our proposed approach overwhelmingly outperforms existing state-of-the-art decision-based attacks in terms of both query efficiency and attack success rate. We show case studies of ADDA against a real-world API in which it is successfully able to fool the Google Cloud Vision API after only a few queries.

Fooling Examples: Another Intriguing Property of Neural Networks.

Neural networks have been proven to be vulnerable to adversarial examples; these are examples that can be recognized by both humans and neural networks, although neural networks give incorrect predictions. As an intriguing property of neural networks, adversarial examples pose a serious threat to the secure application of neural networks. In this article, we present another intriguing property of neural networks: the fact that well-trained models believe some examples to be recognizable objects (often with high confidence), while humans cannot recognize such examples. We refer to these as "fooling examples". Specifically, we take inspiration from the construction of adversarial examples and develop an iterative method for generating fooling examples. The experimental results show that fooling examples can not only be easily generated, with a success rate of nearly 100% in the white-box scenario, but also exhibit strong transferability across different models in the black-box scenario. Tests on the Google Cloud Vision API show that fooling examples can also be recognized by real-world computer vision systems. Our findings reveal a new cognitive deficit of neural networks, and we hope that these potential security threats will be addressed in future neural network applications.

SAGE: Steering the Adversarial Generation of Examples With Accelerations

To generate image adversarial examples, state-of-the-art black-box attacks usually require thousands of queries. However, massive queries will introduce additional costs and exposure risks in the real world. Towards improving the attack efficiency, we carefully design an acceleration framework SAGE for existing black-box methods, which is composed of sLocator (initial point optimization) and sRudder (search process optimization). The core idea of SAGE in terms of 1) saliency map can guide the perturbations towards the most adversarial direction and 2) exploiting bounding box (bbox) to capture those salient pixels in the black-box attack. Meanwhile, we provide a series of observations and experiments that demonstrate bbox holds model invariance and process invariance. We extensively evaluate SAGE on four state-of-the-art black-box attacks involving three popular datasets (MNIST, CIFAR10, and ImageNet). The results show that SAGE could present fundamental improvements even against robust models that use adversarial training. Specifically, SAGE could reduce >20% of queries and improve the success rate of attacks to 95%~100%. Compared with the other acceleration framework, SAGE fulfills the more significant effect in a flexible, stable, and low-overhead manner. Moreover, our practical evaluation (Google Cloud Vision API) shows SAGE can be applied to real-world scenarios.

Exploring the Key Design Attributes of Cafés from User-Generated Contents - A Case Study of Two Cafés of Unique Interior Designs in Yeonnam-dong, Seoul -

In the last couple of decades, the use of social media has consistently increased. Different social media platforms allow users to interact and create diverse types of content, making them able to share their opinions. With brands getting involved in these platforms, it is important to know how their image is portrayed by their customers on social media. This includes the café industry. The purpose of our study is twofold. One is to understand which design attributes make a café appealing to social media users, and another is to examine whether the Google Cloud Vision API is able to differentiate the key design attributes using user-generated content, which are Instagram posts in this research. In order to achieve these aims, we conducted a comparative case study using two cafés of unique interior designs located in Yeonnam-dong in Seoul, South Korea. Specifically, Greem Cafe and Perception Coffee were used. Using these cases as hashtags, we scraped the posts using a Python web scraper and screened out the posts irrelevant to the cafés. After, we ran them through the Google Cloud Vision API to obtain the labels and screened out the labels irrelevant to the interior designs, such as people and amenities. Finally, we were able to categorize the label to different design attributes and compare and contrast the labels from two cafés. The main differences shown in the results from the labels are that Greem Café had Color and Cartoon attribute labels like “Black-and-White” and “Drawing” that clearly represent the cartoonish interior design style. On the other hand, Perception Coffee’s most frequent label, “Wood” along with other labels like “Aeolian Landform” identify their wooden ceiling design. With these results, it is shown that the Google Cloud Vision API is able to distinguish the main design elements from both cafés. This research utilizes a new tool that can be useful for future researchers and designers that deal with big data, also this research brings insight for designers at the time of creating new places by analyzing data made by consumers and promotes the usage of social media data as a tool for design thinking.

Correlating fluency theory-based visual aesthetic liking of landscape with landscape types and features

ABSTRACT People inherently assess landscapes by creating spontaneous aesthetic liking judgments based on the surrounding stimuli. To understand these judgements objectively, use may be made of the fluency theory of aesthetic pleasure (the psychological processes through which people experience beauty). This study aims to predict people’s visual aesthetic preferences based on fluency theory and to correlate these preferences with landscape types and features. An ordinary least squares (OLS) regression model was developed to predict visual aesthetic liking, using image statistics as explanatory variables. We determined types of landscape using Landscape Character Assessment (LCA) and applied viewshed analyses distinguishing between near, medium, and far zones. We identified landscape features by content analysis making use of machine learning-based image recognition supplied by Google Cloud Vision API. The results show that vegetation and geological forms were the most significant features for people’s visual aesthetic liking, followed by waterscapes and built structures/human settlements. The viewshed analyses indicated that ‘medium-altitude, low-gradient artificial areas’ were visible in photographs with high aesthetic visual liking in all zones (i.e., at all distances). When the photographs showing this type of landscape are examined, the artificial areas in the photographs turn out to consist mostly of historical buildings or remains. This finding suggests that historical sites are not just important for their cultural value, but for their visual aesthetic value as well.

Fusion of vectored text descriptors with auto extracted deep CNN features for improved image classification

• Transfer learning approach on pre-trained DCNN models for the WANG dataset. • Using word2cev for creating an embedding vector followed by a combination of DSSM and LSTM to reduce the size of a word vector and using these new word vectors to assign five synonyms for each label or keyword for an image to improve classification accuracy. • Fusion of auto extracted features from pre-trained models and word to vector features obtained from word2vec, DSSM, and LSTM for improved content-based image classification (CBIC). In today's age, massive image data is generated rapidly. This influx has made labeling images tedious and, in turn, made it harder to retrieve images through searching algorithms that rely only on labels, keywords, or other meta-data in the images. Modern Content-Based Image Retrieval (CBIR) techniques rely on the visual features within the image to return relevant results to a search query. Deep Convolutional Neural Network (DCNN) models made great strides in the last decade. This paper relies on these complex pre-trained models to extract visual features from images. The proposed work has used pre-trained models like VGG16, MobileNet, Inceptionv3, and Xception for this task. Some studies in the CBIR space also suggest increased accuracy when both visual and textual features are considered. This paper proposes a novel three-step process for obtaining textual features. Firstly, the proposed model receives keywords for each image using Google Cloud Vision API. Secondly, the proposed model replaces each keyword with a 300-dimensional embedding vector obtained using word2vec, trained on the Google News dataset. Finally, the proposed model trains a combination of the Deep Semantic Similarity Model (DSSM) and Long Short-Term Memory (LSTM) model to reduce the 300-dimensional vector to a 64-dimensional vector. Using these new shortened word vectors, the proposed model computes the cosine similarity to replace each keyword of an image with five of its synonyms. Here, these additional steps increased the accuracy compared to simply using a word embedding technique. Finally, the proposed model combined the visual and textual feature vectors and observed that this feature set showed maximum classification accuracy of 98.33%, which is also compared with and found relatively better than other similar model results.

The Psychological Effects of Park Therapy Components on Campus Landscape Preferences

COVID-19 has doubled the prevalence of mental health problems among young adults. In Indonesia, it extends the vulnerability of families, increases economic uncertainty, interrupts food security, and affects psychological well-being. Accordingly, this research examined the correlation between psychological effects and preferred landscape elements. Experiments were conducted in a campus park, arboretum, and a road. Participants captured attractive views during the walk using the Visitor-Employed Photography method and evaluated psychological effects using the Profile of Mood States (POMS) and the State-Trait Anxiety Inventory (STAI) pre-and post-walking. Google Cloud Vision API was used to get the image annotation keywords. The results indicated a correlation between psychological effects and preferred landscape elements. Park therapy components, including plant, flower, and sky, were negatively correlated with negative moods and anxiety levels. These findings presented scientific evidence for the psychological relaxation outcome of walking and prominent components of park therapy to support therapeutic campus greenspace planning.

SMART HEALTH RECORDS

To manage the updated medical history and records of health care is a comprehensive process, and with the time being technology like Vision API, AutoML Google Cloud has done a remarkable role in data storage and retrieval. As we all know that the manual data process, paperwork, and similar kind of documentation have extensive processes that usually take huge time, and effort to complete the majority of the processes takes a lot of time and money to complete. So to reduce the time and efforts at a low cost now we have an excellent technique to cope up with all these issues named machine learning algorithms. Vector machines and ML-based OCR recognition techniques are deliberately collecting potency in Objects and different document classification methods such as Google's Cloud Vision API and MATLAB's machine learning-based handwriting recognition technology. Researchers of MIT are innovating new techniques with the help of Machine Learning which is the succeeding generation of astute, smart health records, by integrating ML-based tools from scratch to encourage with better analysis, diagnosis, superb treatment recommendations.

A Smart Vision Based System for Assisting Visually Impaired People

Eyes are most important sensory organ of a human body. A person is visually impaired if he cannot see objects as clearly as usual. So its difficult for a visually impaired one to live as normal people can do. They always required some assistive devices or assistants to guide them. To address the concerns, the suggested system will perform multiple tasks as quickly as feasible and accurately without requiring any specific talents.The proposed system bring together the deep learning and cloud APIs for smooth functioning of multiple tasks like face recognition, currency recognition, object labeling, text recognition, online news paper reading, current location ,weather and and date and time accessing.The user interact with the system by providing specific voice command like “who is infront of me” and system trigger corresponding module and return voice output. Face recognition is carried based ondlib’s face recognition. Image labeling and text recognition by Google cloud vision API, currency recognition for Indian currency notes is proposed to do on deep transfer learning model Resnet 101 ,online news paper reading is with the help of Google News API, Weather access is by Open Weather MAp API , location can be accessed with the help of IP address and datetime module of python provide current date and time.

Learning to Learn Transferable Attack

Transfer adversarial attack is a non-trivial black-box adversarial attack that aims to craft adversarial perturbations on the surrogate model and then apply such perturbations to the victim model. However, the transferability of perturbations from existing methods is still limited, since the adversarial perturbations are easily overfitting with a single surrogate model and specific data pattern. In this paper, we propose a Learning to Learn Transferable Attack (LLTA) method, which makes the adversarial perturbations more generalized via learning from both data and model augmentation. For data augmentation, we adopt simple random resizing and padding. For model augmentation, we randomly alter the back propagation instead of the forward propagation to eliminate the effect on the model prediction. By treating the attack of both specific data and a modified model as a task, we expect the adversarial perturbations to adopt enough tasks for generalization. To this end, the meta-learning algorithm is further introduced during the iteration of perturbation generation. Empirical results on the widely-used dataset demonstrate the effectiveness of our attack method with a 12.85% higher success rate of transfer attack compared with the state-of-the-art methods. We also evaluate our method on the real-world online system, i.e., Google Cloud Vision API, to further show the practical potentials of our method.

Comparative analysis of Tesseract and Google Cloud Vision for Thai vehicle registration certificate

Optical character recognition (OCR) is a technology to digitize a paper-based document to digital form. This research studies the extraction of the characters from a Thai vehicle registration certificate via a Google Cloud Vision API and a Tesseract OCR. The recognition performance of both OCR APIs is also examined. The 84 color image files comprised three image sizes/resolutions and five image characteristics. For suitable image type comparison, the greyscale and binary image are converted from color images. Furthermore, the three pre-processing techniques, sharpening, contrast adjustment, and brightness adjustment, are also applied to enhance the quality of image before applying the two OCR APIs. The recognition performance was evaluated in terms of accuracy and readability. The results showed that the Google Cloud Vision API works well for the Thai vehicle registration certificate with an accuracy of 84.43%, whereas the Tesseract OCR showed an accuracy of 47.02%. The highest accuracy came from the color image with 1024×768 px, 300dpi, and using sharpening and brightness adjustment as pre-processing techniques. In terms of readability, the Google Cloud Vision API has more readability than the Tesseract. The proposed conditions facilitate the possibility of the implementation for Thai vehicle registration certificate recognition system.

Does transnational contention lead to transnational memory? The online visual memory of the February 2003 anti-Iraq War protests

ABSTRACT In the last weekend of February 2003, between six and ten million people in 60 different countries protested against the imminent Iraq War. Commentators quickly described the worldwide protests as a manifestation of a new form of transnational contention, which would lead to new forms of transnational memory and identity. Building on efforts to connect social movement and memory studies, this article examines if the massive size and transnational nature of the February 2003 anti-war protests resulted in an equally widely shared and transnational form of memory. Following research that describes the crucial role of visual communication for social movements and memory, it approaches this question by studying the size and shape of the online visual memory of the 2003 anti-war protests. Applying the Google Cloud Vision API to retrieve and analyse 1746 online circulations of twenty-five photographs of the 2003 anti-war protests in the period 2003–2019, we find that, compared to other famous images of protest, the online circulation of images of the protests remains limited. This suggests that the massive protests of 2003 did not result in a widely shared and transnational form of memory. Rather than focusing on individual images, this article argues that studying patterns in their online circulation can explain why some movements remain in the public eye, while others quickly disappear from it.

Procrash: A Solution To Procrastination by Limiting Online Distractions using Optical Character Recognition

For this project, I decided to relieve the tension of procrastination that commonly happens in students and adults. To find a solution to this, I created a program that uses Google Cloud Vision API (Optical Character Recognition) to detect the distracting forms of media such as Twitter, YouTube, and Facebook, and counts the number of times the user visits these websites. After a certain number of visits, the program sends a notification to remind the user to stay focused. If the user ignores the notification message while staying on the unapproved website, the program forces the tab to close. This application was applied to a small user study where a qualitative evaluation of the approach was conducted. After collecting data for two weeks, it concluded that the program was able to effectively reduce and limit the uses of online distractions, allowing the user to manage their time more efficiently by staying off websites they should not visit.

Passive citizen science: The role of social media in wildlife observations.

Citizen science plays an important role in observing the natural environment. While conventional citizen science consists of organized campaigns to observe a particular phenomenon or species there are also many ad hoc observations of the environment in social media. These data constitute a valuable resource for 'passive citizen science'-the use of social media that are unconnected to any particular citizen science program, but represent an untapped dataset of ecological value. We explore the value of passive citizen science, by evaluating species distributions using the photo sharing site Flickr. The data are evaluated relative to those submitted to the National Biodiversity Network (NBN) Atlas, the largest collection of species distribution data in the UK. Our study focuses on the 1500 best represented species on NBN, and common invasive species within UK, and compares the spatial and temporal distribution with NBN data. We also introduce an innovative image verification technique that uses the Google Cloud Vision API in combination with species taxonomic data to determine the likelihood that a mention of a species on Flickr represents a given species. The spatial and temporal analyses for our case studies suggest that the Flickr dataset best reflects the NBN dataset when considering a purely spatial distribution with no time constraints. The best represented species on Flickr in comparison to NBN are diurnal garden birds as around 70% of the Flickr posts for them are valid observations relative to the NBN. Passive citizen science could offer a rich source of observation data for certain taxonomic groups, and/or as a repository for dedicated projects. Our novel method of validating Flickr records is suited to verifying more extensive collections, including less well-known species, and when used in combination with citizen science projects could offer a platform for accurate identification of species and their location.

The effect of facial expressions on crowdfunding performance

We analyze how the facial expressions of people participating in loan-based crowdfunding affect their funding performance. Previously, researchers who studied facial expressions had to rely on a small number of pictures because coding pictures required human input, which is costly. In this study, we utilize Google cloud vision API which allows us to detect the facial expressions from the large number of pictures. We examine the effects of people

Latent Twitter Image Information for Social Analytics

The appearance of images in social messages is continuously increasing, along with user engagement with that type of content. Analysis of social images can provide valuable latent information, often not present in the social posts. In that direction, a framework is proposed exploiting latent information from Twitter images, by leveraging the Google Cloud Vision API platform, aiming at enriching social analytics with semantics and hidden textual information. As validated by our experiments, social analytics can be further enriched by considering the combination of user-generated content, latent concepts, and textual data extracted from social images, along with linked data. Moreover, we employed word embedding techniques for investigating the usage of latent semantic information towards the identification of similar Twitter images, thereby showcasing that hidden textual information can improve such information retrieval tasks. Finally, we offer an open enhanced version of the annotated dataset described in this study with the aim of further adoption by the research community.

Recognition and Classification of Homogeneous Landscape With Visitor–Employed Photography and Cloud Image Annotation API—An Example of the Riverscape in Nihonbashi, Tokyo, Japan

Effective classification of landscape photographs is a vital step in data processing and environment analysis. With the popularity of crowdsourcing geo-information, an increasing number of studies have used geotagged photographs to visualize how people perceive and interact with destinations and explore the aesthetic, cultural, and recreational value of the areas. In recent years, machine-learning algorithms for image recognition have dramatically improved the efficiency of the assignment of keywords and provide possibilities for the automatic classification of numerous photographs. However, the applicability of such methods for the practical landscape classification is still not clear, especially for the photographs presenting a homogeneous landscape that has similar characteristics. This study developed a semi-automatic classifier for homogeneous landscape photographs by using Google Cloud Vision API and multi-level hierarchical clustering. The classifier was applied to the classification of urban riverscape photographs, which is a typical example of homogeneous landscapes in Nihonbashi, Tokyo, Japan. The riverscapes can be classified into 9 characteristic groups by the classifier and the visual impression of these groups matches well with our intuitive feeling. A confusion matrix showed that the overall accuracy was 82.61%, indicating a strong agreement between the classifier and manual classification. Therefore, the classifier is practical for classifying homogeneous riverscape photographs. Such methodology also provides the possibility of public participation in the assessing process, which, in turn, contributes to urban tourism management.