Semi-automatic Annotation Research Articles

Machine learning based quantitative characterization of microstructures

Identifying structure–property relationships of polycrystalline microstructures demands an accurate and precise quantification of their features. Measuring grain sizes is tedious and creates a non-transparent bias when being performed by different individuals, impairing comparability with references. Here, we present a novel use of region-based convolutional neural networks (R-CNNs) to quantify several microstructural characteristics and their distributions: Feret diameter, axis length, area, circumference, dihedral angle and coordination number. We utilize a two-step approach: (i) a semi-automatic annotation tool to generate training data for (ii) a fully automated R-CNN, quantitatively evaluating images. Using Al-doped ZnO as a model system, we trained two R-CNNs, one for ZnO and one for precipitated ZnAl2O4. The R-CNN performs well in evaluating grain size characteristics from images with low contrast and in differentiating uni-, and bimodal grain size distributions on the sub-micron, and nanoscale. An extended statistical analysis of the distributions is performed to extract microstructural parameters quantitatively. This innovative solution makes grain size measuring amenable, time-effective, less biased, consistent, and statistically more precise.

A semi-automatic annotation methodology that combines Summarization and Human-In-The-Loop to create disinformation detection resources

Early detection of disinformation is one of the most challenging big-scale problems facing present day society. This is why the application of technologies such as Artificial Intelligence and Natural Language Processing is necessary. The vast majority of Artificial Intelligence approaches require annotated data, and generating these resources is very expensive. This proposal aims to improve the efficiency of the annotation process with a two-level semi-automatic annotation methodology. The first level extracts relevant information through summarization techniques. The second applies a Human-in-the-Loop strategy whereby the labels are pre-annotated by the machine, corrected by the human and reused by the machine to retrain the automatic annotator. After evaluating the system, the average annotation time per news item is reduced by 50%. In addition, a set of experiments on the semi-automatically annotated dataset that is generated are performed so as to demonstrate the effectiveness of the proposal. Although the dataset is annotated in terms of unreliable content, it is applied to the veracity detection task with very promising results (0.95 accuracy in reliability detection and 0.78 in veracity detection).

Semi-automatic classification of digital heritage on the Aïoli open source 2D/3D annotation platform via machine learning and deep learning

Today, digital tools represent a privileged means of describing and understanding of cultural heritage. However, existing hybrid methods for semantic annotation on digital models are mostly manual and require considerable human intervention. This may cause data dispersion or data alteration issues, preventing the digital collection of multidisciplinary heritage information in collaborative valorization and conservation projects. Tackling this problem, this work presents three different methods for more automated 2D/3D data annotation, making the most of Machine Learning or Deep Learning and of the collaborative, web-based annotation platform Aïoli. The proposed approaches are designed to assist heritage experts in the hybrid annotation of architectural objects, e.g., to identify building components, degradation patterns, mapping of alterations, materials etc. and to share and access 2D/3D information via the web. The work aims at providing public and private stakeholders in charge of restoration and conservation activities with a set of more automated annotation tools, allowing to: i) preserve data in the transition to different representation types; ii) propagate analytically relevant information via digital models and/or images; iii) share heritage-related data on the web with an open source strategy. The validated workflows sketch the outlines of a dedicated and user-friendly toolbar for semi-automatic annotation, directly integrated into the Aïoli user interface, where automated annotation tools are readily available to the community of heritage professionals to properly support documentation and restoration activities.

EESD special issue: AI and data‐driven methods in earthquake engineering – (Part 1)

EESD special issue: AI and data‐driven methods in earthquake engineering – (Part 1)

Multi-Context Grouped Attention for Unsupervised Person Re-Identification

Recent advancements like multiple contextual analysis, attention mechanisms, distance-aware optimization, and multi-task guidance have been widely used for supervised person re-identification (ReID), but the implementation and effects of such methods in unsupervised ReID frameworks are non-trivial and unclear, respectively. Moreover, with increasing size and complexity of image-and video-based ReID datasets, manual or semi-automated annotation procedures for supervised ReID are becoming labor intensive and cost prohibitive, which is undesirable especially considering the likelihood of annotation errors increase with scale/complexity of data collections. Therefore, we propose a new iterative clustering framework that is insensitive to annotation errors and over-fitting ReID annotations (i.e., labels). Our proposed unsupervised framework incorporates (a) a novel multi-context group attention architecture that learns a holistic attention map from multiple local and global contexts, (b) an unsupervised clustering loss function that down-weights easily discriminative identities, and (c) a background diversity term that helps cluster persons across different cross-camera views without leveraging any identification or camera labels. We perform extensive analysis using the DukeMTMC-VideoReID and MARS video-based ReID datasets and the MSMT17 image-based ReID dataset. Our approach is shown to provide a new state-of-the-art performance for unsupervised ReID, reducing the rank-1 performance gap between supervised and unsupervised ReID to 1.1%, 12.1%, and 21.9% from 6.1%, 17.9%, and 22.6% for DukeMTMC, MARS, and MSMT17 datasets, respectively.

An innovative traffic light recognition method using vehicular ad-hoc networks

Car congestion is a pressing issue for everyone on the planet. Car congestion can be caused by accidents, traffic lights, rapid accelerations, deceleration, and hesitation of drivers, as well as a small low-carrying capacity road without bridges. Increasing road width and constructing roundabouts and bridges are solutions to car congestion, but the cost is significant. TLR (traffic light recognition) reduces accidents and traffic congestion caused by traffic lights (TLs). Image processing with convolutional neural network (CNN) lakes dealing with harsh weather. A semi-automatic annotation for traffic light detection employs a global navigation satellite system, raising the cost of automobiles. Data was not collected in harsh conditions, and tracking was not supported. Integrated channel feature tracking (ICFT) combines detection and tracking, but it does not support sharing information with neighbors. This study used vehicular ad-hoc networks (VANETs) for VANET traffic light recognition (VTLR). Information exchange as well as monitoring of the TL status, time remaining before a change, and recommended speeds are supported. Based on testing, it has been determined that VTLR performs better than semi-automatic annotation, image processing with CNN, and ICFT in terms of delay, success ratio, and the number of detections per second.

A forum theater corpus for discrimination awareness

In this article, we present a new multimodal annotated corpus of forum theater scenes on discrimination awareness. The collected interactions include scenes of ordinary sexism and racism situations played out by different actors in different contexts. The corpus also contains scenes of interactions between an author of discriminatory behavior and a witness trying to make the discriminatory actor aware of their behavior. These confrontations scenes have been played considering different social attitudes (denial, aggressive, and conciliatory). The experimental setup, including motion capture and audio-visual recordings, has been specifically designed to allow the semi-automatic annotation of the corpus and a fine-grained analysis of the multimodal cues associated with social behaviors.

Evaluation of an atlas-defined semi-automatic annotation workflow for digital neuropathology: a pilot study in ataxia telangiectasia mouse model

Background: Neuropathology is a discipline constantly suffering from scarcity of staff specialists. Digital pathology-empowered analysis emerges as the technology to improve the efficiency of this discipline. Nevertheless, the uniqueness of neuroanatomy hinders workflow development. Here, we evaluated a new workflow developed by EU-funded EBRAIN project, QUINT, which enables semi-automatic, reference atlas-defined brain image annotation, cell segmentation, and quantification. QUINT was benchmarked against the well established QuPath manual platform using whole slide image(WSI) of Ataxia Telangiectasia(ATM)-knockout mouse models.

A multimodal corpus to study videoconference interactions for techno-pedagogical competence in second language acquisition and teacher education

This article aims to describe the construction and annotation of a multimodal and multilingual (French, English, and Mandarin Chinese) corpus for the study of second language acquisition and professional development in language teaching. The corpus was built within a research project whose main objective is to determine which techno-semio-pedagogical competencies are developed informally and which ones require formal training. In this paper, we explain the theoretical framework adopted, characterized by an ecological approach to the interactive environment. We subsequently illustrate the procedures for the collection of audio, video and eye-tracking data. We also provide details of the annotation of raw data to produce a corpus of analysis drawing on different automatic and semi-automatic annotation tools. Finally, we explain how such a corpus enables us to study the development of learners’ multimodal competence and tutors’ techno-pedagogical competence and point to some other possible research questions.

Active Learning Based on Transfer Learning Techniques for Text Classification

Text preprocessing is a common task in machine learning applications that involves hand-labeling sets. Although automatic and semi-automatic annotation of text data is a growing field, researchers need to develop models that use resources as efficiently as possible for a learning task. The goal of this work was to learn faster with fewer resources. In this paper, the combination of active and transfer learning was examined with the purpose of developing an effective text categorization method. These two forms of learning have proven their efficiency and capacity to train correct models with substantially less training data. We considered three types of criteria for selecting training points: random selection, uncertainty sampling criterion and active transfer selection. Experimental evaluation was performed on five data sets from different domains. The findings of the experiments suggest that by combining active and transfer learning, the algorithm performs better with fewer labels than random selection of training points.

A Query Language for Exploratory Analysis of Video-Based Tracking Data in Padel Matches

Recent advances in sensor technologies, in particular video-based human detection, object tracking and pose estimation, have opened new possibilities for the automatic or semi-automatic per-frame annotation of sport videos. In the case of racket sports such as tennis and padel, state-of-the-art deep learning methods allow the robust detection and tracking of the players from a single video, which can be combined with ball tracking and shot recognition techniques to obtain a precise description of the play state at every frame. These data, which might include the court-space position of the players, their speeds, accelerations, shots and ball trajectories, can be exported in tabular format for further analysis. Unfortunately, the limitations of traditional table-based methods for analyzing such sport data are twofold. On the one hand, these methods cannot represent complex spatio-temporal queries in a compact, readable way, usable by sport analysts. On the other hand, traditional data visualization tools often fail to convey all the information available in the video (such as the precise body motion before, during and after the execution of a shot) and resulting plots only show a small portion of the available data. In this paper we address these two limitations by focusing on the analysis of video-based tracking data of padel matches. In particular, we propose a domain-specific query language to facilitate coaches and sport analysts to write queries in a very compact form. Additionally, we enrich the data visualization plots by linking each data item to a specific segment of the video so that analysts have full access to all the details related to the query. We demonstrate the flexibility of our system by collecting and converting into readable queries multiple tips and hypotheses on padel strategies extracted from the literature.

Instance segmentation of soft‐story buildings from street‐view images with semiautomatic annotation

AbstractIn high seismic risk regions, it is important for city managers and decision makers to create programs to mitigate the risk for buildings. For large cities and regions, a mitigation program relies on accurate information of building stocks, that is, a database of all buildings in the area and their potential structural defects, making them vulnerable to strong ground shaking. Structural defects and vulnerabilities could manifest via the building's appearance. One such example is the soft‐story building—its vertical irregularity is often observable from the facade. This structural type can lead to severe damage or even collapse during moderate or severe earthquakes. Therefore, it is critical to screen large building stock to find these buildings and retrofit them. However, it is usually time‐consuming to screen soft‐story structures by conventional methods. To tackle this issue, we used full image classification to screen them out from street view images in our previous study. However, full image classification has difficulties locating buildings in an image, which leads to unreliable predictions. In this paper, we developed an automated pipeline in which we segment street view images to identify soft‐story buildings. However, annotated data for this purpose is scarce. To tackle this issue, we compiled a dataset of street view images and present a strategy for annotating these images in a semi‐automatic way. The annotated dataset is then used to train an instance segmentation model that can be used to detect all soft‐story buildings from unseen images.

Untargeted lipidomic profiling of grapes highlights the importance of modified lipid species beyond the traditional compound classes

The aim of this paper is to provide a detailed characterisation of grape lipidome. To achieve this objective, it starts by describing a pipeline implemented in R software to allow the semi-automatic annotation of the detected lipid species. It also provides an extensive description of the different properties of each molecule (such as retention time dependencies, mass accuracy, adduct formation and fragmentation patterns), which allowed the annotations to be made more accurately. Most annotated lipids in the grape samples were (lyso)glycerophospholipids and glycerolipids, although a few free fatty acids, hydroxyceramides and sitosterol esters were also observed. The proposed pipeline also allowed the identification of a series of methylated glycerophosphates never previously observed in grapes. The current results highlight the importance of expanding chemical analyses beyond the classical lipid categories.

A Visualization System for Assistance of Semi-Automated Annotation Tasks based on Impression Evaluations

画像の印象を推測する機械学習の構築において一般的に，学習に使用する訓練データ作成は多数の画像にアノテーションを付与する作業が必要である．しかし作業者が有する個人の印象回答が学習結果に依存する問題がある．本研究では画像から受ける印象の個人差を低減し，かつ作業者の負担を低減するために，多人数の印象回答値を機械学習した結果にそって画像の印象のアノテーション付与を半自動化するシステムと，その過程を可視化する手法を提案する．本手法ではまずSD法を採用した印象評価を実施し，続いて各画像の印象値を用いてファジィ決定木を生成する．このファジィ決定木によって画像を自動分類したのち，その結果と過程を可視化することで，ユーザによる画像再分類を支援する．決定木の表示と類似画像の一覧表示を連動させることで，決定木の可読性を向上させ，アノテーション付与の傾向理解を促す．本論文では，1500枚の女性着衣服画像に43人の作業者が印象評価を実施したデータを題材として，本手法による可視化の実行例を示すとともに，ユーザ実験による本研究の有用性を検証する．

An Efficient Method for Detecting Asphalt Pavement Cracks and Sealed Cracks Based on a Deep Data-Driven Model

Thanks to the development of deep learning, the use of data-driven methods to detect pavement distresses has become an active research field. This research makes four contributions to address the problem of efficiently detecting cracks and sealed cracks in asphalt pavements. First, a dataset of pavement cracks and sealed cracks is created, which consists of 10,400 images obtained by a vehicle equipped with a highway condition monitor, with 202,840 labeled distress instances included in these pavement images. Second, we develop a dense and redundant crack annotation method based on the characteristics of the crack images. Compared with traditional annotation, the method we propose generates more object instances, and the localization is more accurate. Next, to achieve efficient crack detection, a semi-automatic crack annotation method is proposed, which reduces the working time by 80% compared with fully manual annotation. Finally, comparative experiments are conducted on our dataset using 13 currently prevailing object detection algorithms. The results show that dense and redundant annotation is effective; moreover, cracks and sealed cracks can be efficiently and accurately detected using the YOLOv5 series model and YOLOv5s is the most balanced model with an F1-score of 86.79% and an inference time of 14.8ms. The pavement crack and sealed crack dataset created in this study is publicly available.

PFN-PT

This article presents PFN-PT, a robust system for the automatic semantic annotation of Portuguese, consisting of a new, parsing-oriented framenet and a rule-based frame- and role-tagger. The framenet provides almost 13,000 valency frames covering 7,300 verb lemmas with 10,700 senses. Frame and role tagging is achieved by iterated matching of syntactic structures and semantic noun types with slot-filler conditions in the framenet. We discuss design principles and present frame and role statistics. In an evaluation run on news data, the system achieved an overall F-score of 92.2% for frame senses.

MTab4D: Semantic annotation of tabular data with DBpedia

Semantic annotation of tabular data is the process of matching table elements with knowledge graphs. As a result, the table contents could be interpreted or inferred using knowledge graph concepts, enabling them to be useful in downstream applications such as data analytics and management. Nevertheless, semantic annotation tasks are challenging due to insufficient tabular data descriptions, heterogeneous schema, and vocabulary issues. This paper presents an automatic semantic annotation system for tabular data, called MTab4D, to generate annotations with DBpedia in three annotation tasks: 1) matching table cells to entities, 2) matching columns to entity types, and 3) matching pairs of columns to properties. In particular, we propose an annotation pipeline that combines multiple matching signals from different table elements to address schema heterogeneity, data ambiguity, and noisiness. Additionally, this paper provides insightful analysis and extra resources on benchmarking semantic annotation with knowledge graphs. Experimental results on the original and adapted datasets of the Semantic Web Challenge on Tabular Data to Knowledge Graph Matching (SemTab 2019) show that our system achieves an impressive performance for the three annotation tasks. MTab4D’s repository is publicly available at https://github.com/phucty/mtab4dbpedia.

Mathematical Methods for IoT-Based Annotating Object Datasets with Bounding Boxes

Object datasets used in the construction of object detectors are typically annotated with horizontal or oriented bounding rectangles for IoT-based. The optimality of an annotation is obtained by fulfilling two conditions: (i) the rectangle covers the whole object and (ii) the area of the rectangle is minimal. Building a large-scale object dataset requires annotators with equal manual dexterity to carry out this tedious work. When an object is horizontal for IoT-based, it is easy for the annotator to reach the optimal bounding box within a reasonable time. However, if the object is oriented, the annotator needs additional time to decide whether the object will be annotated with a horizontal rectangle or an oriented rectangle for IoT-based. Moreover, in both cases, the final decision is not based on any objective argument, and the annotation is generally not optimal. In this study, we propose a new method of annotation by rectangles for IoT-based, called robust semi-automatic annotation, which combines speed and robustness. Our method has two phases. The first phase consists in inviting the annotator to click on the most relevant points located on the contour of the object. The outputs of the first phase are used by an algorithm to determine a rectangle enclosing these points. To carry out the second phase, we develop an algorithm called RANGE-MBR, which determines, from the selected points on the contour of the object, a rectangle enclosing these points in a linear time. The rectangle returned by RANGE-MBR always satisfies optimality condition (i). We prove that the optimality condition (ii) is always satisfied for objects with isotropic shapes. For objects with anisotropic shapes, we study the optimality condition (ii) by simulations. We show that the rectangle returned by RANGE-MBR is quasi-optimal for the condition (ii) and that its performance increases with dilated objects, which is the case for most of the objects appearing on images collected by aerial photography.

Semi-automatic data annotation based on feature-space projection and local quality metrics: An application to cerebral emboli characterization.

We propose a semi-supervised learning approach to annotate a dataset with reduced requirements for manual annotation and with controlled annotation error. The method is based on feature-space projection and label propagation using local quality metrics. First, an auto-encoder extracts the features of the samples in an unsupervised manner. Then, the extracted features are projected by a t-distributed stochastic neighbor embedding algorithm into a two-dimensional (2D) space. A selection of the best 2D projection is introduced based on the silhouette score. The expert annotator uses the obtained 2D representation to manually label samples. Finally, the labels of the labeled samples are propagated to the unlabeled samples using a K-nearest neighbor strategy and local quality metrics. We compare our method against semi-supervised optimum-path forest and K-nearest neighbor label propagation (without considering local quality metrics). Our method achieves state-of-the-art results on three different datasets by labeling more than 96% of the samples with an annotation error from 7% to 17%. Additionally, our method allows to control the trade-off between annotation error and number of labeled samples. Moreover, we combine our method with robust loss functions to compensate for the label noise introduced by automatic label propagation. Our method allows to achieve similar, and even better, classification performances compared to those obtained using a fully manually labeled dataset, with up to 6% in terms of classification accuracy.

LinkingPark: An automatic semantic table interpretation system

In this paper, we present LinkingPark, an automatic semantic annotation system for tabular data to knowledge graph matching. LinkingPark is designed as a modular framework which can handle Cell-Entity Annotation (CEA), Column-Type Annotation (CTA), and Columns-Property Annotation (CPA) altogether. It is built upon our previous SemTab 2020 system, which won the 2nd prize among 28 different teams after four rounds of evaluations. Moreover, the system is unsupervised, stand-alone, and flexible for multilingual support. Its backend offers an efficient RESTful API for programmatic access, as well as an Excel Add-in for ease of use. Users can interact with LinkingPark in near real-time, further demonstrating its efficiency.