And-Or Graph Research Articles

Commonsense Knowledge-Driven Joint Reasoning Approach for Object Retrieval in Virtual Reality

National Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI), China Retrieving out-of-reach objects is a crucial task in virtual reality (VR). One of the most commonly used approaches for this task is the gesture-based approach, which allows for bare-hand, eyes-free, and direct retrieval. However, previous work has primarily focused on assigned gesture design, neglecting the context. This can make it challenging to accurately retrieve an object from a large number of objects due to the one-to-one mapping metaphor, limitations of finger poses, and memory burdens. There is a general consensus that objects and contexts are related, which suggests that the object expected to be retrieved is related to the context, including the scene and the objects with which users interact. As such, we propose a commonsense knowledge-driven joint reasoning approach for object retrieval, where human grasping gestures and context are modeled using an And-Or graph (AOG). This approach enables users to accurately retrieve objects from a large number of candidate objects by using natural grasping gestures based on their experience of grasping physical objects. Experimental results demonstrate that our proposed approach improves retrieval accuracy. We also propose an object retrieval system based on the proposed approach. Two user studies show that our system enables efficient object retrieval in virtual environments (VEs).

An effective process design intent inference method of process data via integrating deep learning and grammar parsing

With the constantly generation of process data in industries, process knowledge reuse and learning from process dataset has shown promising potential in machining process planning. However, existing methods cannot effectively extract different granularity process intents, and thus the process data cannot be effectively utilized. In this paper, an effective process design intent inference method of process data via integrating deep learning and grammar parsing is proposed. First, based on the projection of cutter location segments, an association approach between machining operations and machining features is proposed to extract the process intent of each machining operation. Then, a deep learning based working step process intent sequence decision-making model is proposed to estimate the probability distributions of alternative working step process intents at each time. Moreover, according to the working step process intent sequences of existing parts in an industry, the process knowledge And-Or graph (PK-AOG) is constructed by introducing the probabilistic grammar graph model. Finally, the optimal process intents of each working step are jointly obtained based on grammar parsing by taking the PK-AOG as a guide. In the experiments, the effectiveness of our approach is validated by developing a prototype system, and the results show that our approach is practical in process intents extraction.

Data-Driven and Knowledge-Guided Approach for NC Machining Process Planning

With the wide applications of computer-aided technologies in industries, plenty of process data composed of 3D CAD models and CAM models are increasingly generated, which are the direct and effective carriers of knowledge, intelligence, and experience of skilled designers. However, existing methods cannot effectively learn and mine the implicit and explicit knowledge embedded in process data, and the process scheme generated is difficult to satisfy the manufacturing logics of a part in an industry. In this paper, a data-driven and knowledge-guided approach for NC machining process planning is proposed. First, a multi-level structured NC machining process model based on sub-machining regions is introduced to represent the relations between the 3D CAD model and the CAM model in process data. Then, an attention-based working step sequence generation model is proposed to predict the candidate working step and its associated sub-machining regions. Moreover, a context-free grammar describing the semantic and temporal relationships between working steps is extracted to construct a process knowledge And-Or graph with hierarchy and compositionality (PK-AOG). Finally, guided by PK-AOG, an NC machining process scheme generation method based on grammar parser is proposed to find a joint optimal solution according to the probabilities of the candidate working steps iteratively. A prototype system based on CATIA has been developed to verify the effectiveness of the proposed method.

Suitability of CAD-based Disassembly Sequence Plans for Flexible Remanufacturing

When products reach the end-of-first-life (EoFL), often recovery of valuable components is ignored because of unprofitable disassembly processes. Extracting the wanted component is a highly manual try-and-error process of disassembling multiple components. The creation of dis- and reassembly plans (e.g. for workers or robots) based on CAD files is currently too labour extensive and individually dependent on the EoFL state. Several studies exist for the automatic generation of assembly plans by CAD files with collision-based approaches. However, whether these approaches are suitable for industrial remanufacturing planning has not been discovered. This research compares the suitability of an AND/OR graph (AOG) and a precedence graph (PG) approach on ten exemplary examples. The suitability is measured with graph-based and visual feasibility checks as well as performance measures. The AOG approach is more promising than the PG approach because it considers the flexible formation of subassemblies and thus, better flexibility for scheduling in mixed-model disassembly lines. The results contribute to automatically deriving disassembly and reassembly plans for lot size one and saving labour-extensive high-wage planning effort. Overall, the resulting quality of assembly plans is highly dependent on the CAD model quality. Only two of the ten models resulted in suitable assembly plans. A preliminary CAD quality control during the design phase is needed to improve the applicability of the examined approaches.

Distracted driving detection based on the fusion of deep learning and causal reasoning

Distracted driving is one of the key factors that cause drivers to ignore potential road hazards and then lead to accidents. Existing efforts in distracted behavior recognition are mainly based on deep learning (DL) methods, which identifies distracted behaviors by analyzing static characteristics of images. However, the convolutional neural network (CNN) — based DL methods lack the causal reasoning ability for behavior patterns. The uncertainty of driving behaviors, noise of the collected data, and occlusion between body agents, bring additional challenges to existing DL methods to recognize distracted behaviors continuously and accurately. Therefore, in this paper, we propose a distracted behavior recognition method based on the Temporal–Spatial double-line DL network (TSD-DLN) and causal And-or graph (C-AOG). TSD-DLN fuses the attention feature extracted from the dynamic optical flow information and the spatial feature of the single video frame to recognize the distracted driving posture. Furthermore, a causal knowledge fence based on C-AOG is fused with TSD-DLN to improve the recognition robustness. The C-AOG represents the causality of behavior state fluent change and adopts counterfactual reasoning to suppress behavior recognition failures caused by frame features distortion or occlusion between body agents. We compared the performance of the proposed method with other state-of-the-art (SOTA) DL methods on two public datasets and self-collected dataset. Experimental results demonstrate that proposed method significantly outperforms other SOTA methods when acquiring distracted driving behavior by processing consecutive frames. In addition, the proposed method exhibits accurate continuous recognition and robustness under incomplete observation scenarios.

CAD-based AND/OR Graph Generation Algorithms in (Dis)assembly Sequence Planning of Complex Products

Several studies in production engineering and operations research focused on disassembly and assembly planning for improving the profitability of remanufacturing. The representation of feasible (dis)assembly sequences by analyzing geometrical and technical precedence constraints is an essential necessity of (dis)assembly planning. An AND/OR graph (AOG) is commonly used to represent a product’s feasible (dis)assembly sequences, including alternative subassemblies and parallel operations. In recent years, researchers studied the automatic extraction of geometrical precedence constraints by collision analysis within 3D models. However, since most of the existing approaches focused on collision analysis to identify complex precedence constraints, the generation of AOGs for complex products from collision analysis results remains too inefficient for industrial use cases. In this study, a Computer-Aided Design (CAD) interface from previous studies is used to extract liaison and moving wedge information from 3D models. A top-down and a bottom-up approach for generating complete AOGs from the extracted data are introduced. The approaches’ computing performances are analyzed on sample test cases and CAD models. The bottom-up approach performed better for the given samples. It has been found that the amount of moving wedge constraints has a strong effect on computing performance. It is a strong indicator to estimate the complexity of products under examination. The exponential behavior of the needed computing resources can be estimated beforehand. For complex products, graph simplification or alternative graph representations with less information richness should be considered. The results contribute to automated (dis)assembly sequence planning from complex products and, consequently, increasing remanufacturing profitability.

Patching interpretable And‐Or‐Graph knowledge representation using augmented reality

AbstractWe present a novel augmented reality (AR) interface to provide effective means to diagnose a robot's erroneous behaviors, endow it with new skills, and patch its knowledge structure represented by an And‐Or‐Graph (AOG). Specifically, an AOG representation of opening medicine bottles is learned from human demonstration and yields a hierarchical structure that captures the spatiotemporal compositional nature of the given task, which is highly interpretable for the users. Through a series of psychological experiments, we demonstrate that the explanations of a robotic system, inherited from and produced by the AOG, can better foster human trust compared to other forms of explanations. Moreover, by visualizing the knowledge structure and robot states, the AR interface allows human users to intuitively understand what the robot knows, supervise the robot's task planner, and interactively teach the robot with new actions. Together, users can quickly identify the reasons for failures and conveniently patch the current knowledge structure to prevent future errors. This capability demonstrates the interpretability of our knowledge representation and the new forms of interactions afforded by the proposed AR interface.

Planning optimised multi-tasking operations under the capability for parallel machining

The advent of advanced multi-tasking machines (MTMs) in the metalworking industry has provided the opportunity for more efficient parallel machining as compared to traditional sequential processing. It entailed the need for developing appropriate reasoning schemes for efficient process planning to take advantage of machining capabilities inherent in these machines. This paper addresses an adequate methodical approach for a non-linear process planning with a variety of alternatives, enabled through the STEP_NC standard. A relevant algorithmic approach of high efficacy is developed for feature clustering and operation sequencing based on AND-OR graph modelling. It involves a discrete modelling scheme for setup formulation so that the workload of machine spindles is levelled and its total cycle time is minimized. The so-formulated optimization problem, and related in particular to feature distribution among setups, can be successfully solved by a non-linear generalized reduced gradient (GRC) algorithm. The solution algorithms outlined can be relatively readily implemented in industrial informatics systems of small and medium-sized manufacturing enterprises. The entire methodical approach is validated through illustrative case studies based on exemplary mill-turn parts.

A Flexible and Explainable Vehicle Motion Prediction and Inference Framework Combining Semi-Supervised AOG and ST-LSTM

Accurate trajectory prediction of surrounding vehicles is important for automated vehicles. To solve several existing problems of maneuver-based trajectory prediction, we propose four targeted solutions and establish a trajectory prediction model that integrates semi-supervised And-or Graph (AOG) and Spatio-temporal LSTM (ST-LSTM). To reduce the dependence on the well-labeled dataset, we introduce the concept of sub-maneuvers to improve the classifications of vehicle movements based on the given rough maneuver labels. AOG is used as the backbone of the probabilistic motion inference considering sub-maneuvers. We only define the basic units and inference logics of AOG and design a semi-supervised approach to directly learn the sub-maneuvers and the inference model structure from the training data, without manually specifying the structure (layers and nodes) of the inference model. This approach helps to avoid excessive artificial design or biases. The learned hierarchical motion inference model improves the interpretability of the overall trajectory prediction process. To utilize vehicle interaction information and further yield more accurate prediction, we adopt two different methods to consider vehicle interaction in the two sub-models (maneuver recognition and trajectory prediction). The experiment on NGSIM I-80 dataset shows that the maneuver-based model proposed in this paper (AOG-ST and refined AOG-ST-TB) performs more accurate trajectory prediction results. Although the AOG-ST seems clumsy and slow, we show that it is a flexible and quick model for trajectory prediction for various driving scenarios through the discussion and experiment.

Mining Interpretable AOG Representations From Convolutional Networks via Active Question Answering.

In this paper, we present a method to mine object-part patterns from conv-layers of a pre-trained convolutional neural network (CNN). The mined object-part patterns are organized by an And-Or graph (AOG). This interpretable AOG representation consists of a four-layer semantic hierarchy, i.e., semantic parts, part templates, latent patterns, and neural units. The AOG associates each object part with certain neural units in feature maps of conv-layers. The AOG is constructed with very few annotations (e.g., 3-20) of object parts. We develop a question-answering (QA) method that uses active human-computer communications to mine patterns from a pre-trained CNN, in order to explain features in conv-layers incrementally. During the learning process, our QA method uses the current AOG for part localization. The QA method actively identifies objects, whose feature maps cannot be explained by the AOG. Then, our method asks people to annotate parts on the unexplained objects, and uses answers to discover CNN patterns corresponding to newly labeled parts. In this way, our method gradually grows new branches and refines existing branches on the AOG to semanticize CNN representations. In experiments, our method exhibited a high learning efficiency. Our method used about 1/6- 1/3 of the part annotations for training, but achieved similar or better part-localization performance than fast-RCNN methods.

Machine Number Sense: A Dataset of Visual Arithmetic Problems for Abstract and Relational Reasoning

As a comprehensive indicator of mathematical thinking and intelligence, the number sense (Dehaene 2011) bridges the induction of symbolic concepts and the competence of problem-solving. To endow such a crucial cognitive ability to machine intelligence, we propose a dataset, Machine Number Sense (MNS), consisting of visual arithmetic problems automatically generated using a grammar model—And-Or Graph (AOG). These visual arithmetic problems are in the form of geometric figures: each problem has a set of geometric shapes as its context and embedded number symbols. Solving such problems is not trivial; the machine not only has to recognize the number, but also to interpret the number with its contexts, shapes, and relations (e.g., symmetry) together with proper operations. We benchmark the MNS dataset using four predominant neural network models as baselines in this visual reasoning task. Comprehensive experiments show that current neural-network-based models still struggle to understand number concepts and relational operations. We show that a simple brute-force search algorithm could work out some of the problems without context information. Crucially, taking geometric context into account by an additional perception module would provide a sharp performance gain with fewer search steps. Altogether, we call for attention in fusing the classic search-based algorithms with modern neural networks to discover the essential number concepts in future research.

Causal Reasoning in Multi-Object Interaction on the Traffic Scene: Occlusion-Aware Prediction of Visibility Fluent

Occlusion caused by multi-object interaction makes the traffic scene understanding intractable. In this paper, we focus on predicting the visibility status of vehicle in the framework of causality perception. The visibility fluent is employed to present the varying state of an object, involving visible and occluded. We introduce a probabilistic grammar model, named Hierarchical And-Or Graph (H-AOG), to construct the causal relations between fluents and actions. It consists of a Causal And-Or Graph (C-AOG) module and an Action And-Or Graph (A-AOG) module. An influence field is constructed by the polar coordinate transformation to model interactions in the A-AOG module. This method interprets the occurrence of occlusion due to multi-vehicle interaction. We evaluate our approach on both synthetic data and real data from the KITTI dataset. Compared to the state-of-the-art models like LSTM/GRU, it proves to achieve a promising accuracy in prediction of objects' visibility states and have better generalization on real data.

An efficient metaheuristics for a sequence-dependent disassembly planning

Abstract Disassembly planning (DP) is critical in remanufacturing and value recovery from end-of-life products and has attracted increasing attention due to the recent resurgence of research on circular economy. DP problem is NP-hard and its complexity increases exponentially with the size of problem. Sequence-dependent cost due to varying quality of the parts to be retrieved further increases the complexity of DP problems. This paper investigates the DP considering sequence-dependent costs among disassembly operations. A mathematical model is proposed with the objective to maximize the recovery profit using an AND/OR graph (AOG) subject to sequence-dependent costs. A novel two-phase heuristic method is developed to effectively generate feasible disassembly sequence according to the AOG in reasonable computation time. In addition, an improved genetic algorithm (IGA) is proposed to solve the problem, in combination with the presented two-phase heuristic. The performance of IGA is measured on a series of test problem instances against exact methods including CPLEX and an iterative method. Results indicate that IGA successfully find the near-optimal/optimal solutions and outperforms the other methods in terms of computation time. Finally, the proposed method is applied to compute the disassembly solution of a HG5-20 triaxial five speed mechanical transmission. Compared to the existing disassembly solutions of the transmission, the obtained solutions by IGA can shorten about 11% disassembly time and increase by approximately 7% recovery profit.

Neural Task Planning With AND–OR Graph Representations

This paper focuses on semantic task planning, that is, predicting a sequence of actions toward accomplishing a specific task under a certain scene, which is a new problem in computer vision research. The primary challenges are how to model the task-specific knowledge and how to integrate this knowledge into the learning procedure. In this paper, we propose training a recurrent long short-term memory (LSTM) network to address this problem, that is, taking a scene image (including prelocated objects) and the specified task as input and recurrently predicting action sequences. However, training such a network generally requires large numbers of annotated samples to cover the semantic space (e.g., diverse action decomposition and ordering). To overcome this issue, we introduce a knowledge and – or graph (AOG) for task description, which hierarchically represents a task as atomic actions. With this AOG representation, we can produce many valid samples (i.e., action sequences according to common sense) by training another auxiliary LSTM network with a small set of annotated samples. Furthermore, these generated samples (i.e., task-oriented action sequences) effectively facilitate training of the model for semantic task planning. In our experiments, we create a new dataset that contains diverse daily tasks and extensively evaluates the effectiveness of our approach.

A branch, bound, and remember algorithm for the simple disassembly line balancing problem

In this paper, we deal with a disassembly line balancing problem (DLBP), using an AND/OR graph (AOG) to represent the precedence relations between tasks. The decision maker needs to select a proper processing alternative and assign the corresponding tasks among stations to minimise the number of stations, without violating the cycle time constraint and the precedence relations. The problem was first formulated by Koc et al. (2009) and is denoted as type 1 simple DLBP (SDLBP-1) in this study. We prove that an SDLBP-1 with no parallel tasks is polynomially solvable and develop a branch, bound, and remember (BB&R) algorithm for the general SDLBP-1 with parallel tasks. Moreover, two lower bounding schemes, a strengthened Koc's integer programming (IP) model and a new benchmark instance generation scheme are proposed. Computational results show that the BB&R algorithm is the state-of-the-art exact algorithm for SDLBP-1, and that it can be easily truncated into a state-of-the-art heuristic which optimally solves most instances in very short time. In addition, the lower bounds and the strengthened IP model are also demonstrated to be effective.

Mining deep And-Or object structures via cost-sensitive question-answer-based active annotations

This paper presents a cost-sensitive active Question-Answering (QA) framework for learning a nine-layer And-Or graph (AOG) from web images. The AOG explicitly represents object categories, poses/viewpoints, parts, and detailed structures within the parts in a compositional hierarchy. The QA framework is designed to minimize an overall risk, which trades off the loss and query costs. The loss is defined for nodes in all layers of the AOG, including the generative loss (measuring the likelihood of the images) and the discriminative loss (measuring the fitness to human answers). The cost comprises both the human labor of answering questions and the computational cost of model learning. The cost-sensitive QA framework iteratively selects different storylines of questions to update different nodes in the AOG. Experiments showed that our method required much less human supervision (e.g. labeling parts on 3–10 training objects for each category) and achieved better performance than baseline methods.

A Novel method to represent Access Tree structure by Context Free Grammar with and-or graph in Key Policy based Attribute based Encryption

Important and sensitivity data of users in a third party managed internet or cloud always pose various security as well as privacy issues. Attribute-based encryption (ABE) is a pleasant trend in the literature which addresses above problem in an efficient way, and provides data security and fine-grained access control in a decentralized manner,. Key-policy attribute-based encryption (KP-ABE) is an important type of ABE, in which user can decrypt his message with a set of attributes and private keys are embedded with a access control structure which defines which cipher text an user can be allowed to decrypt. In this paper we use a probabilistic context free grammar with an And-Or structure to represent access control structure. And-Or graph has high expressive power hence access control structure can be represented in an efficient manner.

Transmission line detection based on a hierarchical and contextual model for aerial images

Due to highly cluttered backgrounds and limited visual features, transmission line detection (TLD) in complex and natural outdoor environments is a difficult task. Motivated by image parsing, a TLD algorithm that utilizes a decomposition structure and contextual information is proposed. Considering that cables are generally associated with the presence of pylons, correlations between pylons and cables, as well as correlations among different cable segments, are defined as the contextual information, which is described using a hierarchical model with the structure of an and-or graph. First, pylons in images are detected via an active basis model, which is an appearance model that uses sparse representation. Next, according to the number of pylons, the bottom-up inference in the hierarchical model selects different procedures and corresponding operations to detect cables. Multiple experiments are performed on aerial images captured in practical powerline inspection tasks. The experimental results illustrate that our TLD algorithm can effectively manage images captured under various landforms at different visual distances and from different viewpoints. For highly cluttered environments in particular, our algorithm has obvious advantages in terms of detection performance compared with existing approaches.

Configurable 3D Scene Synthesis and 2D Image Rendering with Per-pixel Ground Truth Using Stochastic Grammars

We propose a systematic learning-based approach to the generation of massive quantities of synthetic 3D scenes and arbitrary numbers of photorealistic 2D images thereof, with associated ground truth information, for the purposes of training, benchmarking, and diagnosing learning-based computer vision and robotics algorithms. In particular, we devise a learning-based pipeline of algorithms capable of automatically generating and rendering a potentially infinite variety of indoor scenes by using a stochastic grammar, represented as an attributed Spatial And-Or Graph, in conjunction with state-of-the-art physics-based rendering. Our pipeline is capable of synthesizing scene layouts with high diversity, and it is configurable inasmuch as it enables the precise customization and control of important attributes of the generated scenes. It renders photorealistic RGB images of the generated scenes while automatically synthesizing detailed, per-pixel ground truth data, including visible surface depth and normal, object identity, and material information (detailed to object parts), as well as environments (e.g., illuminations and camera viewpoints). We demonstrate the value of our synthesized dataset, by improving performance in certain machine-learning-based scene understanding tasks--depth and surface normal prediction, semantic segmentation, reconstruction, etc.--and by providing benchmarks for and diagnostics of trained models by modifying object attributes and scene properties in a controllable manner.

A Generative Model of Phonotactics

We present a probabilistic model of phonotactics, the set of well-formed phoneme sequences in a language. Unlike most computational models of phonotactics (Hayes and Wilson, 2008; Goldsmith and Riggle, 2012), we take a fully generative approach, modeling a process where forms are built up out of subparts by phonologically-informed structure building operations. We learn an inventory of subparts by applying stochastic memoization (Johnson et al., 2007; Goodman et al., 2008) to a generative process for phonemes structured as an and-or graph, based on concepts of feature hierarchy from generative phonology (Clements, 1985; Dresher, 2009). Subparts are combined in a way that allows tier-based feature interactions. We evaluate our models’ ability to capture phonotactic distributions in the lexicons of 14 languages drawn from the WOLEX corpus (Graff, 2012). Our full model robustly assigns higher probabilities to held-out forms than a sophisticated N-gram model for all languages. We also present novel analyses that probe model behavior in more detail.