Open-world Environment Research Articles

The Evolution and Impact of Multiplayer Cooperative Adventure Games in Open World Environments

The popularity of a multiplayer cooperative adventure game increases in proportion to its social interaction and interactivity level. The advancement of graphics processing technology and online connectivity has significantly contributed to the development of these games, enabling players to interact and explore expansive, intricately designed open worlds. This paper conducts a comprehensive analysis of existing literature and case studies to examine the development of these games, significant characteristics, and impact on individuals and communities. Games like "Destiny 2," "Monster Hunter: World," and "Sea of Thieves" have tried to engage players by incorporating shared goals, dynamic events, and social tools. Empirical data demonstrates that these games not only enhance team collaboration and communication skills but also foster the capability to establish and expand virtual communities. This study highlights the capacity of cooperative mechanics, engaging content, and social aspects to stimulate player interaction and foster the building of communities. Future research will focus on implementing innovative designs to enhance player interaction and promote community development in multiplayer cooperative adventure games.

MVDet: Encrypted malware traffic detection via multi-view analysis

Detecting encrypted malware traffic promptly to halt the further propagation of an attack is critical. Currently, machine learning becomes a key technique for extracting encrypted malware traffic patterns. However, due to the dynamic nature of network environments and the frequent updates of malware, current methods face the challenges of detecting unknown malware traffic in open-world environment. To address the issue, we introduce MVDet, a novel method that employs machine learning to mine the behavioral features of malware traffic based on multi-view analysis. Unlike traditional methods, MVDet innovatively characterizes the behavioral features of malware traffic at 4-tuple flows from four views: statistical view, DNS view, TLS view, and business view, which is a more stable feature representation capable of handling complex network environments and malware updates. Additionally, we achieve a short-time behavioral features construction, significantly reducing the time cost for feature extraction and malware detection. As a result, we can detect malware behavior at an early stage promptly. Our evaluation demonstrates that MVDet can detect a wide variety of known malware traffic and exhibits efficient and robust detection in both open-world and unknown malware scenarios. MVDet outperforms state-of-the-art methods in closed-world known malware detection, open-world known malware detection, and open-world unknown malware detection.

Open-world disaster information identification from multimodal social media

The application of multimodal deep learning for emergency response and recovery, specifically in disaster social media analysis, is of utmost importance. It is worth noting that in real-world scenarios, sudden disaster events may differ from the training data, which may require the multimodal network to predict them as unknown classes instead of misclassifying them to known ones. Previous studies have primarily focused on model accuracy in a closed environment and may not be able to directly detect unknown classes. Thus, we propose a novel multimodal model for categorizing social media related to disasters in an open-world environment. Our methodology entails utilizing pre-trained unimodal models as encoders for each modality and performing information fusion with a cross-attention module to obtain the joint representation. For open-world detection, we use a multitask classifier that encompasses both a closed-world and an open-world classifier. The closed-world classifier is trained on the original data to classify known classes, whereas the open-world classifier is used to determine whether the input belongs to a known class. Furthermore, we propose a sample generation strategy that models the distribution of unknown samples using known data, which allows the open-world classifier to identify unknown samples. Our experiments were conducted on two public datasets, namely CrisisMMD and MHII. According to the experimental results, the proposed method outperforms other baselines and approaches in crisis information classification.

Impact of Flashbangs on Eye-Behavior while Navigating and Performing Visual Search Tasks in Stressful Open-World Environments

Impact of Flashbangs on Eye-Behavior while Navigating and Performing Visual Search Tasks in Stressful Open-World Environments

Combating Insider Threat in the Open-World Environments: Identification, Monitoring, and Data Augmentation

Recent years have witnessed a dramatic increase in a class of security threats known as "insider threats". These threats occur when individuals with authorized access to an organization's network engage in harmful activities, potentially leading to the disclosure of vital information or adversely affecting the organization's systems (e.g., financial loss, system crashes, and national security challenges). Distinct from other types of terror attacks, combating insider threats exhibits several unique challenges, including (1) rarity, (2) non-separability, (3) label scarcity, (4) dynamics, and (5) heterogeneity, making themselves extremely difficult to identify and mitigate. We target the challenging problem of combating insider threats in open-world environments by leveraging a variety of data sources (e.g., internal system logs, employee networks, human trafficking, and smuggling networks). To effectively combat these intricate threats, we introduce an interactive learning mechanism that is composed of three mutually beneficial learning modules: insider identification, insider monitoring, and data augmentation. Each module plays a crucial role in enhancing our ability to detect and mitigate insider threats, thereby contributing to a more secure and resilient organizational environment.

A neurosymbolic cognitive architecture framework for handling novelties in open worlds

“Open world” environments are those in which novel objects, agents, events, and more can appear and contradict previous understandings of the environment. This runs counter to the “closed world” assumption used in most AI research, where the environment is assumed to be fully understood and unchanging. The types of environments AI agents can be deployed in are limited by the inability to handle the novelties that occur in open world environments. This paper presents a novel cognitive architecture framework to handle open-world novelties. This framework combines symbolic planning, counterfactual reasoning, reinforcement learning, and deep computer vision to detect and accommodate novelties. We introduce general algorithms for exploring open worlds using inference and machine learning methodologies to facilitate novelty accommodation. The ability to detect and accommodate novelties allows agents built on this framework to successfully complete tasks despite a variety of novel changes to the world. Both the framework components and the entire system are evaluated in Minecraft-like simulated environments. Our results indicate that agents are able to efficiently complete tasks while accommodating “concealed novelties” not shared with the architecture development team.

A Q-learning Novelty Search Strategy for Evaluating Robustness of Deep Reinforcement Learning in Open-world Environments

A Q-learning Novelty Search Strategy for Evaluating Robustness of Deep Reinforcement Learning in Open-world Environments

Recalling Unknowns without Losing Precision: An Effective Solution to Large Model-Guided Open World Object Detection.

Open World Object Detection (OWOD) aims to adapt object detection to an open-world environment, so as to detect unknown objects and learn knowledge incrementally. Existing OWOD methods typically leverage training sets with a relatively small number of known objects. Due to the absence of generic object knowledge, they fail to comprehensively perceive objects beyond the scope of training sets. Recent advancements in large vision models (LVMs), trained on extensive large-scale data, offer a promising opportunity to harness rich generic knowledge for the fundamental advancement of OWOD. Motivated by Segment Anything Model (SAM), a prominent LVM lauded for its exceptional ability to segment generic objects, we first demonstrate the possibility to employ SAM for OWOD and establish the very first SAM-Guided OWOD baseline solution. Subsequently, we identify and address two fundamental challenges in SAM-Guided OWOD and propose a pioneering SAM-Guided Robust Open-world Detector (SGROD) method, which can significantly improve the recall of unknown objects without losing the precision on known objects. Specifically, the two challenges in SAM-Guided OWOD include: (1) Noisy labels caused by the class-agnostic nature of SAM; (2) Precision degradation on known objects when more unknown objects are recalled. For the first problem, we propose a dynamic label assignment (DLA) method that adaptively selects confident labels from SAM during training, evidently reducing the noise impact. For the second problem, we introduce cross-layer learning (CLL) and SAM-based negative sampling (SNS), which enable SGROD to avoid precision loss by learning robust decision boundaries of objectness and classification. Experiments on public datasets show that SGROD not only improves the recall of unknown objects by a large margin (~ 20%), but also preserves highly-competitive precision on known objects. The program codes are available at https://github.com/harrylin-hyl/SGROD.

Improving deep learning with prior knowledge and cognitive models: A survey on enhancing explainability, adversarial robustness and zero-shot learning

We review current and emerging knowledge-informed and brain-inspired cognitive systems for realizing adversarial defenses, eXplainable Artificial Intelligence (XAI), and zero-shot or few-shot learning. Data-driven machine learning models have achieved remarkable performance and demonstrated capabilities surpassing humans in many applications. Yet, their inability to exploit domain knowledge leads to serious performance limitations in practical applications. In particular, deep learning systems are exposed to adversarial attacks, which can trick them into making glaringly incorrect decisions. Moreover, complex data-driven models typically lack interpretability or explainability, i.e., their decisions cannot be understood by human subjects. Furthermore, models are usually trained on standard datasets with a closed-world assumption. Hence, they struggle to generalize to unseen cases during inference in practical open-world environments, thus, raising the zero- or few-shot generalization problem. Although many conventional solutions exist, explicit domain knowledge, brain-inspired neural networks and cognitive architectures offer powerful new dimensions towards alleviating these problems. Prior knowledge is represented in appropriate forms like mathematical relations, logic rules, knowledge graphs, and large language models (LLMs). and incorporated in deep learning frameworks to improve performance. Brain-inspired cognition methods use computational models that mimic the human brain to enhance intelligent behavior in artificial agents and autonomous robots. Ultimately, these models achieve better explainability, higher adversarial robustness and data-efficient learning, and can, in turn, provide insights for cognitive science and neuroscience—that is, to deepen human understanding on how the brain works in general, and how it handles these problems.

Intra- & Extra-Source Exemplar-Based Style Synthesis for Improved Domain Generalization

The generalization with respect to domain shifts, as they frequently appear in applications such as autonomous driving, is one of the remaining big challenges for deep learning models. Therefore, we propose an exemplar-based style synthesis pipeline to improve domain generalization in semantic segmentation. Our method is based on a novel masked noise encoder for StyleGAN2 inversion. The model learns to faithfully reconstruct the image, preserving its semantic layout through noise prediction. Random masking of the estimated noise enables the style mixing capability of our model, i.e. it allows to alter the global appearance without affecting the semantic layout of an image. Using the proposed masked noise encoder to randomize style and content combinations in the training set, i.e., intra-source style augmentation (ISSA\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ extrm{ISSA}$$\\end{document}) effectively increases the diversity of training data and reduces spurious correlation. As a result, we achieve up to 12.4%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$12.4\\%$$\\end{document} mIoU improvements on driving-scene semantic segmentation under different types of data shifts, i.e., changing geographic locations, adverse weather conditions, and day to night. ISSA\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ extrm{ISSA}$$\\end{document} is model-agnostic and straightforwardly applicable with CNNs and Transformers. It is also complementary to other domain generalization techniques, e.g., it improves the recent state-of-the-art solution RobustNet by 3%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$3\\%$$\\end{document} mIoU in Cityscapes to Dark Zürich. In addition, we demonstrate the strong plug-n-play ability of the proposed style synthesis pipeline, which is readily usable for extra-source exemplars e.g., web-crawled images, without any retraining or fine-tuning. Moreover, we study a new use case to indicate neural network’s generalization capability by building a stylized proxy validation set. This application has significant practical sense for selecting models to be deployed in the open-world environment. Our code is available at https://github.com/boschresearch/ISSA.

Geographical Aspects of Open-World Video Games

In recent years, open-world environments in video games have become increasingly popular and immersive. Millions of players are able to explore virtual landscapes that resemble the real world, yet significant differences exist. This study investigates the geographical accuracy of 15 open-world video games. The virtual landscapes in these games were analyzed for horizontal distance compression, increased slopes, idealized climate, simplified vegetation and water features, underpopulation, and spatial segregation of ethnic minorities. The findings show significant differences between the games in terms of their geographical accuracy, with some exhibiting a more realistic representation of the natural and cultural environment compared to others. This study sheds light on the relationship between virtual landscapes in video games and our perception of the real world, offering new insights into this rapidly growing field.

Beyond Trade-Off: An Optimized Binocular Stereo Vision Based Depth Estimation Algorithm for Designing Harvesting Robot in Orchards

Depth estimation is one of the bottleneck parts for harvesting robots to determine whether the operation of grasping or picking succeeds or not directly. This paper proposed a novel disparity completion method combined with bilateral filtering and pyramid fusion to improve the issues of incorrect outputs due to the missed or wrong matching when achieving 3D position from 2D images in open-world environments. Briefly, our proposed method has two significant advantages in general. Firstly, occlusion between leaves, branches, and fruits is a universal phenomenon in unstructured orchard environments, which results in the most depth estimation algorithms facing great challenges to obtain accurate outputs in these occluded regions. To alleviate these issues, unlike other research efforts that already exist, we optimized the semi-global matching algorithm to obtain high accuracy sparse values as an initial disparity map; then, an improved bilateral filtering algorithm is proposed to eliminate holes and discontinuous regions caused by occlusion to obtain precise and density disparity outputs. Secondly, due to taking the practical high-efficiency requirements of the automated harvesting robot in its working status into consideration, we attempted to merge multiple low-resolution bilateral filtering results through the pyramid fusion model which goes beyond the trade-off mechanism to improve the performance of both accuracy and time cost. Finally, a prototype harvesting robot was designed to conduct experiments at three kinds of different distances (0.6~0.75 m, 1~1.2 m, and 1.6~1.9 m). Experiment results showed that our proposed method achieved density disparity maps and eliminated holes and discontinuous defects in the disparity map effectively. The average absolute error of our proposed method is 3.2 mm, and the average relative error is 1.79%. In addition, the time cost is greatly reduced more than 90%. Comprehensive experimental results demonstrate that our proposed algorithm provides a potential possibility for designing harvesting.

Feature Pyramid Network Based on Skip Connect for Vehicle Prediction

The trajectory prediction of the target vehicle can improve the perception ability of autonomous driving vehicles in the open world environment and is an essential technology for realizing high-level fully automatic driving. The traditional vehicle trajectory prediction model based on image input is sensitive to the image style of the dataset and has weak generalization ability. The prediction model based on intermediate features can overcome the problem of different dataset styles. However, there are relatively few explorations on building models based on intermediate semantic features. Inspired by the feature pyramid in image processing, this paper proposes an FPSP (Feature Pyramid network based on Skip connect for vehicle Prediction) network. By adding the pyramid feature branch, the original network is improved. The initial coarse information is hierarchically introduced into corresponding abstract high-dimensional features. Experiments on the KITTI dataset prove that FPSP achieves considerable improvement with only a slight increase in time consumption.

Target Recognition in SAR Images by Deep Learning with Training Data Augmentation.

Mass production of high-quality synthetic SAR training imagery is essential for boosting the performance of deep-learning (DL)-based SAR automatic target recognition (ATR) algorithms in an open-world environment. To address this problem, we exploit both the widely used Moving and Stationary Target Acquisition and Recognition (MSTAR) SAR dataset and the Synthetic and Measured Paired Labeled Experiment (SAMPLE) dataset, which consists of selected samples from the MSTAR dataset and their computer-generated synthetic counterparts. A series of data augmentation experiments are carried out. First, the sparsity of the scattering centers of the targets is exploited for new target pose synthesis. Additionally, training data with various clutter backgrounds are synthesized via clutter transfer, so that the neural networks are better prepared to cope with background changes in the test samples. To effectively augment the synthetic SAR imagery in the SAMPLE dataset, a novel contrast-based data augmentation technique is proposed. To improve the robustness of neural networks against out-of-distribution (OOD) samples, the SAR images of ground military vehicles collected by the self-developed MiniSAR system are used as the training data for the adversarial outlier exposure procedure. Simulation results show that the proposed data augmentation methods are effective in improving both the target classification accuracy and the OOD detection performance. The purpose of this work is to establish the foundation for large-scale, open-field implementation of DL-based SAR-ATR systems, which is not only of great value in the sense of theoretical research, but is also potentially meaningful in the aspect of military application.

SAR Target Recognition via Random Sampling Combination in Open-World Environments

Target recognition in SAR images was widely studied over the years. Most of these works were usually based on the assumption that the targets in the test set belong to a limited set of classes. In the practical scenarios, it is common to encounter various kinds of new targets. It is therefore more meaningful to study target recognition in open-world environments. In these scenes, it is needed to reject the unknown classes while maintain the classification performance on known classes. In the past years, few works were devoted to open set target recognition. Though the detection performance on unknown targets can be improved to a certain extent in the preceding works, most detection schemes are independent of a pretrained feature extractor, leading to potential open space risks. Besides, the model architectures are complicated, resulting in huge computational cost. To solve these problems, a family of new methods for open set target recognition is proposed. Targets indistinguishable from known classes are constructed by random sampling combination strategy. They are further sent into the classifier for feature learning. The original open-world environment is then transformed into a closed-world environment containing the unknown class. Moreover, the special implication of generated unknown targets is highlighted and used to realize unknown detection. Extensive experimental results on the MSTAR benchmark dataset illustrate the effectiveness of the proposed methods.

Open-Metric for Unknown Signal Inference

The closed-set assumption that only known classes can be classified during testing is usually adopted in traditional signal recognition missions. It has been thoroughly explored and made huge progress in the past decade. However, it cannot cope with the unknown categories coming from the changing environment. As waveform diversity technology evolves by leaps and bounds, the existence of novel unknown classes is inevitable in the open-world environment, imposing a huge challenge to the non-cooperative electronic support system. To further satisfy the demands of modern electronic intelligence systems and sense new threats, it is urgent to find an approach that can tackle unknown signals. An Open-Metric strategy is proposed in this article to deduce the unknown classes from the intercepted signals. By virtue of metric learning methods, a plain backbone can be trained to offer more discriminative representations and a consistent proxy for each class. According to extreme value analysis, information about known classes can be converted into an estimated score of unknown classes, creating an open decision layer. The investigated Open-Metric approach demonstrates its superiority in unknown signal inference tasks, facilitating improvements in the intelligence and smartness of the electronic reconnaissance agents to cope with future cognitive electronic warfare.

A review of sequential three-way decision and multi-granularity learning

The concept of three-way decision, interpreted and described as thinking, problem solving, and information processing in “threes”, has been widely studied and applied in machine learning and data engineering in recent years. In open-world environment, the connection and interaction of dynamic and uncertainty by multi-granularity learning gives more vitality to three-way decision. In this paper, we investigate and summarize the initial and development models of three-way decision. Then we revisit the historical line of sequential three-way decision from rough set to granular computing. Besides, we focus on exploring a unified framework of three-way multi-granularity learning with four crucial problems on mining uncertain region continually. Finally, we give some proposals on three-way decision associated with open-continual learning.

Hierarchical Fuzzy Rough Approximations With Three-Way Multigranularity Learning

The approximation learning of fuzzy concepts associated with fuzzy rough sets and three-way decisions is a useful technology for the representation, learning, and transformation of uncertain knowledge. However, how to integrate the advantages of fuzzy rough approximations and three-way approximations has rarely been carefully investigated so far. In this article, we focus on exploring the connection and interplay of these two methods, and further propose the hierarchical fuzzy rough approximations in the dynamic fuzzy open-world environment. We utilize the temporal-spatial perspectives of three-way decisions to construct multigranularity structures and implement multigranularity learning. The temporality of data and the spatiality of model parameters are both considered in such frameworks. Subsequently, we discuss the interpretation and representation of fuzzy three-way regions in fuzzy rough sets with some definitions and properties. The max, double, and aggregated evaluation-based models of three-way approximations are proposed to gradually transform a fuzzy concept to a crisp concept by the time-evolving attributes. Finally, the comparative experimental results between static and dynamic approaches demonstrate the effectiveness of our proposed hierarchical approximation learning models.

Avoiding Negative Side Effects of Autonomous Systems in the Open World

Autonomous systems that operate in the open world often use incomplete models of their environment. Model incompleteness is inevitable due to the practical limitations in precise model specification and data collection about open-world environments. Due to the limited fidelity of the model, agent actions may produce negative side effects (NSEs) when deployed. Negative side effects are undesirable, unmodeled effects of agent actions on the environment. NSEs are inherently challenging to identify at design time and may affect the reliability, usability and safety of the system. We present two complementary approaches to mitigate the NSE via: (1) learning from feedback, and (2) environment shaping. The solution approaches target settings with different assumptions and agent responsibilities. In learning from feedback, the agent learns a penalty function associated with a NSE. We investigate the efficiency of different feedback mechanisms, including human feedback and autonomous exploration. The problem is formulated as a multi-objective Markov decision process such that optimizing the agent’s assigned task is prioritized over mitigating NSE. A slack parameter denotes the maximum allowed deviation from the optimal expected reward for the agent’s task in order to mitigate NSE. In environment shaping, we examine how a human can assist an agent, beyond providing feedback, and utilize their broader scope of knowledge to mitigate the impacts of NSE. We formulate the problem as a human-agent collaboration with decoupled objectives. The agent optimizes its assigned task and may produce NSE during its operation. The human assists the agent by performing modest reconfigurations of the environment so as to mitigate the impacts of NSE, without affecting the agent’s ability to complete its assigned task. We present an algorithm for shaping and analyze its properties. Empirical evaluations demonstrate the trade-offs in the performance of different approaches in mitigating NSE in different settings.

A unified incremental updating framework of attribute reduction for two-dimensionally time-evolving data

In the open-world environment, the incremental updating approaches to attribute reduction based on rough sets are efficient and effective to evaluate and search an optimal subset of attributes from two-dimensionally time-evolving data, which can be interpreted as the complex changes of dynamic data, i.e., four types of combinations induced by the insertion/deletion of objects and the addition/remove of attributes. To avoid the time-consuming and repetitive computation from scratch in such dynamic data, this paper mainly focuses on constructing a unified incremental framework to attribute reduction by the matrix-based accelerated updating strategies. We systematically discuss and present a series of incremental updating mechanisms and algorithms of approximation quality in the neighborhood-based probabilistic rough sets. Besides, a unified framework of dynamic attribute reduction in four situations of changes is proposed to develop the performance of updating reduct. Finally, we report the comparative experiments between the non-incremental and incremental algorithms of reduct to demonstrate the feasibility and efficiency of proposed approaches.