General Learning Framework Research Articles

CONTROL SYSTEMS SYNTHESIS FOR ROBOTS ON THE BASE OF MACHINE LEARNING BY SYMBOLIC REGRESSION

This paper presents a novel numerical method for solving the control system synthesis problem through the application of machine learning techniques, with a particular focus on symbolic regression. Symbolic regression is used to automate the development of control systems by constructing mathematical expressions that describe control functions based on system data. Unlike traditional methods, which often require manual programming and tuning, this approach leverages machine learning to discover optimal control solutions. The paper introduces a general framework for machine learning in control system design, with an emphasis on the use of evolutionary algorithms to optimize the generated control functions. The key contribution of this research lies in the development of an algorithm based on the principle of small variations in the baseline solution. This approach significantly enhances the efficiency of discovering optimal control functions by systematically exploring the solution space with minimal adjustments. The method allows for the automatic generation of control laws, reducing the need for manual coding, which is especially beneficial in the context of complex control systems, such as robotics. To demonstrate the applicability of the method, the research applies symbolic regression to the control synthesis of a mobile robot. The results of this case study show that symbolic regression can effectively automate the process of generating control functions, significantly reducing development time while improving accuracy. However, the paper also acknowledges certain limitations, including the computational demands required for symbolic regression and the challenges associated with real-time implementation in highly dynamic environments. These issues represent important areas for future research, where further optimization and hybrid approaches may enhance the method's practicality and scalability in real-world applications.

MMFA-DTA: Multimodal Feature Attention Fusion Network for Drug-Target Affinity Prediction for Drug Repurposing Against SARS-CoV-2.

The continuous emergence of novel infectious diseases poses a significant threat to global public health security, necessitating the development of small-molecule inhibitors that directly target pathogens. The RNA-dependent RNA polymerase (RdRp) and main protease (Mpro) of SARS-CoV-2 have been validated as potential key antiviral drug targets for the treatment of COVID-19. However, the conventional new drug R&D cycle takes 10-15 years, failing to meet the urgent needs during epidemics. Here, we propose a general multimodal deep learning framework for drug repurposing, MMFA-DTA, to enable rapid virtual screening of known drugs and significantly improve discovery efficiency. By extracting graph topological and sequence features from both small molecules and proteins, we design attention mechanisms to achieve dynamic fusion across modalities. Results demonstrate the superior performance of MMFA-DTA in drug-target affinity prediction over several state-of-the-art baseline methods on Davis and KIBA data sets, validating the benefits of heterogeneous information integration for representation learning and interaction modeling. Further fine-tuning on COVID-19-relevant bioactivity data enhances model predictions for critical SARS-CoV-2 enzymes. Case studies screening the FDA-approved drug library successfully identify etacrynic acid as the potential lead compound against both RdRp and Mpro. Molecular dynamics simulations further confirm the stability and binding affinity of etacrynic acid to these targets. This study proves the great potential and advantages of deep learning and drug repurposing strategies in supporting antiviral drug discovery. The proposed general and rapid response computational framework holds significance for preparedness against future public health events.

A general framework for generative self-supervised learning in non-invasive estimation of physiological parameters using photoplethysmography

Aligning physiological parameter labels with large-scale photoplethysmographic (PPG) data for deep learning is challenging and resource-intensive. While self-supervised representation learning (SSRL) can handle limited annotated data, the challenge lies in learning robust shared representations from vast unlabeled data and integrating various contextual cues to learn distinctive representations. To alleviate these challenges, a generative SSRL framework TS2TC is proposed to collaboratively utilize the temporal, spectrogram, and temporal-spectrogram mixed domains to explore and incorporate the unique features of PPG for universal and non-invasive physiological parameter estimation. Initially, a pretext task named Cross-Temporal Fusion Generative Anchor (CTFGA) is designed, modeling temporal dependencies and reconstructing independent segments at a coarse level to provide robust global feature extraction and local semantic contextual representation. The framework also includes sub-signals from PPG with diverse frequency scales and order derivatives reflecting hemodynamics to facilitate learning shared representations at varying semantic levels. Secondly, an advanced cognitive-inspired dual-process transfer (DPT) strategy is formulated, consisting of prior-dependent autonomous processes and posterior observation reasoning processes, to leverage the independent and integrated advantages of shared and specific representations. Furthermore, TS2TC introduces a novel bilinear temporal-spectrogram fusion method in the mixed domain, aligning latent representations from different domains, and establishing fine-grained contextual interactions at the feature level across multiple sources of information. Extensive experiments on physiological parameter estimation tasks showed that the joint performance of CTFGA and DPT outperforms standard generative learning significantly. TS2TC achieved an average 2.49% improvement in RMSE over the current state-of-the-art estimation methods with only 10% training data.

A general multi-source ensemble transfer learning framework for health prognostic of lithium-ion batteries

This paper proposes a novel multi-source domain adaptation method for the health prediction in the absence of target batteries labels. First, a computationally efficient and effective source domain selection method is proposed. The proposed source domain selection method selects source domains with excellent transferability based on Euclidean distance and Sample Entropy thresholds, thereby improving the performance of multi-source domain adaptation. Second, improved maximum mean discrepancy is proposed to guide the representation alignment. Compared with the original maximum mean discrepancy, it matches the source data with the target data in the order of aging, thus preserving the time order of the distribution. Third, we develop a multi-source domain adaptation network that contains a domain-invariant feature generator and several domain-specific feature generators and estimators. Two out of 12 datasets are used as target LIBs to evaluate the performance of the proposed method. The mean absolute error of the proposed method on these two datasets is only 0.671% and 0.868%, respectively. While the maximum error of MDA-IM-GC on both datasets does not exceed 5.2%. The experiment results indicate that the proposed network provides impressive accuracy with only 1/10 of unlabeled target historical data accessible and outperforms the single-source domain adaptation approach.

Causal Structure Learning for Recommender System

A fundamental challenge of recommender systems (RS) is understanding the causal dynamics underlying users’ decision making. Most existing literature addresses this problem by using causal structures inferred from domain knowledge. However, there are numerous phenomenons where domain knowledge is insufficient, and the causal mechanisms must be learned from the feedback data. Discovering the causal mechanism from RS feedback data is both novel and challenging, since RS itself is a source of intervention that can influence both the users’ exposure and their willingness to interact. Also for this reason, most existing solutions become inappropriate since they require data collected free from any RS. In this paper, we first formulate the underlying causal mechanism as a causal structural model and describe a general Causal Structure Learning framework for RS (CSL4RS) grounded in the real-world working mechanism. The essence of our approach is to acknowledge the unknown nature of RS intervention. We then derive the learning objective from our framework and utilize an augmented Lagrangian solver for efficient optimization. We conduct both simulation and real-world experiments to demonstrate how our approach compares favorably to existing solutions, together with the empirical analysis from sensitivity and ablation studies.

Hyperparameter optimization of two-branch neural networks in multi-target prediction

As a result of the ever increasing complexity of configuring and fine-tuning machine learning models, the field of automated machine learning (AutoML) has emerged over the past decade. However, software implementations like Auto-WEKA and Auto-sklearn typically focus on classical machine learning (ML) tasks such as classification and regression. Our work can be seen as the first attempt at offering a single AutoML framework for most problem settings that fall under the umbrella of multi-target prediction, which includes popular ML settings such as multi-label classification, multivariate regression, multi-task learning, dyadic prediction, matrix completion, and zero-shot learning. Automated problem selection and model configuration are achieved by extending DeepMTP, a general deep learning framework for MTP problem settings, with popular hyperparameter optimization (HPO) methods. Our extensive benchmarking across different datasets and MTP problem settings identifies cases where specific HPO methods outperform others.

DNSRF: Deep Network-based Semi-NMF Representation Framework

Representation learning is an important topic in machine learning, pattern recognition, and data mining research. Among many representation learning approaches, semi-nonnegative matrix factorization (SNMF) is a frequently-used one. However, a typical problem of SNMF is that usually there is no learning rate guidance during the optimization process, which often leads to a poor representation ability. To overcome this limitation, we propose a very general representation learning framework (DNSRF) that is based on deep neural net. Essentially, the parameters of the deep net used to construct the DNSRF algorithms are obtained by matrix element update. In combination with different activation functions, DNSRF can be implemented in various ways. In our experiments, we tested nine instances of our DNSRF framework on six benchmark datasets. In comparison with other state-of-the-art methods, the results demonstrate superior performance of our framework, which is thus shown to have a great representation ability.

Multi-Instance Multi-Task Learning for Joint Clinical Outcome and Genomic Profile Predictions From the Histopathological Images.

With the remarkable success of digital histopathology and the deep learning technology, many whole-slide pathological images (WSIs) based deep learning models are designed to help pathologists diagnose human cancers. Recently, rather than predicting categorical variables as in cancer diagnosis, several deep learning studies are also proposed to estimate the continuous variables such as the patients' survival or their transcriptional profile. However, most of the existing studies focus on conducting these predicting tasks separately, which overlooks the useful intrinsic correlation among them that can boost the prediction performance of each individual task. In addition, it is sill challenge to design the WSI-based deep learning models, since a WSI is with huge size but annotated with coarse label. In this study, we propose a general multi-instance multi-task learning framework (HistMIMT) for multi-purpose prediction from WSIs. Specifically, we firstly propose a novel multi-instance learning module (TMICS) considering both common and specific task information across different tasks to generate bag representation for each individual task. Then, a soft-mask based fusion module with channel attention (SFCA) is developed to leverage useful information from the related tasks to help improve the prediction performance on target task. We evaluate our method on three cancer cohorts derived from the Cancer Genome Atlas (TCGA). For each cohort, our multi-purpose prediction tasks range from cancer diagnosis, survival prediction and estimating the transcriptional profile of gene TP53. The experimental results demonstrated that HistMIMT can yield better outcome on all clinical prediction tasks than its competitors.

Anomaly Detection in Blockchain Networks Using Unsupervised Learning: A Survey

In decentralized systems, the quest for heightened security and integrity within blockchain networks becomes an issue. This survey investigates anomaly detection techniques in blockchain ecosystems through the lens of unsupervised learning, delving into the intricacies and going through the complex tapestry of abnormal behaviors by examining avant-garde algorithms to discern deviations from normal patterns. By seamlessly blending technological acumen with a discerning gaze, this survey offers a perspective on the symbiotic relationship between unsupervised learning and anomaly detection by reviewing this problem with a categorization of algorithms that are applied to a variety of problems in this field. We propose that the use of unsupervised algorithms in blockchain anomaly detection should be viewed not only as an implementation procedure but also as an integration procedure, where the merits of these algorithms can effectively be combined in ways determined by the problem at hand. In that sense, the main contribution of this paper is a thorough study of the interplay between various unsupervised learning algorithms and how this can be used in facing malicious activities and behaviors within public and private blockchain networks. The result is the definition of three categories, the characteristics of which are recognized in terms of the way the respective integration takes place. When implementing unsupervised learning, the structure of the data plays a pivotal role. Therefore, this paper also provides an in-depth presentation of the data structures commonly used in unsupervised learning-based blockchain anomaly detection. The above analysis is encircled by a presentation of the typical anomalies that have occurred so far along with a description of the general machine learning frameworks developed to deal with them. Finally, the paper spotlights challenges and directions that can serve as a comprehensive compendium for future research efforts.

Knowledge-Embedded Mutual Guidance for Visual Reasoning.

Visual reasoning between visual images and natural language is a long-standing challenge in computer vision. Most of the methods aim to look for answers to questions only on the basis of the analysis of the offered questions and images. Other approaches treat knowledge graphs as flattened tables to search for the answer. However, there are two major problems with these works: 1) the model disregards the fact that the world we surrounding us interlinks our hearing and speaking of natural language and 2) the model largely ignores the structure of the acrlong KG. To overcome these challenging deficiencies, a model should jointly consider two modalities of vision and language, as well as the rich structural and logical information embedded in knowledge graphs. To this end, we propose a general joint representation learning framework for visual reasoning, namely, knowledge-embedded mutual guidance. It realizes mutual guidance not only between visual data and natural language descriptions but also between knowledge graphs and reasoning models. In addition, it exploits the knowledge derived from the reasoning model to boost knowledge graphs when applying the visual relation detection task. The experimental results demonstrate that the proposed approach performs dramatically better than state-of-the-art methods on two benchmarks for visual reasoning.

Score Dynamics: Scaling Molecular Dynamics with Picoseconds Time Steps via Conditional Diffusion Model.

We propose score dynamics (SD), a general framework for learning accelerated evolution operators with large timesteps from molecular dynamics (MD) simulations. SD is centered around scores or derivatives of the transition log-probability with respect to the dynamical degrees of freedom. The latter play the same role as force fields in MD but are used in denoising diffusion probability models to generate discrete transitions of the dynamical variables in an SD time step, which can be orders of magnitude larger than a typical MD time step. In this work, we construct graph neural network-based SD models of realistic molecular systems that are evolved with 10 ps timesteps. We demonstrate the efficacy of SD with case studies of the alanine dipeptide and short alkanes in aqueous solution. Both equilibrium predictions derived from the stationary distributions of the conditional probability and kinetic predictions for the transition rates and transition paths are in good agreement with MD. Our current SD implementation is about 2 orders of magnitude faster than the MD counterpart for the systems studied in this work. Open challenges and possible future remedies to improve SD are also discussed.

Sample Efficient Reinforcement Learning Using Graph-Based Memory Reconstruction

Reinforcement learning (RL) algorithms typically require orders of magnitude more interactions than humans to learn effective policies. Research on memory in neuroscience suggests that humans' learning efficiency benefits from associating their experiences and reconstructing potential events. Inspired by this finding, we introduce a human brain-like memory structure for agents and build a general learning framework based on this structure to improve the RL sampling efficiency. Since this framework is similar to the memory reconstruction process in psychology, we name the newly proposed RL framework as Graph-Based Memory Reconstruction (GBMR). In particular, GBMR first maintains an attribute graph on the agent's memory and then retrieves its critical nodes to build and update potential paths among these nodes. This novel pipeline drives the RL agent to learn faster with its memory-enhanced value functions and reduces interactions with the environment by reconstructing its valuable paths. Extensive experimental analyses and evaluations in the Grid Maze and some challenging Atari environments demonstrate GBMR's superiority over traditional RL methods. We will release the source code and trained models to facilitate further studies in this research direction.

Physics-informed machine learning for reservoir management of enhanced geothermal systems

With the energy demand arising globally, geothermal recovery by Enhanced Geothermal Systems (EGS) becomes a promising option to bring sustainable energy supply along with mitigating CO2 emission. However, reservoir management of EGS primarily relies on reservoir simulation, which is quite expensive due to the reservoir heterogeneity, the interaction of matrix and fractures, and the intrinsic multi-physics coupled nature. Therefore, a robust optimization framework is critical for the management of EGS. We develop a general Physics-Informed Machine Learning (PIML) framework for reservoir management with multiple optimization options. A robust forward surrogate model fl is developed based on a convolutional neural network, and it successfully learns the nonlinear relationship between input reservoir model parameters (e.g., fracture permeability field) and interested state variables (e.g., temperature field and produced fluid temperature). fl is trained using simulation data from the EGS coupled thermal-hydro simulation model by sampling reservoir model parameters. As fl is accurate, efficient and fully differentiable, EGS thermal efficiency can be optimized following two schemes: (1) training a control network fc to map reservoir geological parameters to reservoir decision parameters by coupling it with fl; (2) directly optimizing the reservoir decision parameters based on coupling the existing optimizers with fl. We evaluate the impact of reservoir model parameters on the thermal recovery based on simulation datasets through sensitivity analyses, and demonstrated that injection mass rate dominates thermal recovery. Further, the forward model fl performs accurate and stable predictions of evolving temperature fields (relative error 1.27 ± 0.89%) in EGS and the time series of produced fluid temperature (relative error 0.26 ± 0.46%), and its speedup to the counterpart high-fidelity simulator is 4564 times. When optimizing with fc, we achieve thermal recovery with a reasonable accuracy but significantly low CPU time during inference, 0.11 seconds/task. When optimizing with Adam optimizer, we achieve the objective perfectly with relatively high CPU time, 4.58 seconds/task. This is because the former optimization scheme requires a training stage for fc but its inference is non-iterative, while the latter scheme requires an iterative inference without training. We also investigate the option to use fc inference as an initial guess for Adam optimizer, which decreases Adam’s CPU time but achieves excellent convergence in the objective function. This is the highest recommended option among the three evaluated. The efficiency, scalability and accuracy observed in our reservoir management framework makes it highly applicable to near real-time reservoir management in EGS as well as other similar system management processes.

An open‐source general purpose machine learning framework for individual animal re‐identification using few‐shot learning

Abstract Animal re‐identification remains a challenging problem due to the cost of tagging systems and the difficulty of permanently attaching a physical marker to some animals, such as sea stars. Due to these challenges, photo identification is a good fit to solve this problem whether evaluated by humans or through machine learning. Accurate machine learning methods are an improvement over manual identification as they are capable of evaluating a large number of images automatically and recent advances have reduced the need for large training datasets. This study aimed to create an accurate, robust, general purpose machine learning framework for individual animal re‐identification using images both from publicly available data as well as two groups of sea stars of different species under human care. Open‐source code was provided to accelerate work in this space. Images of two species of sea star (Asterias rubens and Anthenea australiae) were taken using a consumer‐grade smartphone camera and used as original datasets to train a machine learning model to re‐identify an individual animal using few examples. The model's performance was evaluated on these original sea star datasets which contained between 39–54 individuals and 983–1204 images, as well as using six publicly available re‐identification datasets for tigers, beef cattle noses, chimpanzee faces, zebras, giraffes and ringed seals ranging between 45–2056 individuals and 829–6770 images. Using time aware‐splits, which are a data splitting technique ensuring that the model only sees an individual's images from a previous collection event during training to avoid information leaking, the model achieved high (>99%) individual re‐identification mean average precision for the top prediction (mAP@1) for the two species of sea stars. The re‐identification mAP@1 for the mammalian datasets was more variable, ranging from 83% to >99%. However, this model outperformed published state‐of‐the‐art re‐identification results for the publicly available datasets. The reported approach for animal re‐identification is generalizable, with the same machine learning framework achieving good performance in two distinct species of sea stars with different physical attributes, as well as seven different mammalian species. This demonstrates that this methodology can be applied to nearly any species where individual re‐identification is required. This study presents a precise, practical, non‐invasive approach to animal re‐identification using only basic image collection methods.

GenURL: A General Framework for Unsupervised Representation Learning.

Unsupervised representation learning (URL) that learns compact embeddings of high-dimensional data without supervision has achieved remarkable progress recently. However, the development of URLs for different requirements is independent, which limits the generalization of the algorithms, especially prohibitive as the number of tasks grows. For example, dimension reduction (DR) methods, t-SNE and UMAP, optimize pairwise data relationships by preserving the global geometric structure, while self-supervised learning, SimCLR and BYOL, focuses on mining the local statistics of instances under specific augmentations. To address this dilemma, we summarize and propose a unified similarity-based URL framework, GenURL, which can adapt to various URL tasks smoothly. In this article, we regard URL tasks as different implicit constraints on the data geometric structure that help to seek optimal low-dimensional representations that boil down to data structural modeling (DSM) and low-dimensional transformation (LDT). Specifically, DSM provides a structure-based submodule to describe the global structures, and LDT learns compact low-dimensional embeddings with given pretext tasks. Moreover, an objective function, general Kullback-Leibler (GKL) divergence, is proposed to connect DSM and LDT naturally. Comprehensive experiments demonstrate that GenURL achieves consistent state-of-the-art performance in self-supervised visual learning, unsupervised knowledge distillation (KD), graph embeddings (GEs), and DR.

A General Deep Learning Framework Guided by Sparse Matching for Disparity Estimation in High-Resolution Satellite Stereo Imagery

A General Deep Learning Framework Guided by Sparse Matching for Disparity Estimation in High-Resolution Satellite Stereo Imagery

Continual Learning, Fast and Slow.

According to the Complementary Learning Systems (CLS) theory (McClelland et al. 1995) in neuroscience, humans do effective continual learning through two complementary systems: a fast learning system centered on the hippocampus for rapid learning of the specifics, individual experiences; and a slow learning system located in the neocortex for the gradual acquisition of structured knowledge about the environment. Motivated by this theory, we propose DualNets (for Dual Networks), a general continual learning framework comprising a fast learning system for supervised learning of pattern-separated representation from specific tasks and a slow learning system for representation learning of task-agnostic general representation via Self-Supervised Learning (SSL). DualNets can seamlessly incorporate both representation types into a holistic framework to facilitate better continual learning in deep neural networks. Via extensive experiments, we demonstrate the promising results of DualNets on a wide range of continual learning protocols, ranging from the standard offline, task-aware setting to the challenging online, task-free scenario. Notably, on the CTrL (Veniat et al. 2020) benchmark that has unrelated tasks with vastly different visual images, DualNets can achieve competitive performance with existing state-of-the-art dynamic architecture strategies (Ostapenko et al. 2021). Furthermore, we conduct comprehensive ablation studies to validate DualNets efficacy, robustness, and scalability.

DATA POISONING ATTACKS ON FEDERATED MACHINE LEARNING

Federated machine learning which enables resource constrained node devices (e.g., mobile phones and IoT devices) to learn a shared model while keeping the training data local, can provide privacy, security and economic benefits by designing an effective communication protocol. However, the communication protocol amongst different nodes could be exploited by attackers to launch data poisoning attacks, which has been demonstrated as a big threat to most machine learning models. In this paper, we attempt to explore the vulnerability of federated machine learning. More specifically, we focus on attacking a federated multi-task learning framework, which is a federated learning framework via adopting a general multi-task learning framework to handle statistical challenges. We formulate the problem of computing optimal poisoning attacks on federated multitask learning as a bilevel program that is adaptive to arbitrary choice of target nodes and source attacking nodes. Then we propose a novel systems-aware optimization method, ATTack on Federated Learning (AT2FL), which is efficiency to derive the implicit gradients for poisoned data, and further compute optimal attack strategies in the federated machine learning. Our work is an earlier study that considers issues of data poisoning attack for federated learning. To the end, experimental results on real-world datasets show that federated multi-task learning model is very sensitive to poisoning attacks, when the attackers either directly poison the target nodes or indirectly poison the related nodes by exploiting the communication protocol.

A General Spatial-Frequency Learning Framework for Multimodal Image Fusion.

multimodal image fusion involves tasks like pan-sharpening and depth super-resolution. Both tasks aim to generate high-resolution target images by fusing the complementary information from the texture-rich guidance and low-resolution target counterparts. They are inborn with reconstructing high-frequency information. Despite their inherent frequency domain connection, most existing methods only operate solely in the spatial domain and rarely explore the solutions in the frequency domain. This study addresses this limitation by proposing solutions in both the spatial and frequency domains. We devise a Spatial-Frequency Information Integration Network, abbreviated as SFINet for this purpose. The SFINet includes a core module tailored for image fusion. This module consists of three key components: a spatial-domain information branch, a frequency-domain information branch, and a dual-domain interaction. The spatial-domain information branch employs the spatial convolution-equipped invertible neural operators to integrate local information from different modalities in the spatial domain. Meanwhile, the frequency-domain information branch adopts a modality-aware deep Fourier transformation to capture the image-wide receptive field for exploring global contextual information. In addition, the dual-domain interaction facilitates information flow and the learning of complementary representations. We further present an improved version of SFINet, SFINet++, that enhances the representation of spatial information by replacing the basic convolution unit in the original spatial domain branch with the information-lossless invertible neural operator. We conduct extensive experiments to validate the effectiveness of the proposed networks and demonstrate their outstanding performance against state-of-the-art methods in two representative multimodal image fusion tasks: pan-sharpening and depth super-resolution. The source code is publicly available at https://github.com/manman1995/Awaresome-pansharpening.

Learning Virtual View Selection for 3D Scene Semantic Segmentation.

2D-3D joint learning is essential and effective for fundamental 3D vision tasks, such as 3D semantic segmentation, due to the complementary information these two visual modalities contain. Most current 3D scene semantic segmentation methods process 2D images "as they are", i.e., only real captured 2D images are used. However, such captured 2D images may be redundant, with abundant occlusion and/or limited field of view (FoV), leading to poor performance for the current methods involving 2D inputs. In this paper, we propose a general learning framework for joint 2D-3D scene understanding by selecting informative virtual 2D views of the underlying 3D scene. We then feed both the 3D geometry and the generated virtual 2D views into any joint 2D-3D-input or pure 3D-input based deep neural models for improving 3D scene understanding. Specifically, we generate virtual 2D views based on an information score map learned from the current 3D scene semantic segmentation results. To achieve this, we formalize the learning of the information score map as a deep reinforcement learning process, which rewards good predictions using a deep neural network. To obtain a compact set of virtual 2D views that jointly cover informative surfaces of the 3D scene as much as possible, we further propose an efficient greedy virtual view coverage strategy in the normal-sensitive 6D space, including 3-dimensional point coordinates and 3-dimensional normal. We have validated our proposed framework for various joint 2D-3D-input or pure 3D-input based deep neural models on two real-world 3D scene datasets, i.e., ScanNet v2 and S3DIS, and the results demonstrate that our method obtains a consistent gain over baseline models and achieves new top accuracy for joint 2D and 3D scene semantic segmentation. Code is available at https://github.com/smy-THU/VirtualViewSelection.