Complicated Architectures Research Articles

Probing the effects of broken symmetries in machine learning

Abstract Symmetry is one of the most central concepts in physics, and it is no surprise that it has also been widely adopted as an inductive bias for machine-learning models applied to the physical sciences. 
This is especially true for models targeting the properties of matter at the atomic scale. Both established and state-of-the-art approaches, with almost no exceptions, are built to be exactly equivariant to translations, permutations, and rotations of the atoms.
Incorporating symmetries -- rotations in particular -- constrains the model design space and implies more complicated architectures that are often also computationally demanding. There are indications that unconstrained models can easily learn symmetries from data, and that doing so can even be beneficial for the accuracy of the model.
We demonstrate that an unconstrained architecture can be trained to achieve a high degree of rotational invariance, testing the impacts of the small symmetry breaking in realistic scenarios involving simulations of gas-phase, liquid, and solid water. 
We focus specifically on physical observables that are likely to be affected -- directly or indirectly -- by non-invariant behavior under rotations, finding negligible consequences when the model is used in an interpolative, bulk, regime. Even for extrapolative gas-phase predictions, the model remains very stable, even though symmetry artifacts are noticeable. 
We also discuss strategies that can be used to systematically reduce the magnitude of symmetry breaking when it occurs, and assess their impact on the convergence of observables.

Double pseudo-polymeric complexes of [Au(S2CNBu2)2][AgCl2] and [Au(S2CNHm)2]3[AgCl2][Ag0.59Au0.41Cl2]2: Preparation, supramolecular self-assembly, thermal behaviour and anti-mycobacterial activity

Pseudo-polymeric double complexes of [Au{S2CN(C4H9)2}2][AgCl2] (1) and [Au{S2CN(CH2)6}2]3[AgCl2][Ag0.59Au0.41Cl2]2 have been obtained and chemically identified using solution (1H, 13C{1H}) NMR and FT-IR spectroscopy. According to XRD analysis, it was established that both of the above crystalline compounds form complicated supramolecular architectures. Self-assembly and structural stabilization of these supramolecular formations are achieved due to numerous secondary interactions: Ag⋯S, Cl⋯S (in 1, 2) and Au⋯Cl, Au⋯S (in 2), which arose between the ionic structural units of the complexes. The combined manifestation of these interionic secondary interactions leads to the construction of two types of pseudo-polymeric chains of (⋯[Au(S2CNBu2)2]⋯[AgCl2]⋯)n (1) and (⋯[Au2(S2CNHm)4]⋯ [AgCl2]⋯)n (2). To study the thermal behaviour of the prepared compounds, the simultaneous thermal analysis (STA) technique was used, which allowed us to determine the conditions for the quantitative recovery of bound metals. Moreover, a high in vitro biological activity for each of the prepared gold(III)-silver(I) compounds against the non-pathogenic strain Mycolicibacterium smegmatis was experimentally revealed.

Evaluating text classification: A benchmark study

This paper presents an impartial and extensive benchmark for text classification involving five different text classification tasks, 20 datasets, 11 different model architectures, and 42,800 algorithm runs. The five text classification tasks are fake news classification, topic detection, emotion detection, polarity detection, and sarcasm detection. While in practice, especially in Natural Language Processing (NLP), research tends to focus on the most sophisticated models, we hypothesize that this is not always necessary. Therefore, our main objective is to investigate whether the largest state-of-the-art (SOTA) models are always preferred, or in what cases simple methods can compete with complex models, i.e. for which dataset specifications and classification tasks. We assess the performance of different methods with varying complexity, ranging from simple statistical and machine learning methods to pretrained transformers like robustly optimized BERT (Bidirectional Encoder Representations from Transformers) pretraining approach (RoBERTa). This comprehensive benchmark is lacking in existing literature, with research mainly comparing similar types of methods. Furthermore, with increasing awareness of the ecological impacts of extensive computational resource usage, this comparison is both critical and timely.We find that overall, bidirectional long short-term memory (LSTM) networks are ranked as the best-performing method albeit not statistically significantly better than logistic regression and RoBERTa. Overall, we cannot conclude that simple methods perform worse although this depends mainly on the classification task. Concretely, we find that for fake news classification and topic detection, simple techniques are the best-ranked models and consequently, it is not necessary to train complicated neural network architectures for these classification tasks. Moreover, we also find a negative correlation between F1 performance and complexity for the smallest datasets (with dataset size less than 10,000). Finally, the different models’ results are analyzed in depth to explain the model decisions, which is an increasing requirement in the field of text classification.

Configurable dual-topological-interface-states induced reflection in hybrid multilayers consisting of a Ge2Sb2Te5 film

Active control of induced reflection is crucial for many potential applications ranging from slowing light to biosensing devices. However, most previous approaches require patterned nanostructures to achieve controllable induced reflection, which hinders their further applications due to complicated architectures. Herein, we propose a lithography-free multilayered structure to achieve the induced reflection through the coupling of dual-topological-interface-states. The multilayers consist of two one-dimensional (1D) photonic crystals (PCs) and an Ag film separated by a Spacer, topological edge state (TES) and topological Tamm state (TTS) can be excited simultaneously and their coupling induces the reflection window. The coupled-oscillator model is proposed to mimic the coupling between the TES and TTS, and the analytical results are in good agreement with finite element method (FEM). In addition, the TES-TTS induced reflection is robust to the variation of structural parameters. By integrating an ultra-thin phase-change film of Ge2Sb2Te5 (GST) into the multilayers, the induced reflection can be switched through the phase transition of the GST film. The multipole decomposition reveals that the vanished reflection window is arising from the disappearance of TTS associated with the toroidal dipole (TD) mode.

MRG-T: Mask-Relation-Guided Transformer for Remote Vision-Based Pedestrian Attribute Recognition in Aerial Imagery

Nowadays, with the rapid development of consumer Unmanned Aerial Vehicles (UAVs), utilizing UAV platforms for visual surveillance has become very attractive, and a key part of this is remote vision-based pedestrian attribute recognition. Pedestrian Attribute Recognition (PAR) is dedicated to predicting multiple attribute labels of a single pedestrian image extracted from surveillance videos and aerial imagery, which presents significant challenges in the computer vision community due to factors such as poor imaging quality and substantial pose variations. Despite recent studies demonstrating impressive advancements in utilizing complicated architectures and exploring relations, most of them may fail to fully and systematically consider the inter-region, inter-attribute, and region-attribute mapping relations simultaneously and be stuck in the dilemma of information redundancy, leading to the degradation of recognition accuracy. To address the issues, we construct a novel Mask-Relation-Guided Transformer (MRG-T) framework that consists of three relation modeling modules to fully exploit spatial and semantic relations in the model learning process. Specifically, we first propose a Masked Region Relation Module (MRRM) to focus on precise spatial attention regions to extract more robust features with masked random patch training. To explore the semantic association of attributes, we further present a Masked Attribute Relation Module (MARM) to extract intrinsic and semantic inter-attribute relations with an attribute label masking strategy. Based on the cross-attention mechanism, we finally design a Region and Attribute Mapping Module (RAMM) to learn the cross-modal alignment between spatial regions and semantic attributes. We conduct comprehensive experiments on three public benchmarks such as PETA, PA-100K, and RAPv1, and conduct inference on a large-scale airborne person dataset named PRAI-1581. The extensive experimental results demonstrate the superior performance of our method compared to state-of-the-art approaches and validate the effectiveness of mask-relation-guided modeling in the remote vision-based PAR task.

Progress in Lewis-Acid-Templated Diels-Alder Reactions.

The synthesis of natural products with complicated architectures often requires the use of segments with functional groups that can be structurally transformed with the desired stereogenic centers. Bicyclic 𝛾-lactones have great potential as a suitable segment for natural product synthesis. However, the stereoselective construction of such functionalized bicyclic 𝛾-lactones is not as straightforward as one might expect. The template-mediated Diels-Alder reaction is one of the most powerful and versatile methods for providing bicyclic 𝛾-lactones with high regioselectivity and stereoselectivity. In this reaction, the diene is linked to the dienophile by a temporary tether, allowing the reaction to proceed efficiently, yielding a product that can be used for natural product synthesis. This review describes some important instances of the template-mediated Diels-Alder reaction and its application to the synthesis of biologically active compounds.

Diffusion tractography of kidney by high angular resolution diffusion imaging

Diffusion magnetic resonance imaging (MRI) has been utilized to probe the renal microstructures but investigating the three-dimensional (3D) tubular network still presents significant challenges due to the complicated architecture of kidney. This study aims to assess whether high angular resolution diffusion imaging (HARDI) could improve the reconstruction of 3D tubular architectures. Kidneys from both mice and rats were imaged using 3D diffusion-weighted pulse sequences at 9.4 T. Five healthy mouse kidneys were scanned at an isotropic spatial resolution of 40 μm, with a b value of 1500 s/mm2 across 46 diffusion encoding directions. The study employed diffusion tensor imaging (DTI) and generalized Q-sampling imaging (GQI) to examine the tubular orientation distributions and tractography, validated by conventional histology. Fractional anisotropy (FA) and mean diffusivity (MD) were quantified and compared among the inner medullar (IM), outer medullar (OM), and cortex (CO) at different angular resolutions. FA values, estimated with 6 diffusion-weighted images (DWIs), were significantly overestimated by 49.9% (p < 0.001) in IM, 179.4% (p < 0.001) in OM, and 225.5% (p < 0.001) in CO, compared to using 46 DWIs. In contrast, MD exhibited less variations to angular resolution variations (3.4% in IM, 4.2% in OM, and 4.6% in CO). Both DTI and GQI at high angular resolution successfully traced renal tubular structures throughout the kidney, with GQI demonstrating superior performance in generating more continuous tracts. Furthermore, disrupted renal tubule structures were observed in a chronic kidney disease (CKD) rat model. HARDI, especially when combined with the GQI approach, holds promise in tracking complicated 3D tubule architectures and may serve as a potent tool for kidney disease research.

The impact of memory on learning sequence-to-sequence tasks

The recent success of neural networks in natural language processing has drawn renewed attention to learning sequence-to-sequence (seq2seq) tasks. While there exists a rich literature that studies classification and regression tasks using solvable models of neural networks, seq2seq tasks have not yet been studied from this perspective. Here, we propose a simple model for a seq2seq task that has the advantage of providing explicit control over the degree of memory, or non-Markovianity, in the sequences—the stochastic switching-Ornstein–Uhlenbeck (SSOU) model. We introduce a measure of non-Markovianity to quantify the amount of memory in the sequences. For a minimal auto-regressive (AR) learning model trained on this task, we identify two learning regimes corresponding to distinct phases in the stationary state of the SSOU process. These phases emerge from the interplay between two different time scales that govern the sequence statistics. Moreover, we observe that while increasing the integration window of the AR model always improves performance, albeit with diminishing returns, increasing the non-Markovianity of the input sequences can improve or degrade its performance. Finally, we perform experiments with recurrent and convolutional neural networks that show that our observations carry over to more complicated neural network architectures.

Creation of polystyrene nanoparticle patterns for structural color application

The ability to regulate the spatial distribution of nanoparticles to generate patterns/arrays makes them promising applications in various fields. Here, a versatile and potential method, based on the polymer brushes patterned surface with hydrophilic-hydrophobic contrast characteristic, was developed to create the nanoparticle patterns with complicated architectures and controllable structural colors on the silicon/glass substrate. Employing DMD-mediated Photo-ATRP process, the selective growth of hydrophilic poly (ethylene glycol) methyl acrylate (PEGMA) brushes on the surface functionalized with hydrophobic initiators was achieved. Subsequently, the as-obtained hydrophilic-hydrophobic patterned substrate was acted as a template for regulating the assembly behavior and spatial arrangement of polystyrene (PS) nanoparticles, leading to the generation of nanoparticle patterns with desired architectures. Especially, the assembly densities could be controlled via tuning the grafting density of patterned PEGMA brushes, which enabled the effective regulation of structural color on the patterned surface. On the basis of the spatiotemporal control of patterned hydrophilic region formation, the sequential assembly of different nanoparticles on the same substrate could be achieved, allowing for a multi-component nanoparticle pattern for structural color-based anticounterfeiting application. The presented technique for the creation of nanoparticle patterns/arrays exhibits a tremendous practical potential in various fields including high-throughput preparation of structural colors, anticounterfeiting/ information encryption, and optical hypersurfaces.

Vessel trajectory prediction with recurrent neural networks: An evaluation of datasets, features, and architectures

Maritime situational awareness tasks such as port management, collision avoidance, and search-and-rescue missions rely on accurate knowledge of vessel locations. The availability of historical vessel trajectory data through the Automatic Identification System (AIS) has enabled the development of prediction methods, with a recent focus on trajectory prediction via recurrent neural networks (RNNs) and other deep learning architectures. While these methods have shown promising performance benefits over kinematic and clustering-based models, comparing among RNN-based models remains difficult due to variations in evaluation datasets, region sizes, vessel types, and numerous other design choices. As a result, it is not clear whether recent methods based on highly-sophisticated network architectures are necessary to achieve strong prediction performance. In this work, we present a simple fusion-based RNN approach to vessel trajectory prediction that allows for easy incorporation of exogenous variables. We perform an extensive ablation study to measure the impact of various modeling choices, including preprocessing, loss functions, and the choice of features, as well as the first usage of surface current information in vessel trajectory prediction. We demonstrate that our approach achieves state-of-the-art performance on three large regions off the United States coast, obtaining an improvement of up to 0.88 km over competing methods when predicting three hours into the future. We conclude that our simple architecture can outperform more complicated architectures while incurring a lower memory cost. Further, we show that the choice of loss function and the inclusion of surface current information both have significant impact on prediction performance.

High performance machine learning approach for reference evapotranspiration estimation

Accurate reference evapotranspiration (ET0) estimation has an effective role in reducing water losses and raising the efficiency of irrigation water management. The complicated nature of the evapotranspiration process is illustrated in the amount of meteorological variables required to estimate ET0. Incomplete meteorological data is the most significant challenge that confronts ET0 estimation. For this reason, different machine learning techniques have been employed to predict ET0, but the complicated structures and architectures of many of them make ET0 estimation very difficult. For these challenges, ensemble learning techniques are frequently employed for estimating ET0, particularly when there is a shortage of meteorological data. This paper introduces a powerful super learner ensemble technique for ET0 estimation, where four machine learning models: Extra Tree Regressor, Support Vector Regressor, K-Nearest Neighbor and AdaBoost Regression represent the base learners and their outcomes used as training data for the meta learner. Overcoming the overfitting problem that affects most other ensemble methods is a significant advantage of this cross-validation theory-based approach. Super learner performances were compared with the base learners for their forecasting capabilities through different statistical standards, where the results revealed that the super learner has better accuracy than the base learners, where different combinations of variables have been used whereas Coefficient of Determination (R2) ranged from 0.9279 to 0.9994 and Mean Squared Error (MSE) ranged from 0.0026 to 0.3289 mm/day but for the base learners R2 ranged from 0.5592 to 0.9977, and MSE ranged from 0.0896 to 2.0118 mm/day therefore, super learner is highly recommended for ET0 prediction with limited meteorological data.

On reading and interpreting black box deep neural networks

The deep neural networks used in computer vision and in recent large language models are widely recognized as black boxes, a term that describes their complicated architectures and opaque decision-making mechanisms. This essay outlines several different strategies through which humanist researchers and critics of machine learning might better understand and interpret the class of deep learning methods known as Transformers. These strategies expose different aspects of what might be “learned” as Transformers are trained and used in the analysis of language and can help critics at least partially open the black box of machine learning. They are also especially useful for digital humanists using these models as part of a research program informed by tool criticism in which the use of computational tools is conceived of as a metainterpretive act.

Double pseudo-polymeric complexes of [Au(S2CNR2)2][TlCl4], R2 = (CH2)5, (CH2)6: preparation, structural organization, and thermal behavior (low-temperature reduction and crystallization of gold)

The capability of thallium(I) alkylenedithiocarbamates to bind gold(III) from solutions in 2 M HCl was studied. Double complexes of [Au{S2CN(CH2)5}2][TlCl4] and [Au{S2CN(CH2)6}2][TlCl4] were preparatively isolated as individual forms of gold(III) binding into the solid phase. In the crystalline state, the ionic structural units of the complexes are combined together due to the secondary Au···S, Tl···S and S···Cl bonds that results in complicated supramolecular architectures. Thermal behavior of compounds was studied using the simultaneous thermal analysis technique; both reduced elemental gold and thallium(I) chloride were identified as the final thermolysis products.

Parameter Efficient Neural Networks With Singular Value Decomposed Kernels.

Traditionally, neural networks are viewed from the perspective of connected neuron layers represented as matrix multiplications. We propose to compose these weight matrices from a set of orthogonal basis matrices by approaching them as elements of the real matrices vector space under addition and multiplication. Making use of the Kronecker product for vectors, this composition is unified with the singular value decomposition (SVD) of the weight matrix. The orthogonal components of this SVD are trained with a descent curve on the Stiefel manifold using the Cayley transform. Next, update equations for the singular values and initialization routines are derived. Finally, acceleration for stochastic gradient descent optimization using this formulation is discussed. Our proposed method allows more parameter-efficient representations of weight matrices in neural networks. These decomposed weight matrices achieve maximal performance in both standard and more complicated neural architectures. Furthermore, the more parameter-efficient decomposed layers are shown to be less dependent on optimization and better conditioned. As a tradeoff, training time is increased up to a factor of 2. These observations are consequently attributed to the properties of the method and choice of optimization over the manifold of orthogonal matrices.

Cross-Modal Object Detection Via UAV

UAV-based object detection aims at locating and recognizing targets in aerial images, which is widely applied to traffic surveillance, disaster rescue and anomaly monitoring. However, due to expensive sensors and complicated architectures, it is unrealistic to deploy precise but heavy multi-modal object detectors into UAV nodes. To get over the dilemma, inspired by model compression and cross-modal signal processing techniques, this paper proposes a cross-modal knowledge distillation (CKD) enabled object detection paradigm, which achieves comparable detection performance with multi-modal techniques, yet requires less computational resource. On the one hand, in order to avoid transferring redundant feature knowledge, we design a Selective Feature Imitation (SFI) to selectively shorten the distance between cross-modal features according to their types. On the other hand, in order to transfer the most valuable prediction knowledge, we design an Adaptive Prediction Imitation (API). It evaluates the quality of prediction knowledge, and then adaptively adjusts the distillation intensity for cross-modal prediction. Extensive experiments on the DroneVehicle dataset have shown the performance improvement of the proposed scheme.

4D Additive-Subtractive Manufacturing of Shape Memory Ceramics.

The development of high-temperature structural materials, such as ceramics, is limited by their extremely high melting points and the difficulty in building complicated architectures. Four-dimensional (4D) printing helps enhance the geometrical flexibility of ceramics. However, ceramic 4D printing systems are limited by the separate processes for shape and material transformations, low accuracy of morphing systems, low resolution of ceramic structures, and their time-intensive nature. Here, a paradigm for a one-step shape/material transformation, high-2D/3D/4D-precision, high-efficiency, and scalable 4D additive-subtractive manufacturing of shape memory ceramics is developed. Original/reverse and global/local multimode shape memory capabilities are achieved using macroscale SiOC-based ceramic materials. The uniformly deposited Al2 O3 -rich layer on the printed SiOC-based ceramic lattice structures results in an unusually high flame ablation performance of the complex-shaped ceramics. The proposed framework is expected to broaden the applications of high-temperature structural materials in the aerospace, electronics, biomedical, and art fields.

Post-coordination photoinitiated macromolecular thiol-ene coupling reaction for construction of metallo-supramolecular complexes of ABCL2-type miktoarm star copolymers and self-assembly of the complexes in water

In this paper, we focus on investigating synthesis of metallo-supramolecular architectures of ABCL2-type miktoarm star copolymers PCL(-b-PDPA)-b-[(Tpyp)2-b-PMPC] (A, PCL; B, PDPA; C, PMPC; L, Tpyp) with ruthenium(II) (Ru(II)) ions and exploring self-assembly behavior of the complexes in water. Direct coordination of PCL(-b-PDPA)-b-[(Tpyp)2-b-PMPC] to Ru(II) ions and post-coordination photoinitiated macromolecular thiol-ene coupling reaction strategies were explored to construct Ru(II) coordination complexes, respectively. The result indicates that the post-coordination coupling reaction was more suitable strategy for constructing this kind of coordination complexes, which may avoid a larger steric hindrance generated from the coexistence of the three arms of PCL(-b-PDPA)-b-[(Tpyp)2-b-PMPC] leading to the disadvantageous influence on the formation of the bis(terpyridine)-Ru(II) complex. It should be mentioned that, however, although a photoinitiated macromolecular thiol-ene radical coupling reaction is facile, so far to achieve complete reaction of the target groups (i.e., alkene groups) has remained a challenge. Therefore, herein a post-coordination thiol-ene coupling reaction was explored carefully and an appropriate condition for the complete reaction of the alkene groups was given. Interestingly, the Ru(II) coordination complexes could self-assemble to form 2D platelets. This should be attributed to the unique metallo-supramolecular architecture. It is expected that the exploration can provide strategies for construction of complicated and exquisite metallo-supramolecular architectures and novel self-assemblies.

A damage-tolerant Ti-rich multiphase metallic-glass composite with hierarchically heterogeneous architecture

Lightweight materials with good combinations of properties, such as high strength, ductility and damage tolerance, are highly desirable in engineering applications; nevertheless, it remains a challenge to “defeat” the generally exclusive relationships between different properties for structural materials. One viable means is to construct complicated architectures in materials. Here, we employ a lightweight Ti-based metallic glass (MG) and pure Ti as constituents and assemble them into a hierarchically-structured multiphase composite with tailored heterogeneity. The Ti-rich multiphase composite exhibits a relatively low density of 4.95 g∙cm−3 owing to its high Ti content and a high strength of ∼1408 MPa derived primarily from the MG phase. The hierarchical architecture with tailored heterogeneity promotes the formation of abundant shear bands and microcracks in MG, yet inhibits their extension to avoid catastrophic fracture, thereby endowing the Ti-rich multiphase composite with a remarkable tensile ductility of ∼7.1% which is nearly two times that of the MG composite having similar phase constitution but a uniform structure. Moreover, this architecture causes crack-tip blunting by promoting plastic deformation, together with crack bifurcation and microcracking ahead of the crack tip, which bestows a high (notched) fracture toughness of ∼109 MPa∙m1/2 which is around three times that of its counterpart. This study is intended to give insight for the architectural design of new materials by offering a viable means for their architectural construction to achieve good combinations of properties.

Elevating performance of AlOxFy anti-reflection coating via controllable HF vapor fluorination

The introduction of anti-reflective coatings (ARCs) to optical devices has provided a simple and effective solution to improve their power efficiency. Among the various methods of preparing ARCs, the solution process is widely applied due to its simplicity, scalability, and naturally formable pores embedded in the film, which effectively reduce the refractive index of the material. Additionally, since low refractive index materials are always preferred for an effective ARC, strategies of fluorine substitution have been considered. Herein, we successfully introduce fluorine into Al2O3 ARCs by controlled HF vapor through abundant pores in the film. The exposure and reaction conditions were finely tunable, resulting in a precisely controlled amount of fluorine in the AlOxFy ARC. Finally, the gaseous HF vapor fluorination resulted in an effective porous ARC on a sapphire window by matching the refractive index between the surrounding media; thus, fluorinated ARC improves transparency. The maximum and the average transmittances of the ARC-applied window reach ∼99.15% and ∼97.04%, respectively. The suggested fluorination method is anticipated to have broad applicability for situations where surface chemistry modification is necessary. Highly reliable and controllable, this strategy can be used with complicated architectures without causing structural damage.

Local and global knowledge distillation with direction-enhanced contrastive learning for single-image deraining

Single image deraining (SID) is a challenging problem since the rainy images contain a variety of backgrounds with rain streaks of different directions and densities. Though previous convolutional neural network based methods for SID have shown promising performance, their characterization ability is limited as they suffer from the local receptive fields and ignore long-range dependencies. Some methods try to solve it by designing complicated network architectures, which yields high computational costs. To address these problems, we consider both local and global features in the proposed recurrent attention-distilling network (RADN). Specifically, in the shallow layer, the local contextual information is extracted by a local extract module with channel and pixel attention blocks to focus on features of important channels and spatial locations. In the deep layer, the global intrinsic features are characterized by a refined transformer block with a self-attention (SA) mechanism to capture the long-range dependencies. Meanwhile, a teacher network is pre-trained by a rain-to-rain mapping to supervise the proposed RADN learning the rain intrinsic features quicker with lightweight. Moreover, we propose a direction-enhanced contrastive learning (DeCL) strategy which incorporates the potential rainy images into negative contrastive pairs to improve the model generalization ability. Extensive experiments show that the proposed method can achieve a competitive performance to state-of-the-art methods on both real and synthetic datasets. The source code is available at https://github.com/Meky-hqd/RADN.