Input Length Research Articles

High-Fidelity and Efficient Pluralistic Image Completion With Transformers.

Image completion has made tremendous progress with convolutional neural networks (CNNs), because of their powerful texture modeling capacity. However, due to some inherent properties (e.g., local inductive prior, spatial-invariant kernels), CNNs do not perform well in understanding global structures or naturally support pluralistic completion. Recently, transformers demonstrate their power in modeling the long-term relationship and generating diverse results, but their computation complexity is quadratic to input length, thus hampering the application in processing high-resolution images. This paper brings the best of both worlds to pluralistic image completion: appearance prior reconstruction with transformer and texture replenishment with CNN. The former transformer recovers pluralistic coherent structures together with some coarse textures, while the latter CNN enhances the local texture details of coarse priors guided by the high-resolution masked images. To decode diversified outputs from transformers, auto-regressive sampling is the most common method, but with extremely low efficiency. We further overcome this issue by proposing a new decoding strategy, temperature annealing probabilistic sampling (TAPS), which firstly achieves more than 70× speedup of inference at most, meanwhile maintaining the high quality and diversity of the sampled global structures. Moreover, we find the full CNN architecture will lead to suboptimal solutions for guided upsampling. To render more realistic and coherent contents, we design a novel module, named texture-aware guided attention, to concurrently consider the procedures of texture copy and generation, meanwhile raising several important modifications to solve the boundary artifacts. Through dense experiments, we found the proposed method vastly outperforms state-of-the-art methods in terms of four aspects: 1) large performance boost on image fidelity even compared to deterministic completion methods; 2) better diversity and higher fidelity for pluralistic completion; 3) exceptional generalization ability on large masks and generic dataset, like ImageNet. 4) Much higher decoding efficiency over previous auto-regressive based methods.

Speech Isolation and Recognition in Crowded Noise Using a Dual-Path Recurrent Neural Network

Speech separation is crucial for effective speech processing in multi-talker conditions, especially in real-time, low-latency applications. In this study, the Time-Domain Audio Separation Network (TasNet) and Dual-Path Recurrent Neural Network were used to perform a time-domain multiple-speaker speech separation challenge. One-dimensional conventional recurrent neural networks (RNNs) are not capable of accurately simulating long sequences. When their receptive length exceeds the sequence field, 1-D CNNs cannot recreate utterance-level sequences. Dual-Path Recurrent Neural Network (DPRNN) breaks up the lengthy sequential input that progressively performs intra- and inter-chunk operations with input lengths proportional to the square root of the beginning sequence length. Model outputs are more efficient than earlier systems, improving performance on the Libri Mix dataset. Investigations show that the DPRNN, sample-level modeling, and time-domain audio separation network can replace present methods. EEND-SS and other separation algorithms perform worse than DPRNN. The suggested model was able to achieve (12.376) SI-SDR, (0.969) STOI (short-time objective intelligibility), (12.363) SDR, (9.363) DER, and (97.193) SCA.

High radiation efficiency and tunable resonance planar double-turn spiral antenna for manipulation of nitrogen-vacancy centers.

Nitrogen-vacancy (NV) centers in diamond are promising quantum sensors, where microwave antennas play a crucial role in manipulating the spin states accurately. Conventional microwave antennas often struggle to balance radiation efficiency and bandwidth. To address this challenge, we design a planar double-turn spiral antenna (PDTSA), based on the ring microstrip antenna (RMA). PDTSA demonstrates an ∼4.5-fold increase in radiation efficiency compared to RMA. In addition, the PDTSA allows linear tunability of the resonance frequency up to 500MHz by adjusting the spiral input length. This feature addresses the limitations of a narrow working frequency range, which are typically caused by the narrowband in high-radiation-efficiency antennas. The experimental results show that at an absolute input power of 1W, the PDTSA increases the Rabi frequency from 1.72 to 8.06MHz compared to the RMA. This enhancement accelerates quantum state manipulation and reduces phase accumulation errors. These characteristics make PDTSA suitable for applications in quantum sensing and precision measurements using NV centers.

Exploring Early Number Abilities With Multimodal Transformers.

Early number skills represent critical milestones in children's cognitive development and are shaped over years of interacting with quantities and numerals in various contexts. Several connectionist computational models have attempted to emulate how certain number concepts may be learned, represented, and processed in the brain. However, these models mainly used highly simplified inputs and focused on limited tasks. We expand on previous work in two directions: First, we train a model end-to-end on video demonstrations in a synthetic environment with multimodal visual and language inputs. Second, we use a more holistic dataset of 35 tasks, covering enumeration, set comparisons, symbolic digits, and seriation. The order in which the model acquires tasks reflects input length and variability, and the resulting trajectories mostly fit with findings from educational psychology. The trained model also displays symbolic and non-symbolic size and distance effects. Using techniques from interpretability research, we investigate how our attention-based model integrates cross-modal representations and binds them into context-specific associative networks to solve different tasks. We compare models trained with and without symbolic inputs and find that the purely non-symbolic model employs more processing-intensive strategies to determine setsize.

A code change‐oriented approach to just‐in‐time defect prediction with multiple input semantic fusion

AbstractRecent research found that fine‐tuning pre‐trained models is superior to training models from scratch in just‐in‐time (JIT) defect prediction. However, existing approaches using pre‐trained models have their limitations. First, the input length is constrained by the pre‐trained models.Secondly, the inputs are change‐agnostic.To address these limitations, we propose JIT‐Block, a JIT defect prediction method that combines multiple input semantics using changed block as the fundamental unit. We restructure the JIT‐Defects4J dataset used in previous research. We then conducted a comprehensive comparison using eleven performance metrics, including both effort‐aware and effort‐agnostic measures, against six state‐of‐the‐art baseline models. The results demonstrate that on the JIT defect prediction task, our approach outperforms the baseline models in all six metrics, showing improvements ranging from 1.5% to 800% in effort‐agnostic metrics and 0.3% to 57% in effort‐aware metrics. For the JIT defect code line localization task, our approach outperforms the baseline models in three out of five metrics, showing improvements of 11% to 140%.

PULMO: Precise utterance-level modeling for speech anti-spoofing

In recent years, most state-of-the-art approaches for spoofed speech detection have been based on convolutional neural networks (CNNs). Most neural networks, including CNNs, are trained in minibatch units, where all input data in each minibatch must have the same shape. Therefore, for minibatch training, each utterance is first either padded or truncated because utterances are variable-length sequences and thus cannot be directly fed into networks in minibatch units. However, modeling either a padded or truncated utterance, rather than the original one, makes it unfeasible to capture the entire context as is: padding could propagate even unwanted information, like artifacts, in the original utterance, and truncation inevitably loses some information. With these information distortions, model could get stuck in a suboptimal solution. To fill this gap, we proposeÚ a method for precise utterance-level modeling that enables minibatch-wise utterance-level modeling of variable-length utterances while minimizing the information distortions. The proposed method comprises sequence segmentation followed by segment aggregation. Sequence segmentation feeds variable-length utterances in the minibatch unit by decomposing each of them into fixed-length segments, which enables parallel processing of variable-length utterances without the uncertainty in input length. Segment aggregation plays a role in aggregating the segment embeddings by utterance to encode the entire information of each utterance. The experimental results of the evaluation trials of ASVspoof 2019 and 2021 indicate that the proposed method shows up to 84.9 % and 97.6 % relative equal error rate reductions on logical and physical access scenarios, respectively. Furthermore, the proposed method reduced the FLOPs for an epoch by 6 %.

TurkishBERTweet: Fast and reliable large language model for social media analysis

Turkish is one of the most spoken languages in the world; however, it is still among the low-resource languages. Wide us of this language on social media platforms such as Twitter, Instagram, or Tiktok and strategic position of the country in the world politics makes it appealing for the social network researchers and industry. To address this need, we introduce TurkishBERTweet, the first large scale pre-trained language model for Turkish social media built using over 894 million Turkish tweets. The model shares the same architecture as RoBERTa-base model with smaller input length, making TurkishBERTweet lighter than the most used model, called BERTurk, and can have significantly lower inference time. We trained our model using the same approach for RoBERTa model and evaluated on two tasks: Sentiment Classification and Hate Speech Detection. We demonstrate that TurkishBERTweet outperforms the other available alternatives on generalizability and its lower inference time gives significant advantage to process large-scale datasets. We also show custom preprocessors for social media can acquire information from platform specific entities. We also conduct comparison with the commercial solutions like OpenAI and Gemini, and other available Turkish LLMs in terms of cost and performance to demonstrate TurkishBERTweet is scalable and cost-effective.

Random Projection‐Based Locality‐Sensitive Hashing in a Memristor Crossbar Array with Stochasticity for Sparse Self‐Attention‐Based Transformer

AbstractSelf‐attention mechanism is critically central to the state‐of‐the‐art transformer models. Because the standard full self‐attention has quadratic complexity with respect to the input's length L, resulting in prohibitively large memory for very long sequences, sparse self‐attention enabled by random projection (RP)‐based locality‐sensitive hashing (LSH) has recently been proposed to reduce the complexity to O(L log L). However, in current digital computing hardware with a von Neumann architecture, RP, which is essentially a matrix multiplication operation, incurs unavoidable time and energy‐consuming data shuttling between off‐chip memory and processing units. In addition, it is known that digital computers simply cannot generate provably random numbers. With the emerging analog memristive technology, it is shown that it is feasible to harness the intrinsic device‐to‐device variability in the memristor crossbar array for implementing the RP matrix and perform RP‐LSH computation in memory. On this basis, sequence prediction tasks are performed with a sparse self‐attention‐based Transformer in a hybrid software‐hardware approach, achieving a testing accuracy over 70% with much less computational complexity. By further harnessing the cycle‐to‐cycle variability for multi‐round hashing, 12% increase in the testing accuracy is demonstrated. This work extends the range of applications of memristor crossbar arrays to the state‐of‐the‐art large language models (LLMs).

A hybrid deep learning approach to predict hourly riverine nitrate concentrations using routine monitored data

With high-frequency data of nitrate (NO3–N) concentrations in waters becoming increasingly important for understanding of watershed system behaviors and ecosystem managements, the accurate and economic acquisition of high-frequency NO3–N concentration data has become a key point. This study attempted to use coupled deep learning neural networks and routine monitored data to predict hourly NO3–N concentrations in a river. The hourly NO3–N concentration at the outlet of the Oyster River watershed in New Hampshire, USA, was predicted through neural networks with a hybrid model architecture coupling the Convolutional Neural Networks and the Long Short-Term Memory model (CNN-LSTM). The routine monitored data (the river depth, water temperature, air temperature, precipitation, specific conductivity, pH and dissolved oxygen concentrations) for model training were collected from a nested high-frequency monitoring network, while the high-frequency NO3–N concentration data obtained at the outlet were not included as inputs. The whole dataset was separated into training, validation, and testing processes according to the ratio of 5:3:2, respectively. The hybrid CNN-LSTM model with different input lengths (1d, 3d, 7d, 15d, 30d) displayed comparable even better performance than other studies with lower frequencies, showing mean values of the Nash-Sutcliffe Efficiency 0.60–0.83. Models with shorter input lengths demonstrated both the higher modeling accuracy and stability. The water level, water temperature and pH values at monitoring sites were main controlling factors for forecasting performances. This study provided a new insight of using deep learning networks with a coupled architecture and routine monitored data for high-frequency riverine NO3–N concentration forecasting and suggestions about strategies about variable and input length selection during preprocessing of input data.

Short-term multi-step-ahead sector-based traffic flow prediction based on the attention-enhanced graph convolutional LSTM network (AGC-LSTM)

AbstractAccurate sector-based air traffic flow predictions are essential for ensuring the safety and efficiency of the air traffic management (ATM) system. However, due to the inherent spatial and temporal dependencies of air traffic flow, it is still a challenging problem. To solve this problem, some methods are proposed considering the relationship between sectors, while the complicated spatiotemporal dynamics and interdependencies between traffic flow of route segments related to the sector are not taken into account. To address this challenge, the attention-enhanced graph convolutional long short-term memory network (AGC-LSTM) model is applied to improve the short-term sector-based traffic flow prediction, in which spatial structures of route segments related to the sector are considered for the first time. Specifically, the graph convolutional networks (GCN)-LSTM network model was employed to capture spatiotemporal dependencies of the flight data, and the attention mechanism is designed to concentrate on the informative features from key nodes at each layer of the AGC-LSTM model. The proposed model is evaluated through a case study of the typical enroute sector in the central–southern region of China. The prediction results show that MAE reduces by 14.4% compared to the best performing GCN-LSTM model among the other five models. Furthermore, the study involves comparative analyses to assess the influence of route segment range, input and output sequence lengths, and time granularities on prediction performance. This study helps air traffic managers predict flight situations more accurately and avoid implementing overly conservative or excessively aggressive flow management measures for the sectors.

Comparison between parameter-efficient techniques and full fine-tuning: A case study on multilingual news article classification.

Adapters and Low-Rank Adaptation (LoRA) are parameter-efficient fine-tuning techniques designed to make the training of language models more efficient. Previous results demonstrated that these methods can even improve performance on some classification tasks. This paper complements existing research by investigating how these techniques influence classification performance and computation costs compared to full fine-tuning. We focus specifically on multilingual text classification tasks (genre, framing, and persuasion techniques detection; with different input lengths, number of predicted classes and classification difficulty), some of which have limited training data. In addition, we conduct in-depth analyses of their efficacy across different training scenarios (training on the original multilingual data; on the translations into English; and on a subset of English-only data) and different languages. Our findings provide valuable insights into the applicability of parameter-efficient fine-tuning techniques, particularly for multilabel classification and non-parallel multilingual tasks which are aimed at analysing input texts of varying length.

Automatic Identification of Emergency Sounds for Self-Driving Cars

As the development of Self-Driving Cars (SDCs) advances, one important feature that requires attention is their ability to sense and respond to emergencies on the road. Drivers of emergency vehicles prioritize speed over safety during emergencies to save lives, potentially leading to accidents. However, manual cars sometimes may not understand the urgency of the situation, further increasing the risk of collisions. In such cases, self-driving cars offer a better solution to replace manual cars and minimize road accidents. These cars with emergency sound detections offer a level of responsiveness, accuracy, and consistency that surpasses manual cars, contributing to improved road safety and efficiency in emergencies. Therefore, the key objective of the research is to improve the listening capability of self-driving cars to detect emergency vehicles with the help of a hybrid feature extraction technique. The suggested technique leverages a combination of Complex Morlet Wavelet and Co-occurrence Matrix to obtain statistical features from the emergency sounds. The proposed technique can work with the input length of 1.2 seconds of raw waveforms. This research work investigates that self-driving cars can accurately identify emergency vehicles by examining the distinctive emergency sound patterns emitted by the emergency vehicle with the highest accuracy of 94%. At the same time, the proposed technique reduces the computational cost by 20 – 40 milliseconds when compared with other techniques. The result of this work not only provides better accuracy but also reduces detection time, which is a crucial requirement for real-time applications such as self-driving cars.

Revisiting Bag of Words Document Representations for Efficient Ranking with Transformers

Modern transformer-based information retrieval models achieve state-of-the-art performance across various benchmarks. The self-attention of the transformer models is a powerful mechanism to contextualize terms over the whole input but quickly becomes prohibitively expensive for long input as required in document retrieval. Instead of focusing on the model itself to improve efficiency, this paper explores different bag of words document representations that encode full documents by only a fraction of their characteristic terms, allowing us to control and reduce the input length. We experiment with various models for document retrieval on MS MARCO data, as well as zero-shot document retrieval on Robust04, and show large gains in efficiency while retaining reasonable effectiveness. Inference time efficiency gains are both lowering the time and memory complexity in a controllable way, allowing for further trading off memory footprint and query latency. More generally, this line of research connects traditional IR models with neural “NLP” models and offers novel ways to explore the space between (efficient, but less effective) traditional rankers and (effective, but less efficient) neural rankers elegantly.

Global horizontal irradiance prediction model for multi-site fusion under different aerosol types

Accurate prediction of global horizontal irradiance (GHI) is crucial for anticipating the volatility of solar power output. Under clear-sky conditions, aerosols play a significant role in influencing GHI, with different types of aerosols exhibiting distinct radiation effects. In this study, we employed the Planck mean aerosol optical depth (AOD) —Informer model to forecast future GHI, utilizing historical data of AOD, meteorological parameters, and GHI as input variables. Two data-fusion schemes were proposed, and the calculation results indicate that Scheme 1, organized in the order of Xianghe—Beijing—Shangdianzi (XBS) based on the day as a unit, demonstrates superior prediction performance. Additionally, the site information was blurred, allowing for the flexible selection of input and output lengths. The calculation results remained stable and reliable, suggesting that the inclusion of fuzzy site information does not degrade the prediction effect. Building upon this, we categorized aerosols into three types based on their time span and data volume. GHI predictions for different aerosol types revealed that the mean square error (0.145) for marine aerosols were significantly lower than those for the other types. The GHI predictions for continental aerosols were closest to the true values.

Retrieve What You Need: A Mutual Learning Framework for Open-domain Question Answering

Abstract An open-domain question answering (QA) system usually follows a retrieve-then-read paradigm, in which a retriever is used to retrieve relevant passages from a large corpus, and then a reader generates answers based on the retrieved passages and the original question. In this paper, we propose a simple and novel mutual learning framework to improve the performance of retrieve-then-read-style models via an intermediate module named the knowledge selector, which we train with reinforcement learning. The key benefits of our proposed intermediate module are: 1) no requirement for additional annotated question-passage pairs; 2) improvements in both retrieval and QA performance, as well as computational efficiency, compared to prior competitive retrieve-then-read models; 3) with no finetuning, improvement in the zero-shot performance of large-scale pre-trained language models, e.g., ChatGPT, by encapsulating the input with relevant knowledge without violating the input length constraint.

Effect of tokenization on transformers for biological sequences.

Deep-learning models are transforming biological research, including many bioinformatics and comparative genomics algorithms, such as sequence alignments, phylogenetic tree inference, and automatic classification of protein functions. Among these deep-learning algorithms, models for processing natural languages, developed in the natural language processing (NLP) community, were recently applied to biological sequences. However, biological sequences are different from natural languages, such as English, and French, in which segmentation of the text to separate words is relatively straightforward. Moreover, biological sequences are characterized by extremely long sentences, which hamper their processing by current machine-learning models, notably the transformer architecture. In NLP, one of the first processing steps is to transform the raw text to a list of tokens. Deep-learning applications to biological sequence data mostly segment proteins and DNA to single characters. In this work, we study the effect of alternative tokenization algorithms on eight different tasks in biology, from predicting the function of proteins and their stability, through nucleotide sequence alignment, to classifying proteins to specific families. We demonstrate that applying alternative tokenization algorithms can increase accuracy and at the same time, substantially reduce the input length compared to the trivial tokenizer in which each character is a token. Furthermore, applying these tokenization algorithms allows interpreting trained models, taking into account dependencies among positions. Finally, we trained these tokenizers on a large dataset of protein sequences containing more than 400 billion amino acids, which resulted in over a 3-fold decrease in the number of tokens. We then tested these tokenizers trained on large-scale data on the above specific tasks and showed that for some tasks it is highly beneficial to train database-specific tokenizers. Our study suggests that tokenizers are likely to be a critical component in future deep-network analysis of biological sequence data. Code, data, and trained tokenizers are available on https://github.com/technion-cs-nlp/BiologicalTokenizers.

Abstract 3489: MetFlow: Automated nextflow pipeline for Ion AmpliSeq methylation assay designs

Abstract Introduction: DNA methylation plays a crucial role in development and progression of cancer. Identification of DNA methylation profiles in cancer is indispensable for early detection. To address this research need, Ion AmpliSeqTM offers targeted next-generation sequencing (NGS) methylation community amplicon panels for DNA methylation profiling in cancer-associated genes. However, the process of designing custom methylation amplicon panels using the Ion AmpliSeq primer design pipeline is intricate, labor-intensive, and time-consuming. Here, we developed a one-step, fully automated, faster, and reproducible nextflow-based primer design pipeline (MetFlow) for development of custom Ion AmpliSeqTM methylation panels. MetFlow streamlines the development of custom methylation panels for cancer-related genes, and thus significantly reduces the time and effort traditionally required for amplicon design. Methods: The fully automated MetFlow pipeline is implemented using Nextflow. MetFlow necessitate an input configuration file for the execution of methylation Ion AmpliSeqTM designs. The configuration file specifies the essential parameters, including target CpG input, reference genome, pool number, and amplicon lengths, which are required for designing the amplicons. The MetFlow pipeline designs candidate amplicons for CpG targets on both Watson and Crick strands, facilitating the subsequent creation of files required for the generation of a methylation panel. Results: The fully automated MetFlow pipeline has been used for designing the customized Ion AmpliSeqTM methylation amplicon panel(s) for cancer and age-related genes. The amplicons lengths in these AmpliSeqTM methylation panels were ranged from 125 bp to 175 bp for formalin-fixed, paraffin-embedded (FFPE) research samples. These customized methylation panels were run on the Ion GeneStudioTM S5 sequencer followed by performance analysis using Ion TorrentTM Suite. The published data for these custom amplicon panels suggest that the Ion AmpliSeqTM has sufficient read depth and mapping efficiency, higher sensitivity and accuracy for methylation measurement, and strong correlation (r=0.993) for methylation profiling among sample replicates. Conclusion: We present MetFlow, an advanced automated primer design pipeline for designing the custom Ion AmpliSeqTM methylation amplicon panels tailored to targeted NGS applications in cancer research. The custom designed Ion AmpliSeqTM methylation panels exhibited sufficient read depth, higher sensitivity and accuracy, and reproducibility. For Research Use Only. Not for use in diagnostic procedures. Citation Format: Renesh Bedre, Drishti Kaul, Anupma Sharma, Andrew Hatch, Loni Pickle, Na Li, Fiona Hyland. MetFlow: Automated nextflow pipeline for Ion AmpliSeq methylation assay designs [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 3489.

Shear Wave Velocity Prediction Based on the Long Short-Term Memory Network with Attention Mechanism

Shear wave velocity (VS) is a vital prerequisite for rock geophysics. However, due to historical, cost, and technical reasons, the shear wave velocity of some wells is missing. To reduce the deviation of the description of underground oil and gas distribution, it is urgent to develop a high-precision neural network prediction method. In this paper, an attention module is designed to automatically calculate the weight of each part of the input value. Then, the weighted data are fed into the long short-term memory network to predict shear wave velocities. Numerical simulations demonstrate the efficacy of the proposed method, which achieves a significantly lower MAE of 38.89 compared to the LSTM network’s 45.35 in Well B. In addition, the relationship between network input length and prediction accuracy is further analyzed.

Transfer Learning with ResNet3D-101 for Global Prediction of High Aerosol Concentrations

In order to better predict the high aerosol concentrations associated with air pollution and climate change, a machine learning model was developed using transfer learning and the segmentation process of global satellite images. The main concept of transfer learning lies on convolutional neural networks and works by initializing the already trained model weights to better adapt the weights when the network is trained on a different dataset. The transfer learning technique was tested with the ResNet3D-101 model pre-trained from a 2D ImageNet dataset. This model has performed well for contrail detection to assess climate impact. Aerosol distributions can be monitored via satellite remote sensing. Satellites can monitor some aerosol optical properties like aerosol optical thickness. Aerosol optical thickness snapshots were the input dataset for the model and were obtained from NASA’s Terra-Modis satellite; the output images were segmented by comparing the pixel values with a threshold value of 0.8 for aerosol optical thickness. Hyperparameter optimization finds a tuple of hyperparameters that yields an optimal model that minimizes a predefined loss function on given independent data. The model structure was adjusted in order to improve the performance of the model by applying methods and hyperparameter optimization techniques such as grid search, batch size, threshold, and input length. According to the criteria defined by the authors, the distance domain criterion and time domain criterion, the developed model is capable of generating adequate data and finding patterns in the time domain. As observed from the comparison of relative coefficients for the criteria metrics proposed by the authors, ddc and dtc, the deep learning model based on ConvLSTM layers developed in our previous studies has better performance than the model developed in this study with transfer learning.

Flexible and Energy-Efficient Crypto-Processor for Arbitrary Input Length Processing in Blockchain-Based IoT Applications

Flexible and Energy-Efficient Crypto-Processor for Arbitrary Input Length Processing in Blockchain-Based IoT Applications