Raw Input Research Articles

Exploring Domain Incremental Video Highlights Detection with the LiveFood Benchmark

Video highlights detection (VHD) is an active research field in computer vision, aiming to locate the most user-appealing clips given raw video inputs. However, most VHD methods are based on the closed world assumption, i.e., a fixed number of highlight categories is defined in advance and all training data are available beforehand. Consequently, existing methods have poor scalability with respect to increasing highlight domains and training data. To address above issues, we propose a novel video highlights detection method named Global Prototype Encoding (GPE) to learn incrementally for adapting to new domains via parameterized prototypes. To facilitate this new research direction, we collect a finely annotated dataset termed LiveFood, including over 5,100 live gourmet videos that consist of four domains: ingredients, cooking, presentation, and eating. To the best of our knowledge, this is the first work to explore video highlights detection in the incremental learning setting, opening up new land to apply VHD for practical scenarios where both the concerned highlight domains and training data increase over time. We demonstrate the effectiveness of GPE through extensive experiments. Notably, GPE surpasses popular domain incremental learning methods on LiveFood, achieving significant mAP improvements on all domains. Concerning the classic datasets, GPE also yields comparable performance as previous arts. The code is available at: https://github.com/ForeverPs/IncrementalVHD_GPE.

RL-SeqISP: Reinforcement Learning-Based Sequential Optimization for Image Signal Processing

Hardware image signal processing (ISP), aiming at converting RAW inputs to RGB images, consists of a series of processing blocks, each with multiple parameters. Traditionally, ISP parameters are manually tuned in isolation by imaging experts according to application-specific quality and performance metrics, which is time-consuming and biased towards human perception due to complex interaction with the output image. Since the relationship between any single parameter’s variation and the output performance metric is a complex, non-linear function, optimizing such a large number of ISP parameters is challenging. To address this challenge, we propose a novel Sequential ISP parameter optimization model, called the RL-SeqISP model, which utilizes deep reinforcement learning to jointly optimize all ISP parameters for a variety of imaging applications. Concretely, inspired by the sequential tuning process of human experts, the proposed model can progressively enhance image quality by seamlessly integrating information from both the image feature space and the parameter space. Furthermore, a dynamic parameter optimization module is introduced to avoid ISP parameters getting stuck into local optima, which is able to more effectively guarantee the optimal parameters resulting from the sequential learning strategy. These merits of the RL-SeqISP model as well as its high efficiency are substantiated by comprehensive experiments on a wide range of downstream tasks, including two visual analysis tasks (instance segmentation and object detection), and image quality assessment (IQA), as compared with representative methods both quantitatively and qualitatively. In particular, even using only 10% of the training data, our model outperforms other SOTA methods by an average of 7% mAP on two visual analysis tasks.

Unified Convolutional Sparse Transformer for Disease Diagnosis, Monitoring, Drug Development, and Therapeutic Effect Prediction from EEG Raw Data.

Electroencephalogram (EEG) analysis plays an indispensable role across contemporary medical applications, which encompasses diagnosis, monitoring, drug discovery, and therapeutic assessment. This work puts forth an end-to-end deep learning framework that is uniquely tailored for versatile EEG analysis tasks by directly operating on raw waveform inputs. It aims to address the challenges of manual feature engineering and the neglect of spatial interrelationships in existing methodologies. Specifically, a spatial channel attention module is introduced to emphasize the critical inter-channel dependencies in EEG signals through channel statistics aggregation and multi-layer perceptron operations. Furthermore, a sparse transformer encoder is used to leverage selective sparse attention in order to efficiently process long EEG sequences while reducing computational complexity. Distilling convolutional layers further concatenates the temporal features and retains only the salient patterns. As it was rigorously evaluated on key EEG datasets, our model consistently accomplished a superior performance over the current approaches in detection and classification assignments. By accounting for both spatial and temporal relationships in an end-to-end paradigm, this work facilitates a versatile, automated EEG understanding across diseases, subjects, and objectives through a singular yet customizable architecture. Extensive empirical validation and further architectural refinement may promote broader clinical adoption prospects.

MT error detection and correction by Chinese language learners

Abstract This article analyzes texts that have been generated by machine translation (MT) and post-edited by native English-speaking trainee translators (English>Chinese) who are also Chinese language learners enrolled in a four-year undergraduate translation program. The project examines the work product of trainee translators to categorize 122 errors that are (un)noticed and (un)corrected by them. MT errors in the Accuracy category were best identified and corrected, followed by those in the Lexicon and Fluency categories. Trainee translators who were advanced language learners outperformed the intermediate group in MT error detection and correction, especially in the Lexicon category. This study sheds light upon the use of raw MT output as meaningful input for trainee translators who are in the process of learning Chinese. Its findings provide information regarding the type of exercises needed in language learning and translation training for students with different levels of language proficiency.

Abstract 863: Scratch: A highly modular pipeline for single-cell cancer research

Abstract Background: Single-cell RNA sequencing (scRNA-seq) is a robust approach to facilitate cancer research, revealing insights into tumor heterogeneity, microenvironment, and treatment response. However, scRNA-seq results frequently encounter reproducibility challenges due to i) high data complexity, intensified by human intervention, and ii) insufficient methods standardization, leading to inconsistent findings. Methods: Here, we introduce SCRATCH (Single-Cell RnA-seq Toolkit and Pipeline for Cancer researcH), a Nextflow-based pipeline designed to improve reproducibility through a layered and modular architecture. SCRATCH follows FAIR principles and guidelines provided by the nf-core community. Result/Discussion: The pipeline provides three execution modes: end-to-end, iterative, and custom. In the end-to-end mode, the pipeline processes data from raw input to downstream analyses automatically. This mode employs ranking- and aggregation-based approaches. For instance, the ranking approach leverages benchmark metrics to select the most suitable method in distinct steps, e.g., batch correction. Therefore, ensuring a consistent and data-driven selection. On another hand, the aggregation approach uses multiple predictions to increase confidence levels, such as on CNV inference and malignant cell identification. These strategies minimize human intervention, ideal for beginner users, enabling rapid access to preliminary results and biological insights. Alternatively, the iterative mode allows intermediate users to define workflow breakpoints in a layered-based fashion. Users can pause, review results, and adjust decisions at stages (e.g., TME annotation), facilitating a "semi-supervised" approach for a more tailored analysis while retaining the SCRATCH framework. Thirdly, the custom mode enables precise executions based on modules for similar tasks (e.g., trajectory analysis and cell-cell communication), allowing experienced users to bypass the SCRATCH workflow and use it as an on-demand toolkit. This mode leverages pipeline parallelism for efficient processing, perfect for ongoing single-cell projects. SCRATCH produces HTML reports to ensure traceability and reproducibility. Conclusion: SCRATCH, an evolving project, comprises 05 subworkflows, 18 modules, and 25 tools. We envisage SCRATCH as an open-source tool and invite developers to leverage its modules for their pipelines. For more information, please visit https://break-through-cancer.github.io/btc-scrna-training. Citation Format: Andre F. Fonseca, Guangchun Han, Marcel Ribeiro-Dantas, Enyu Dai, Diljot Grewal, Matthew Zatzman, Eliyahu Havasov, Andrew McPherson, Break Through Cancer, Data Science TeamLab, Michael N. Noble, Rameen Beroukhim, Rachel Karchin, Sohrab P. Shah, Linghua Wang. Scratch: A highly modular pipeline for single-cell cancer research [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 863.

Assessing the Carbon Footprint of Plastic Bottle Blow Mold Based on Product Life Cycle for Managing the Mold Industry’s Carbon Emission

Calculating the carbon footprint (CF) holds paramount importance in today’s world as it provides a tangible measure of our impact on the environment. In the corporate realm, businesses armed with CF data can optimize operations, reduce waste, and adopt greener technologies, leading to both environmental and economic benefits. In this study, carbon emissions—a significant global issue—are investigated through the lens of the ISO 14067-ISO Product Based Carbon Footprint (CF) standard, focusing on the operations of a mold company. The primary innovation lies in meticulously tracing every stage of plastic bottle blow mold production, the most prevalent product in the mold industry, from its raw material input to its final form as a mold in the factory. Subsequently, detailed calculations and analysis are conducted to quantify the carbon footprint associated with this process and its impact on the environment. The calculated CF for one ton of PBBM produced by Petka Mold Industry is presented. This study fills a critical gap in the literature by providing a holistic understanding of the carbon footprint of plastic bottle blow mold (PBBM) production, thereby offering valuable insights for managing carbon emissions and promoting sustainability within the mold industry. By integrating a life cycle product carbon footprint thinking into industrial practices, a greener, more sustainable future can be paved, mitigating the ecological footprint of the PBBM.

Detection of Female Anopheles Mosquito-Infected Cells: Exploring CNN, ReLU, and Sigmoid Activation Methods

INTRODUCTION: Deep learning uses multi-layer neural networks where the algorithm decides for itself whether aspects are essential for analysis based on the raw input. In general, deep learning networks get better as more data is used to train them. For a variety of applications, convolutional neural networks are frequently used to analyse, categorize, and detect images. OBJECTIVES: The proposed system technique is used for automated analysis of malaria-detecting frameworks. A female Anopheles mosquito bite is the primary method of transmission of the blood disease malaria. It is still common to manually count and identify parasitized cells during microscopic examination of either thick or thin layers of haemoglobin, which takes time for disease prognosis. METHODS: The current research uses a neural network based on convolution to catalogue images of cells with and without malaria infection. This method improves the precision of classification for the datasets under study. The ReLu activation function used by this model enables it to learn more quickly and perform more effectively. RESULTS: The prediction of infected and healthy cells was done accurately by the proposed model, which uses only 3 layers of convolution, and this was the idea behind the implementation. The model achieved an improved accuracy of 99.77% across 12 iterations (epochs). CONCLUSION: The proposed model is straightforward and successful in differentiating between malaria-infected and uninfected cells.

CGAN-ECT: Reconstruction of Electrical Capacitance Tomography images from capacitance measurements using Conditional Generative Adversarial Networks

Due to the rapid growth of applications of Electrical Capacitance Tomography (ECT) across diverse industrial sectors, there is a pressing demand for swift and high-fidelity methodologies for image reconstruction from raw capacitance measurements. Deep learning, as an effective non-linear mapping tool for complicated functions, has been going viral in many fields including electrical tomography. This paper introduces a pioneering contribution by proposing a Conditional Generative Adversarial Network (CGAN) model explicitly tailored for the purpose of reconstructing ECT images from capacitance measurements. We propose a novel modulator to transform the raw input capacitance measurements into a 2D image. To the best of our knowledge, this is the first time to represent the capacitance measurements in an image form. We have created a new massive ECT dataset of synthetic image-measurements pairs for training and testing purposes of the proposed model. This extensive ECT dataset comprises 320,000 synthetic image-measurement pairs. It forms the basis for evaluating the feasibility and generalizability of the CGAN-ECT model, with assessments conducted on testing data, contaminated data, and flow patterns which are intentionally excluded from the model’s training phase. The evaluation results prove the efficacy of the proposed CGAN model in generating ECT images with heightened accuracy compared to traditional methods and alternative deep learning-based image reconstruction algorithms. The CGAN-ECT model has succeeded in a achieving an average image correlation coefficient exceeding 99.3%, coupled with an average relative image error of approximately 0.07.

An automatic mesh generator for coupled 1D–2D hydrodynamic models

Abstract. Two-dimensional (2D), depth-averaged shallow water equation (SWE) models are routinely used to simulate flooding in coastal areas – areas that often include vast networks of channels and flood-control topographic features and/or structures, such as barrier islands and levees. Adequately resolving these features within the confines of a 2D model can be computationally expensive, which has led to coupling 2D simulation tools to less expensive one-dimensional (1D) models. Under certain 1D–2D coupling approaches, this introduces internal constraints that must be considered in the generation of the 2D computational mesh used. In this paper, we further develop an existing automatic unstructured mesh generation tool for SWE models, ADMESH+, to sequentially (i) identify 1D constraints from the raw input data used in the mesh generation process, namely the digital elevation model (DEM) and land–water delineation data; (ii) distribute grid points along these internal constraints, according to feature curvature and user-prescribed minimum grid spacing; and (iii) integrate these internal constraints into the 2D mesh size function and mesh generation processes. The developed techniques, which include a novel approach for determining the so-called medial axis of a polygon, are described in detail and demonstrated on three test cases, including two inland watersheds with vast networks of channels and a complex estuarine system on the Texas, USA, coast.

Study on Drying of Seaweeds and Importance of Glass Transition and Stabilization

In this study, the thermal transition of seaweed Saccharina latissima (raw and blanched) during drying and quality stabilization was considered in view of understanding physico-chemical changes, color changes, sorption changes and thermal property changes with respect to drying kinetics. The variations in the effective moisture diffusivity coefficient with shrinkage changes and temperature lie between 1.0 and 5.0 × 10−10 m2 s−1 (raw) and 0.5 and 3.6 × 10−10 m2 s−1 (blanched), respectively. Noticeable physical and chemical changes were observed during longer drying times, especially in the case of blanched seaweeds. At the temperature of 38.0 °C, a more yellow-colored product was obtained from raw form input materials. The blanched seaweeds accumulated moisture in a linear manner with an increase in the relative humidity of the drying air in the range of 20.0~80.0%, which resulted in high level of hysteresis between the sorption and desorption behavior. Shrinkage changes during the drying of blanched and raw samples were also calculated. The thermal stabilization of raw and blanched forms started at different initial moisture contents showed changed glass transition phenomena during a wide range of temperature sand one melting endotherm between 141.9 and 167.9 °C. Some glass transitions were driven by the presence of water-soluble contents in the material. The dried seaweeds at low temperatures showed a partial glassy state and an amorphous/crystalline state. This study evaluated the effects of process parameters on the properties of dried product.

Physical scene understanding

AbstractCurrent AI systems still fail to match the flexibility, robustness, and generalizability of human intelligence: how even a young child can manipulate objects to achieve goals of their own invention or in cooperation, or can learn the essentials of a complex new task within minutes. We need AI with such embodied intelligence: transforming raw sensory inputs to rapidly build a rich understanding of the world for seeing, finding, and constructing things, achieving goals, and communicating with others. This problem of physical scene understanding is challenging because it requires a holistic interpretation of scenes, objects, and humans, including their geometry, physics, functionality, semantics, and modes of interaction, building upon studies across vision, learning, graphics, robotics, and AI. My research aims to address this problem by integrating bottom‐up recognition models, deep networks, and inference algorithms with top‐down structured graphical models, simulation engines, and probabilistic programs.

S4D-ECG: A Shallow State-of-the-Art Model for Cardiac Abnormality Classification

PurposeThis study introduces an algorithm specifically designed for processing unprocessed 12-lead electrocardiogram (ECG) data, with the primary aim of detecting cardiac abnormalities.MethodsThe proposed model integrates Diagonal State Space Sequence (S4D) model into its architecture, leveraging its effectiveness in capturing dynamics within time-series data. The S4D model is designed with stacked S4D layers for processing raw input data and a simplified decoder using a dense layer for predicting abnormality types. Experimental optimization determines the optimal number of S4D layers, striking a balance between computational efficiency and predictive performance. This comprehensive approach ensures the model's suitability for real-time processing on hardware devices with limited capabilities, offering a streamlined yet effective solution for heart monitoring.ResultsAmong the notable features of this algorithm is its strong resilience to noise, enabling the algorithm to achieve an average F1-score of 81.2% and an AUROC of 95.5% in generalization. The model underwent testing specifically on the lead II ECG signal, exhibiting consistent performance with an F1-score of 79.5% and an AUROC of 95.7%.ConclusionIt is characterized by the elimination of pre-processing features and the availability of a low-complexity architecture that makes it suitable for implementation on numerous computing devices because it is easily implementable. Consequently, this algorithm exhibits considerable potential for practical applications in analyzing real-world ECG data. This model can be placed on the cloud for diagnosis. The model was also tested on lead II of the ECG alone and has demonstrated promising results, supporting its potential for on-device application.

Terrestrial Evapotranspiration Over China From 1982 to 2020: Consistency of Multiple Data Sets and Impact of Input Data

AbstractDue to limited direct measurements, regional or global terrestrial evapotranspiration (ET) is generally derived from a combination of meteorological and satellite observations. Although the inhomogeneity of the observed climate data has been widely reported, its impact on the calculated ET has not been adequately quantified. This study aimed to calculate ET using the modified Penman‐Monteith (MPM) model with raw and homogenized meteorological data. Additionally, we compared the calculated ET with those estimates from variable methods (water balance, satellite‐based, and reanalysis) in China and its six major river basins from 1982 to 2020. During the overlapping period of 1997–2018, ET calculated from raw input data decreased slightly at −0.39 mm yr−2 (p = 0.64) in China, whereas homogenized ET showed a significant increasing trend of 0.93 mm yr−2 (p = 0.02), with a better agreement with water balance ET (1.93 mm yr−2, p = 0). Global Land Evaporation Amsterdam Model (GLEAM) and Modern‐Era Retrospective Analysis for Research and Applications, version 2 (MERRA2) could reproduce the increasing trends with 2.08 mm yr−2 (p = 0) and 2.59 mm yr−2 (p = 0). The intercomparison of input variables (solar radiation, relative humidity, wind speed, precipitation, and air temperature) among ET products revealed substantial differences, which can account for the discrepancies in ET estimates. Homogenized ET, GLEAM and MERRA2 exhibited significant increasing trends in China and most river basins from 1982 to 2020. Our findings underscore the importance of utilizing homogenized input data for more accurate ET estimation.

Intelligent robotics harvesting system process for fruits grasping prediction

This paper proposes and executes an in-depth learning-based image processing approach for self-picking apples. The system includes a lightweight one-step detection network for fruit recognition. As well as computer vision to analyze the point class and anticipate a correct approach position for each fruit before grabbing. Using the raw inputs from a high-resolution camera, fruit recognition and instance segmentation are done on RGB photos. The computer vision classification and grasping systems are integrated and outcomes from tree-grown foods are provided as input information and output methodology poses for every apple and orange to robotic arm execution. Before RGB picture data is acquired from laboratory and plantation environments, the developed vision method will be evaluated. Robot harvest experiment is conducted in indoor as well as outdoor to evaluate the proposed harvesting system's performance. The research findings suggest that the proposed vision technique can control robotic harvesting effectively and precisely where the success rate of identification is increased above 95% in case of post prediction process with reattempts of less than 12%.

Exploiting Cross-Modal Prediction and Relation Consistency for Semisupervised Image Captioning.

The task of image captioning aims to generate captions directly from images via the automatically learned cross-modal generator. To build a well-performing generator, existing approaches usually need a large number of described images (i.e., supervised image-sentence pairs), requiring a huge effects on manual labeling. However, in real-world applications, a more general scenario is that we only have limited amount of described images and a large number of undescribed images. Therefore, a resulting challenge is how to effectively combine the undescribed images into the learning of cross-modal generator (i.e., semisupervised image captioning). To solve this problem, we propose a novel image captioning method by exploiting the cross-modal prediction and relation consistency (CPRC), which aims to utilize the raw image input to constrain the generated sentence in the semantic space. In detail, considering that the heterogeneous gap between modalities always leads to the supervision difficulty while using the global embedding directly, CPRC turns to transform both the raw image and corresponding generated sentence into the shared semantic space, and measure the generated sentence from two aspects: 1) prediction consistency: CPRC utilizes the prediction of raw image as soft label to distill useful supervision for the generated sentence, rather than employing the traditional pseudo labeling and 2) relation consistency: CPRC develops a novel relation consistency between augmented images and corresponding generated sentences to retain the important relational knowledge. In result, CPRC supervises the generated sentence from both the informativeness and representativeness perspectives, and can reasonably use the undescribed images to learn a more effective generator under the semisupervised scenario. The experiments show that our method outperforms state-of-the-art comparison methods on the MS-COCO "Karpathy" offline test split under complex nonparallel scenarios, for example, CPRC achieves at least 6% improvements on the CIDEr-D score.

Applying machine learning random forest (RF) method in predicting the cement products with a co-processing of input materials: Optimizing the hyperparameters

Co-processing recycled waste during cement production, i.e., using alternative materials such as secondary raw materials or secondary raw fuels, is widely practiced in developed countries. Alternative raw materials or fuels contain high concentrations of heavy metals and other hazardous chemicals, which might lead to the potential for dangerous heavy metals and hazardous chemicals to be transferred to clinker or cement products, resulting in exposure and emissions to people or the environment. Managing input materials and predicting which inputs affect the final concentration is essential to prevent potential hazards. We used the data of six heavy metals by input raw materials and input fuels of cement manufacturers in 2016–2017. The concentrations of Pb and Cu in cement were about 10–200 times and 4 to 200 times higher than other heavy metals (Cr, As, Cd, Hg), respectively.We profiled the influence of heavy metal concentration of each input material using the principal component analysis (PCA), which analyzed the leading causes of each heavy metal. The Random Forest (RF) ensemble model predicted cement heavy metal concentrations according to input materials. In the case of Cu, Cd, and Cr, the training performance showed R square values of 0.71, 0.71, and 0.92, respectively, as a result of predicting the cement heavy metal concentration according to the heavy metal concentration of each cement input material using the RF model, which is a machine learning model. The results of this study show that the RF model can be used to predict the amount and concentration of alternative raw materials and alternative fuels by controlling the concentration of heavy metals in cement through the concentration of heavy metals in the input materials.

Multimodal consistency-specificity fusion based on information bottleneck for sentiment analysis

Sentiment analysis, a subtask of affective computing, endows machines with the ability to sense and comprehend emotions. Recently, research attention has shifted from traditional isolated modality to ubiquitous multi-modalities, requiring to model the complex relationships between modalities and extract the task-relevant information. However, current methods only focus on modality consistency neglecting the complementary relationship and specificity information within each modality. Furthermore, most studies regard multimodal fusion as a mere process of information integration without considering the control of information flow. We propose a variational model that explicitly decomposes unimodal representations into two types, capturing consistency and specificity information, respectively. Following the information bottleneck principle, the implementation optimizes the fusion representation by maximizing its mutual information with consistency and specificity representations while minimizing mutual information with raw inputs. With information integrity constraints and task label supervision, the fusion representation preserves task-relevant information and discards irrelevant noise. Finally, quantitative and qualitative experiments on two benchmark datasets show that our method achieves competitive performance compared to recent baselines, for multimodal sentiment analysis.

Speaker voice normalization for end-to-end speech translation

Speaker voices exhibit acoustic variation. Our preliminary experiments reveal that normalized voice can significantly improve end-to-end speech translation. To mitigate the negative impact of acoustic voice variation across speakers on speech translation, we propose SVN-ST, a Speaker-Voice-Normalized end-to-end Speech Translation framework. In SVN-ST, we use synthetic speech inputs generated from a Text-to-Speech system to complement raw speech inputs. In order to explore synthetic speech inputs, we introduce two essential components for SVN-ST: an alignment adapter at the encoder side and a normalized speech knowledge distillation module at the decoder side. The former forces the representations of raw speech inputs to be close to those of synthetic (normalized) speech inputs while the latter attempts to guide the translations of raw speech inputs with those yielded from synthetic speech inputs. Two additional losses are also defined to equip with the two components. Experimental results on the MuST-C benchmark dataset demonstrate that SVN-ST outperforms previous state-of-the-art end-to-end non-normalized speech translation systems by 0.4 BLEU and cascaded speech translation systems by 2.3 BLEU. On the Covost 2 testset, SVN-ST also outperforms other normalized speech methods on robustness. Further analyses suggest that our model effectively aligns speech representations from different speakers, enhances robustness, and significantly improves sentence-level translation quality.

Audio-Based Emotion Recognition Using Self-Supervised Learning on an Engineered Feature Space.

Emotion recognition models using audio input data can enable the development of interactive systems with applications in mental healthcare, marketing, gaming, and social media analysis. While the field of affective computing using audio data is rich, a major barrier to achieve consistently high-performance models is the paucity of available training labels. Self-supervised learning (SSL) is a family of methods which can learn despite a scarcity of supervised labels by predicting properties of the data itself. To understand the utility of self-supervised learning for audio-based emotion recognition, we have applied self-supervised learning pre-training to the classification of emotions from the CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU- MOSEI)'s acoustic data. Unlike prior papers that have experimented with raw acoustic data, our technique has been applied to encoded acoustic data with 74 parameters of distinctive audio features at discrete timesteps. Our model is first pre-trained to uncover the randomly masked timestamps of the acoustic data. The pre-trained model is then fine-tuned using a small sample of annotated data. The performance of the final model is then evaluated via overall mean absolute error (MAE), mean absolute error (MAE) per emotion, overall four-class accuracy, and four-class accuracy per emotion. These metrics are compared against a baseline deep learning model with an identical backbone architecture. We find that self-supervised learning consistently improves the performance of the model across all metrics, especially when the number of annotated data points in the fine-tuning step is small. Furthermore, we quantify the behaviors of the self-supervised model and its convergence as the amount of annotated data increases. This work characterizes the utility of self-supervised learning for affective computing, demonstrating that self-supervised learning is most useful when the number of training examples is small and that the effect is most pronounced for emotions which are easier to classify such as happy, sad, and angry. This work further demonstrates that self-supervised learning still improves performance when applied to the embedded feature representations rather than the traditional approach of pre-training on the raw input space.

A Visual Computing Unified Application Using Deep Learning and Computer Vision Techniques

Vision Studio aims to utilize a diverse range of modern deep learning and computer vision principles and techniques to provide a broad array of functionalities in image and video processing. Deep learning is a distinct class of machine learning algorithms that utilize multiple layers to gradually extract more advanced features from raw input. This is beneficial when using a matrix as input for pixels in a photo or frames in a video. Computer vision is a field of artificial intelligence that teaches computers to interpret and comprehend the visual domain. The main functions implemented include deepfake creation, digital ageing (de-ageing), image animation, and deepfake detection. Deepfake creation allows users to utilize deep learning methods, particularly autoencoders, to overlay source images onto a target video. This creates a video of the source person imitating or saying things that the target person does. Digital aging utilizes generative adversarial networks (GANs) to digitally simulate the aging process of an individual. Image animation utilizes first-order motion models to create highly realistic animations from a source image and driving video. Deepfake detection is achieved by using advanced and highly efficient convolutional neural networks (CNNs), primarily employing the EfficientNet family of models.