Multimodal Applications Research Articles

Aircraft human‐machine interaction assistant design: A novel multimodal data processing and application framework

AbstractDuring aircraft operations, pilots rely on human‐machine interaction platforms to access essential information services. However, the development of a highly usable aerial assistant necessitates the incorporation of two interaction modes: active‐command and passive‐response modes, along with three input modes: voice inputs, situation inputs, and plan inputs. This research focuses on the design of an aircraft human‐machine interaction assistant (AHMIA), which serves as a multimodal data processing and application framework for human‐to‐machine interaction in a fully voice‐controlled manner. For the voice mode, a finetuned FunASR model is employed, leveraging private aeronautical datasets to enable specific aeronautical speech recognition. For the situation mode, a hierarchical situation events extraction model is proposed, facilitating the utilization of high‐level situational features. For the plan mode, a multi‐formations double‐code network plan diagram with a timeline is utilized to effectively represent plan information. Notably, to bridge the gap between human language and machine language, a hierarchical knowledge engine named process‐event‐condition‐order‐skill (PECOS) is introduced. PECOS provides three distinct products: the PECOS model, the PECOS state chart, and the PECOS knowledge description. Simulation results within the air confrontation scenario demonstrate that AHMIA enables active‐command and passive‐response interactions with pilots, thereby enhancing the overall interaction modality.

Multi-Objective Design and Optimization of Hardware-Friendly Grid-Based Sparse MIMO Arrays

A comprehensive design framework is proposed for optimizing sparse MIMO (multiple-input, multiple-output) arrays to enhance multi-target detection. The framework emphasizes efficient utilization of antenna resources, including strategies for minimizing inter-element mutual coupling and exploring alternative grid-based sparse array (GBSA) configurations by efficiently separating interacting elements. Alternative strategies are explored to enhance angular beamforming metrics, including beamwidth (BW), peak-to-sidelobe ratio (PSLR), and grating lobe limited field of view. Additionally, a set of performance metrics is introduced to evaluate virtual aperture effectiveness and beamwidth loss factors. The framework explores optimization strategies for the partial sharing of antenna elements, specifically tailored for multi-mode radar applications, utilizing the desirability function to enhance performance across various operational modes. A novel machine learning initialization approach is introduced for rapid convergence. Key observations include the potential for peak-to-sidelobe ratio (PSLR) reduction in dense arrays and insights into GBSA feasibility and performance compared to uniform arrays. The study validates the efficacy of the proposed framework through simulated and measured results. The study emphasizes the importance of effective sparse array processing in multi-target scenarios and highlights the advantages of the proposed design framework. The proposed design framework for grid-spaced sparse arrays stands out for its superior efficiency and applicability in processing hardware compared to both uniform and non-uniform arrays.

Advances of Triboelectric and Piezoelectric Nanogenerators toward Continuous Monitoring and Multimodal Applications in the New Era

Abstract Benefiting from the widespread potential applications in the era of Internet of Thing (IoT) and metaverse, triboelectric and piezoelectric nanogenerators have attracted considerably increasing attention. Their outstanding characteristics, such as self-powered ability, high output performance, integration compatibility, cost effectiveness, simple configurations and versatile operation modes, could effectively expand the lifetime of vastly distributed wearable, implantable and environmental devices, eventually achieving self-sustainable, maintenance-free and reliable systems. However, current triboelectric/piezoelectric based active (i.e., self-powered) sensors still encounter serious bottlenecks in continuous monitoring and multimodal applications due to their intrinsic limitations of monomodal kinetic response and discontinuous transient output. This work systematically summarizes and evaluates the recent research endeavours to addressing the above challenges, with detailed discussions on the challenge origins, designing strategies, device performance and corresponding diverse applications. Finally, conclusions and outlook regarding the research gap in self-powered continuous multimodal monitoring systems are provided, proposing the necessity of future research development in this field.

Conditional Deployable Biometrics: Matching Periocular and Face in Various Settings

In this paper, we introduce the concept of Conditional Deployable Biometrics (CDB), designed to deliver consistent performance across various biometric matching scenarios, including intra-modal, multimodal, and cross-modal applications. The CDB framework provides a versatile and deployable biometric authentication system that ensures reliable matching regardless of the biometric modality being used. To realize this framework, we have developed CDB-Net, a specialized deep neural network tailored for handling both periocular and face biometric modalities. CDB-Net is engineered to handle the unique challenges associated with these different modalities while maintaining high accuracy and robustness. Our extensive experimentation with CDB-Net across five diverse and challenging in-the-wild datasets illustrates its effectiveness in adhering to the CDB paradigm. These datasets encompass a wide range of real-world conditions, further validating the model’s capability to manage variations and complexities inherent in biometric data. The results confirm that CDB-Net not only meets but exceeds expectations in terms of performance, demonstrating its potential for practical deployment in various biometric authentication scenarios.

Advances and prospects of multi-modal ophthalmic artificial intelligence based on deep learning: a review.

In recent years, ophthalmology has emerged as a new frontier in medical artificial intelligence (AI) with multi-modal AI in ophthalmology garnering significant attention across interdisciplinary research. This integration of various types and data models holds paramount importance as it enables the provision of detailed and precise information for diagnosing eye and vision diseases. By leveraging multi-modal ophthalmology AI techniques, clinicians can enhance the accuracy and efficiency of diagnoses, and thus reduce the risks associated with misdiagnosis and oversight while also enabling more precise management of eye and vision health. However, the widespread adoption of multi-modal ophthalmology poses significant challenges. In this review, we first summarize comprehensively the concept of modalities in the field of ophthalmology, the forms of fusion between modalities, and the progress of multi-modal ophthalmic AI technology. Finally, we discuss the challenges of current multi-modal AI technology applications in ophthalmology and future feasible research directions. In the field of ophthalmic AI, evidence suggests that when utilizing multi-modal data, deep learning-based multi-modal AI technology exhibits excellent diagnostic efficacy in assisting the diagnosis of various ophthalmic diseases. Particularly, in the current era marked by the proliferation of large-scale models, multi-modal techniques represent the most promising and advantageous solution for addressing the diagnosis of various ophthalmic diseases from a comprehensive perspective. However, it must be acknowledged that there are still numerous challenges associated with the application of multi-modal techniques in ophthalmic AI before they can be effectively employed in the clinical setting.

Minting Progress: Multifunctional Bismuth Oxide Nanoparticles via Mentha arvensis extract for Biomedical Advancements

AbstractTo surpass the constraints associated with conventional therapeutic methods in clinical therapy, nanoparticles have been rapidly prepared and developed as a separate treatment method for diseases. In this study, we have synthesized bismuth oxide nanoparticles (BiNPs) using aqueous leaf extract of Mentha arvensis (M. arvensis) as a biogenic approach. The formation of BiNPs was confirmed by the characteristic peak observed at 276 nm in the UV–vis spectrum and PXRD analysis exhibited the crystalline nature with monoclinic phase. The TEM images revealed the quasi‐spherical shape of BiNPs with an average particle size of 12 nm. For biomedical studies, BiNPs exhibited significant cytotoxicity against A431 cell line (lung cancer) with IC50 value of 92.45 µg/mL. Further studies revealed significantly higher apoptosis, antibacterial, anti‐inflammatory, antidiabetic, and antioxidant activities of the biosynthesized BiNPs. The collective results from the study suggest that the M. arvensis mediated synthesis of multifunctional BiNPs enhances their biological potential, positioning it as a potential candidate for the multimodal therapeutic applications.

Multimodal applications of green carbon dots derived from Potentilla indica (mock strawberry): Antioxidant, antimicrobial, and quenching based quercetin sensor

This study utilized a simple hydrothermal technique to synthesize negatively charged carbon dots from Potentilla indica fruit (MSTCDs) with antioxidant and broad-spectrum antimicrobial properties, as well as a specific capacity to detect quercetin. The characterization of MSTCDs was performed using spectrophotometry (UV–vis, fluorescence, X-ray photoelectron, and Fourier-transform infrared), zeta potential, and transmission electron microscopy. The prepared MSTCDs had an average size of 8 ± 0.22 nm and zeta potential of −21 mV. The prepared MSTCDs showed negligible cytotoxicity and were found to be biocompatible with human microglial cells. MSTCDs used the static quenching and inner filter effect as their two sensing mechanisms to detect quercetin. The fluorescence intensity of the MSTCDs decreased as the dose of quercetin increased, showing a linear correlation within the dose range of 0.01–70 μM (R2 = 0.9980) and the lowest detection limit of 2 nM. The prepared MSTCDs were applied to detect quercetin in the orange and grape juice, yielding values of 3.27 and 7.04 μM, respectively. Further, the MSTCDs displayed antioxidant activity by scavenging 2,2′-azino-bis (3-ethylbenzothiazoline-6-sulfonic acid) (ABTS), 2,2-diphenyl-1-picrylhydrazyl, and hydroxyl radicals, with an EC50 value of 9.3, 140.82, and 187.91 µg/mL, respectively. The antiradical activity of MSTCDs was slightly lower than the ascorbic acid except for ABTS radicals, where MSTCDs showed superior scavenging ability (EC50 = 9.3 µg/mL) compared to the control (EC50 = 33.7 µg/mL). MSTCDs exhibited dose-dependent antimicrobial activity against fungi, Gram-negative bacteria, and Gram-positive bacteria. Thus, the developed CDs showed potential as antioxidant, antimicrobial, and quenching-based quercetin sensor.

Investigation of a cascaded piezoelectric transducer with cone horn for multi-mode power ultrasound application

To meet the ultrasonic application requirement of high power intensity and large mechanical displacement, a new theoretical method is used to study the multi-mode characteristic of the cascaded piezoelectric transducer with cone horn in this paper. Based on equivalent circuit and Kirchhoff’s law to obtain the frequency equation and vibration velocity expression of the transducer, then the resonance frequency and effective electromechanical coupling coefficient can be calculated, the velocity amplification ratio can be obtained in the meantime. This method is distinguish from traditional theoretical analysis, which avoids complex transformation of the multi-excitation sources in cascaded structure, and can calculate more performance parameters. The relationships between these performance parameters and the excitation position of the piezoelectric stack, the length and output end radius of the cone horn are analyzed and compared with the numerical simulation, and the optimized parameters of the transducer size are given. A transducer is manufactured for experimental test, the results show that the experimental value is in good agreement with theoretical calculation and finite element simulation. This work is expected to be used in the optimal analysis of the multi-mode transducer in high power ultrasonic field.

Thermal modeling and simulation applied to novel microwave hybrid heating process

Owing to the unique encouraging characteristics of microwave hybrid heating, primarily volumetric heating, and additional potentials such as being repeatable, quick, economical, and green; it has been utilized in various processing techniques. The efficient joining of mild steel pipes through microwave hybrid heating in a multimode applicator at 2.45 GHz and 900 W for an operational time of 480 s has already been performed. The modeling and simulation of the process have been performed in this research paper as the numerical analysis of the working environment is crucial for evaluating various aspects of a technique, decreases process-design cycle time, and found to be more economical than experimental trials. The numerical analysis provides in-depth insight-taking into consideration of the electromagnetic field distribution, its interaction with the materials, heat generation and transfer, along with the thermal analysis of the experimental assembly, in addition to the comprehensive parametric analysis. The numerical model of the assembled set-up was developed in order to simulate a real-life heating environment by solving electromagnetic and heat transfer equations and providing analytically predicted results with an accuracy of 3.75% against the experimental results. The analytical modeling and simulation have been strategically fragmented into three phases which are pre-processing, processing, and post-processing phase and elucidated extensively, providing a systematic working of the analytical model. This research will be utilized further in optimizing the microwave hybrid heating process in order to make it time-efficient and inexpensive for its applications to industrial environments.

The Panhellenic student competition “Hack the Art: Yanoulis Halepas”: Extending art experience, communication, and learning through multimodal digital applications

This paper presents the case of an innovative national digital student competition for school groups of secondary education, entitled: “Hack the Art: Yanoulis Halepas.” Conducted remotely in Greece during the 2022 – 2023 school year, the competition was organized by the Onassis Foundation and the Onassis Library in partnership with the Teloglion Fine Arts Foundation, the National Gallery of Greece–Alexandros Soutsos Museum, the Cultural Foundation of Tinos, and the Cultural Center of Panormos-Municipality of Tinos. Accredited by the Hellenic Ministry for Education and Religious Affairs, it was realized under the aegis of the General Secretariat for Greeks Abroad and Public Diplomacy. This competition was the fourth iteration of this digital format since 2020. This time, we invited student teams from Greek high schools, both in Greece and abroad, led by at least one educator, to learn about the life and work of the great Greek sculptor Yanoulis Halepas and to communicate art by creating projects of augmented reality, virtual reality, digital storytelling, and 3D video games for PC and android devices. This innovative educational project enhanced students’ experience and conceptual understanding of art, motivating them to interact with cultural heritage and experiment with multimodal digital tools. Furthermore, it offered an alternative educational paradigm for teaching history, literature, visual arts, and technology by applying object-based learning, game-based learning, and project-based learning methodologies. According to the assessment results, this learning approach successfully integrated significant artworks into the school curriculum, extending their lifespan and visibility through digital transformations and reinterpretations.

Microstructurally resolved electrochemical evolution of mechanical- and irradiation-induced damage in nuclear alloys

There is a need for high-throughput, scale-relevant, and direct electrochemical analysis to understand the corrosion behavior and sensitivity of nuclear materials that are exposed to extreme (high pressure, temperature, and radiation exposure) environments. We demonstrate the multi-scale, multi-modal application of scanning electrochemical cell microscopy (SECCM) to electrochemically profile corrosion alterations in nuclear alloys in a microstructurally resolved manner. Particularly, we identify that both mechanically deformed and irradiated microstructures show reduced charge-transfer resistance that leads to accelerated oxidation. We highlight that the effects of mechanical deformation and irradiation are synergistic, and may in fact, superimpose each other, with implications including general-, galvanic-, and/or irradiation-activated stress-corrosion cracking. Taken together, we highlight the ability of non-destructive, electrochemical interrogations to ascertain how microstructural alterations result in changes in the corrosion tendency of a nuclear alloy: knowledge which has implications to rank, qualify and examine alloys for use in nuclear construction applications.

Multimodal anti-counterfeiting and optical storage application based on luminescence reversible modification and color change of photochromic phosphor

Reversible upconversion luminescence (RUCL) modification based on photochromism has received considerable interest due to its potential applications in optical storage and invisible optical anti-counterfeiting. Herein, white β-Ba2ScAlO5: Yb3+, Er3+ (β-BSAO-YE) phosphor was irradiated with 254 nm light, which caused the phosphor to turn blue due to the increase of oxygen vacancy. The blue β-BSAO-YE phosphor returns to its original color upon stimulation with 808 nm laser light or 125 °C, exhibiting a double photo-induced reversible photochromic change. The green and red upconversion luminescence (UCL) of the β-BSAO-YE were reversibly modified according to their photochromic properties. The UCL modification changes the luminescence color of the pattern, indicating that β-BSAO-YE phosphor is an ideal compound anti-counterfeit agent. The light information recorded on the β-BSAO-YE binary photochromic dot matrix can be read under ultraviolet (UV) and near-infrared (NIR) excitation, demonstrating the potential application of β-BSAO-YE phosphors as optical data storage media. In addition, the cyclic experiment showed good photochromic and UCL modulation repeatability. The new luminescent β-BSAO-YE not only proposes a new way to exploring upconversion red light materials, but also provides new materials for optical storage, optical information and optical anti-counterfeiting.

A Multilayer Architecture towards the Development and Distribution of Multimodal Interface Applications on the Edge.

Today, Smart Assistants (SAs) are supported by significantly improved Natural Language Processing (NLP) and Natural Language Understanding (NLU) engines as well as AI-enabled decision support, enabling efficient information communication, easy appliance/device control, and seamless access to entertainment services, among others. In fact, an increasing number of modern households are being equipped with SAs, which promise to enhance user experience in the context of smart environments through verbal interaction. Currently, the market in SAs is dominated by products manufactured by technology giants that provide well designed off-the-shelf solutions. However, their simple setup and ease of use come with trade-offs, as these SAs abide by proprietary and/or closed-source architectures and offer limited functionality. Their enforced vendor lock-in does not provide (power) users with the ability to build custom conversational applications through their SAs. On the other hand, employing an open-source approach for building and deploying an SA (which comes with a significant overhead) necessitates expertise in multiple domains and fluency in the multimodal technologies used to build the envisioned applications. In this context, this paper proposes a methodology for developing and deploying conversational applications on the edge on top of an open-source software and hardware infrastructure via a multilayer architecture that simplifies low-level complexity and reduces learning overhead. The proposed approach facilitates the rapid development of applications by third-party developers, thereby enabling the establishment of a marketplace of customized applications aimed at the smart assisted living domain, among others. The supporting framework supports application developers, device owners, and ecosystem administrators in building, testing, uploading, and deploying applications, remotely controlling devices, and monitoring device performance. A demonstration of this methodology is presented and discussed focusing on health and assisted living applications for the elderly.

Mechanochromic luminescent material with high quantum yield for multi-mode anti-counterfeiting applications

The multi-mode anti-counterfeiting materials, constructed by mechanochromic luminescent materials with high quantum yield, can effectively prevent illegal counterfeiting, but their design and construction still present significant challenges. Herein, a twisting D-A-D conjugated molecule, FBtF, is obtained with a special hybridized local and charge transfer excited state, and shows approximately 100 % photoluminescence quantum yield in the coating film. The self-assembled FBtF microplates exhibit bathochromic-shifted emission from green to orange, accompanied by the photoluminescence quantum yield increase from 47.8 % to 89.0 % upon grinding, which is the highest in reported MCL materials, to the best of our knowledge. In contrast, a D-π-D type molecule (FBenF) and a D-D’-D molecule (FAnF) maintain the blue-emission after grinding. Through mechanistic investigation, the force-induced molecular conformational planarization and the breakage of the intermolecular charge transfer collaboratively lead to the emission-color change and photoluminescence quantum yield increase. The FBtF microplates are designed as ink for anti-counterfeiting applications based on multi-mode output signals, including the UV light-excited image and the grinding-induced emission color change together with the photoluminescence quantum yield increase. The written tags exhibit high stability in various harsh environments, and demonstrate superior anti-counterfeiting performance in practical applications, such as medicine anti-counterfeiting.

Large Language Models in Healthcare and Medical Domain: A Review

The deployment of large language models (LLMs) within the healthcare sector has sparked both enthusiasm and apprehension. These models exhibit the remarkable ability to provide proficient responses to free-text queries, demonstrating a nuanced understanding of professional medical knowledge. This comprehensive survey delves into the functionalities of existing LLMs designed for healthcare applications and elucidates the trajectory of their development, starting with traditional Pretrained Language Models (PLMs) and then moving to the present state of LLMs in the healthcare sector. First, we explore the potential of LLMs to amplify the efficiency and effectiveness of diverse healthcare applications, particularly focusing on clinical language understanding tasks. These tasks encompass a wide spectrum, ranging from named entity recognition and relation extraction to natural language inference, multimodal medical applications, document classification, and question-answering. Additionally, we conduct an extensive comparison of the most recent state-of-the-art LLMs in the healthcare domain, while also assessing the utilization of various open-source LLMs and highlighting their significance in healthcare applications. Furthermore, we present the essential performance metrics employed to evaluate LLMs in the biomedical domain, shedding light on their effectiveness and limitations. Finally, we summarize the prominent challenges and constraints faced by large language models in the healthcare sector by offering a holistic perspective on their potential benefits and shortcomings. This review provides a comprehensive exploration of the current landscape of LLMs in healthcare, addressing their role in transforming medical applications and the areas that warrant further research and development.

Deep learning for cross-domain data fusion in urban computing: Taxonomy, advances, and outlook

As cities continue to burgeon, Urban Computing emerges as a pivotal discipline for sustainable development by harnessing the power of cross-domain data fusion from diverse sources (e.g., geographical, traffic, social media, and environmental data) and modalities (e.g., spatio-temporal, visual, and textual modalities). Recently, we are witnessing a rising trend that utilizes various deep-learning methods to facilitate cross-domain data fusion in smart cities. To this end, we propose the first survey that systematically reviews the latest advancements in deep learning-based data fusion methods tailored for urban computing. Specifically, we first delve into data perspective to comprehend the role of each modality and data source. Secondly, we classify the methodology into four primary categories: feature-based, alignment-based, contrast-based, and generation-based fusion methods. Thirdly, we further categorize multi-modal urban applications into seven types: urban planning, transportation, economy, public safety, society, environment, and energy. Compared with previous surveys, we focus more on the synergy of deep learning methods with urban computing applications. Furthermore, we shed light on the interplay between Large Language Models (LLMs) and urban computing, postulating future research directions that could revolutionize the field. We firmly believe that the taxonomy, progress, and prospects delineated in our survey stand poised to significantly enrich the research community. The summary of the comprehensive and up-to-date paper list can be found at https://github.com/yoshall/Awesome-Multimodal-Urban-Computing.

A novel fluoroapatite-type phosphor Ca2Y8(BO4)2(SiO4)4F2:Eu3+ with high quantum efficiency and good thermal stability for multimodal applications☆

A novel Eu3+ doped fluorapatite red phosphor Ca2Y8(BO4)2(SiO4)4F2:Eu3+ with pure phase was synthesized in this study. Density functional theory (DFT) calculation and diffuse reflection spectrum analysis reveal its potential as a matrix for phosphors excited by ultraviolet light. Eu3+ has a 7F0→5L6 transition at 394 nm, and the prepared phosphor exhibits a high emission intensity at 614 nm, which may be attributed to the 5D0-7F2 energy transition at the lower symmetry site of Eu3+. The optimal doping concentration of the phosphor is determined to be 11 mol%, with concentration quenching attributed to the exchange interaction mechanism. The overall color purity of the phosphor is up to 99.88%, with an internal quantum efficiency as high as 91.15%. Notably, Ca2Y8(BO4)2(SiO4)4F2:11 mol%Eu3+ (CYBSF:11 mol%Eu3+) phosphors exhibit good thermal stability, with a thermal quenching temperature (T1/2) of 552 K and the intensity of emission at 423 K still at 88.89% of that at 298 K. The activation energy of the phosphor is up to 0.30287 eV. Its comprehensive luminescence performance surpasses that of commercial red phosphor, making it suitable for near ultraviolet excited warm white light emitting diode (NUV-WLED) with a high color rendering index (Ra = 82) and a correlation color temperature (CCT) of 4339 K. Moreover, the phosphor achieves latent fingerprint visualization and anti-counterfeiting ink on different material surfaces: glass, aluminum foil, plastic and paper. Overall, the fluorapatite CYBSF:11 mol%Eu3+ phosphor holds great potential for multimodal applications due to its high quantum efficiency and good thermal stability.

A novel multifunctional Eu3+/Mn4+ co-doping double perovskite phosphor for applications in LEDs and optical thermometry

In recent years, phosphors have played important roles in solid-state lighting, plant growth lighting and optical temperature measurement. In this work, novel La3Mg2NbO9:Eu3+,Mn4+ phosphors were successfully prepared by the high temperature solid phase method and innovative multimode applications were investigated in optical thermometry, plant growth and White-LEDs. The crystal structure, surface morphology, photoluminescence, temperature sensing properties and luminescence mechanism of La3Mg2NbO9:Eu3+,Mn4+ phosphors were investigated in detail. Benefiting from the different responses of Eu3+ and Mn4+ ions to temperature, the La3Mg2NbO9:0.06Eu3+,0.0095Mn4+ sample shows excellent optical thermometry performance in the range of 303 ∼ 478 K. The maximum values of relative sensitivity (Sr) reach 2.7 % K-1 (@478 K) according to the fluorescence intensity ratio method and the maximum of Sr is 5.2 % K−1 (@428 K) based on the fluorescence lifetime method. Furthermore, the electrochromic spectrum of the red pc-LED prepared using La3Mg2NbO9:0.06Eu3+,0.005Mn4+ phosphor perfectly overlapped with the absorption bands of the photosensitive pigments for planting, and another white pc-LED prepared using La3Mg2NbO9:0.06Eu3+,0.0025Mn4+ phosphor emitted bright white light with a color rendering index of 89.8, color coordinate of (0.3468, 0.3557) and correlated color temperature of 4945 K. All the results demonstrate that the La3Mg2NbO9:Eu3+,Mn4+phosphors are promising for the field of non-contact optical temperature thermometry, plant lighting lamps and pc-LED backlit displays.

Self-Reporting Conjugated Polymer Nanoparticles for Superoxide Generation and Detection.

Conjugated polymer nanoparticles (CPNs or Pdots) have become increasingly popular fluorophores for multimodal applications that combine imaging with phototherapeutic effects. Reports of CPNs in photodynamic therapy applications typically focus on their ability to generate singlet oxygen. Alternatively, CPN excited states can interact with oxygen to form superoxide radical anion and a CPN-based hole polaron, both of which can have deleterious effects on fluorescence properties. Here, we demonstrate that CPNs prepared from the common conjugated polymer poly[(9,9-dioctylfluorenyl-2,7-diyl)-alt-co-(1,4-benzo-{2,1',3}-thiadiazole)] (PFBT, also known as F8BT) generate superoxide upon irradiation. We use the same CPNs to detect superoxide by doping them with a superoxide-responsive hydrocyanine dye developed by Murthy and co-workers. Superoxide induces off-to-on fluorescence switching by converting quenching hydrocyanine dyes to fluorescent cyanine dyes that act as fluorescence resonance energy transfer (FRET) acceptors for PFBT chromophores. Amplified FRET from the multichromophoric CPNs yields fluorescence signal intensities that are nearly 50 times greater than when the dye is excited directly or over 100 times greater when signal readout is from the CPN channel. The dye loading level governs the maximum amount of superoxide that induces a change in fluorescence properties and also influences the rate of superoxide generation by furnishing competitive excited state deactivation pathways. These results suggest that CPNs can be used to deliver superoxide in applications in which it is desirable and provide a caution for fluorescence-based CPN applications in which superoxide can damage fluorophores.

Multimodal federated learning: Concept, methods, applications and future directions

Multimodal learning mines and analyzes multimodal data in reality to better understand and appreciate the world around people. However, how to exploit this rich multimodal data without violating user privacy is a key issue. Federated learning is a privacy-conscious alternative to centralized machine learning, therefore many researchers have combined federated learning with multimodal learning to break down data barriers for the purpose of jointly leveraging multiple modal data from different clients for modeling. In order to provide a systematic summarize of multimodal federated learning, this paper describes the basic mode of multimodal federated learning, multimodal fusion based on federated learning, multimodal federated learning optimization and multimodal federated learning application, and introduces each type of multimodal federated learning methods in detail. Finally, the future research trends of multimodal federated learning are discussed and analyzed, mainly including the optimization of multimodal federated learning, privacy-preserving techniques for multimodal federated learning, multimodal federated few-shot learning & multimodal federated semi-supervised learning, and data and knowledge-driven multimodal federated learning.