Multimodal Services Research Articles

Cross-Modal Stream Transmission: Architecture, Strategy, and Technology

Since the multi-modal services, which elaborately integrate audio, video, and haptic streams, are able to substantially improve the users' immersive experience, they are gradually becoming the mainstream applications. However, the existing solutions fail to simultaneously satisfy the diverse transmission requests of multi-modal services in terms of lower latency, higher reliability, and higher throughput, especially in the resource-constrained and dynamic environment. To circumvent this dilemma, this work constructs a cross-modal stream transmission architecture by exploring the intra-modal redundancy and the inter-modal correlation. Specifically, we propose an adaptive stream scheduling strategy, which can not only substantially decrease the stream traffics, but also dramatically reduce the impact of the channel uncertainty and the haptic signals' stochastic arrival on the delay jitter and reliability. Moreover, through prioritizing the resource provision of haptic signals, we design a joint uplink and downlink resource allocation scheme based on the prediction model. In particular, the transmitter sends the predicted haptic signals via the allocated resource blocks in advance according to the optimal window size, so as to offset the latency and remain the symmetry during the bilateral haptic transmission. Meanwhile, we develop an audio and video stream transmission scheme in a hierarchical and reusing structure by integrating the power-domain NOMA with LMDC-FEC, which fully utilizes the remaining limited resources to maximize the robustness of audio-video transmission.

How to Cache Important Contents for Multi-Modal Service in Dynamic Networks: A DRL-Based Caching Scheme

How to Cache Important Contents for Multi-Modal Service in Dynamic Networks: A DRL-Based Caching Scheme

A pentagonal type-2 fuzzy variable defuzzification model with application in humanitarian supply chains

Disasters can be caused by humans or natural hazards, and they impact the vulnerability of communities worldwide. This study considers disasters caused by humans, including the rescue operation taken post-accidents. The objective of the research is to minimise time and cost to save people’s lives in the early stages of rescue operations from the fuzzy logic perspective by proposing a novel pentagonal type-2 fuzzy variable (PT2FV) defuzzification method. The multi-objective multi-modal services at the crucial early hours of a disaster solve a solid transportation problem mathematically using a weighted sum method and neutrosophic compromise programming in Lingo solver software. The novelty of the research has been justified by the sensitivity analysis test on various levels of credibility at different intervals of PT2FV. The various credibility level intervals have also been discussed, and the best possible interval is suggested. The outcome suggests an effective plan for humanitarian relief in difficult areas using multiple modes of transit for delivery of reliefs, availing rescue team and saving of victims along with a notion of an efficient contingency planning.

Estimating Travel Demand with a Multimodal Transport Model Including E-Scooters in Iași, Romania

Cities, even medium-sized and small ones, have become overwhelmed by traffic and congestion. Innovative solutions are required and recent studies have focused on sustainable approaches. This study aims to develop a multimodal transport model consisting of an urban public transportation (UPT) service combined with an e-scooter sharing service. The main purpose of the model is a comparison between two travel alternatives (car and UPT + e-scooter), calculating their associated levels of utility. The methodology is based on a multinomial logit model, implemented in Matlab software, using results from an online mobility survey (socio-economic characteristics of potential users of e-scooters). The additional aim is related to the assessment of the inhabitants’ availability to shift from car to multimodal service or to simple e-scooter service. The developed micromodel was applied in Iași, a mid-sized city located in the northeast of Romania. Several price strategies were analysed in order to find their influence on users’ mobility behaviour. It turned out that the price for the shared e-scooter is too high at this moment to be used for daily travel. Without a price decrease and public authorities’ involvement in facilitating the multimodality, the e-scooter remains only an entertainment activity.

Dynamic Transmission Mode Selection for Multi-Modal Services

Multi-modal services, which integrate emerging haptic signals with audio-video signals, are expected to further improve user’s immersive experience. However, traditional audio-video communication schemes fail to simultaneously meet various transmission requests of low latency, high reliability, and high throughput. To resolve this problem, based on the existing communication paradigms, a dynamic multi-modal service transmission mode selection strategy is proposed. Firstly, according to different types of multi-modal services, three transmission strategies are proposed to satisfy different transmission requests. And a rule for mode selection and resource allocation is designed. Secondly, an improved multi-objective optimization algorithm based on NSGA3 is proposed to select modes and allocate resources, which improves communication performance as much as possible, and achieves better resource utilization efficiency. Finally, simulation results show the effectiveness of proposed method.

Perception-Aware Cross-Modal Signal Reconstruction: From Audio-Haptic to Visual

Cross-modal communications, devoting to collaboratively delivering and processing audio, visual, and haptic signals, have gradually become the supporting technology for the emerging multi-modal services. However, the inevitable resource competitions among different modality signals as well as the unexpected packet loss and latency during transmission seriously affect quality of the received signals and end user's immersive experience (especially visual experience). To overcome these dilemmas, this paper proposes a cross-modal signal reconstruction strategy from the perspective of human's perceptual facts. It tries to guarantee visual signal quality by considering potential correlations among modalities when processing audio and haptic signals. On the one hand, a time-frequency masking-based audio-haptic redundancy elimination mechanism is designed by resorting to the similarity of audio-haptic characteristics and human's masking effects. On the other hand, based on the fact that non-visual perception can assist to form and enhance visual perception, an audio-haptic fused visual signal restoration (AHFVR) approach for handling the impaired and delayed visual signals is proposed. Experiments on a standard multi-modal database and a constructed practical platform evaluate the performance of the proposed perception-aware cross-modal signal reconstruction strategy.

Mind-body medicine in integrative uro-oncology : Studies and areas of application

Mind-body medicine (MBM) complements somatically oriented medical practice with behavioral and lifestyle-oriented approaches: Thus, health-promoting attitudes and behaviors are strengthened in everyday life. In integrative oncology, it helps promote emotional and physical well-being. Guideline recommendations and the current study situation in integrative uro-oncology are presented. During and after completion of primary therapy, mindfulness-based stress reduction (MBSR) shows positive effects on anxiety, stress, and fatigue. However, it appears that the offer needs to be better tailored to the needs of prostate cancer patients. The effects of yoga are well documented, especially on fatigue, quality of life, and sexual function. Prostate cancer patients also showed asignificantly increased immune response after completing ayoga intervention. Tai Ji Quan/Qigong improve quality of life, fatigue, and other symptoms. Hypnosis-especially in the palliative setting-mitigates anxiety, and relaxation techniques alleviate sleep problems and nausea/vomiting. Multimodal services improve quality of life on numerous levels. Higher resilience correlates with better quality of life and stronger male self-esteem. MBSR alleviates many symptoms but needs to be adapted to the needs of prostate cancer patients; yoga and Tai Ji Chuan/Qigong alleviate fatigue and improve quality of life. Hypnosis and relaxation training reduce nausea/vomiting, and improve sleep and anxiety. Resilience promotion is important to support oncological patients.

Cross-Modal Semantic Communications

It is well known that multi-modal services, including video, audio, and haptic signals, aim to provide immersive experience with low latency and high reliability. Although multi-modal signals have differences in structure, transmission delay, and jitter, there are potential cross-modal relations among these multi-modal signals that can be explored to realize multi-modal signal processing and streaming. As a revolutionary communication paradigm, semantic communications can not only provide human-oriented services by delivering the intended meanings, but also break the modality barriers through compact semantic information extracted from multi-modal signals, which can help to achieve cross-modal applications. However, due to the polysemy and ambiguity issues, how to guarantee high reliability for semantic communications is a tremendous challenge. To overcome this dilemma, this article designs a cross-modal semantic communication paradigm that involves three modules: The cross-modal knowledge graph (CKG) provides essential background knowledge and signal patches for encoding and decoding respectively; the cross-modal semantic encoder infers potential implicit semantics to reduce encoding polysemy; and the cross-modal semantic decoder ensures consistency between source signals and recovered signals in terms of bit level and semantic level in order to reduce decoding ambiguity. They collaboratively improve the reliability of the communication system, which is verified by the simulation result.

Exploring the Benefits of Cross-Modal Coding

Multi-modal services, typically integrating such signals as audio, video, and haptic, will become an inevitable application trend of the 5G and beyond. However, due to the essential differences among the haptic and audio/video signals, the existing coding schemes usually fail to satisfy the critical requirements in terms of the rate distortion performance. Inspired by the phenomenon that hearing, sight and touch are highly correlated, we provide an affirmative answer by proposing the framework of cross-modal coding, which compresses multi-modal signals aided by their semantic correlation. In particular, the highlights of this work lie in addressing three fundamental technical problems: i) how to exploit the semantic correlation among different modalities, ii) to what extent of benefit we can get from cross-modal coding, and iii) how to design a general cross-modal codec. On the theoretical end, we determine the minimum number of bits required to compress haptic signals under the rate conditions of video streams through investigating their semantic correlation. On the technical end, we design a general cross-modal codec to approach the optimal compression limit by using the AI-enabled cross-modal prediction and channel coding. Numerical results demonstrate that the proposed cross-modal coding can achieve significant benefits relative to the existing schemes, especially when multi-modal signals have strong semantic correlation.

Planning of integrated mobility-on-demand and urban transit networks

We envision a multimodal transportation system where Mobility-on-Demand (MoD) service is used to serve the first mile and last mile of transit trips. For this purpose, the current research formulates an optimization model for designing an integrated MoD and urban transit system. The proposed model is a mixed-integer non-linear programming model that captures the strategic behavior of passengers in a multimodal network through a passenger assignment model. It determines which transit routes to operate, the frequency of the operating routes, the fleet size of vehicles required in each transportation analysis zone to serve the demand, and the passenger flow on both road and transit networks. A Benders decomposition approach with several enhancements is proposed to solve the given optimization program. Computational experiments are presented for the Sioux Falls multimodal network. The results show a significant improvement in the congestion in the city center with the introduction and optimization of an integrated transportation system. The proposed design allocates more vehicles to the outskirt zones in the network (to serve the first mile and last mile of transit trips) and more frequency to the transit routes in the city center. The integrated system significantly improves the share of transit passengers and their level of service in comparison to the base optimized transit system. The sensitivity analysis of the bus and vehicle fleet shows that increasing the number of buses has more impact on improving the level of service of passengers compared to increasing the number of MoD vehicles. Finally, we provide managerial insights for deploying such multimodal service.

Modeling an optimal supply chain management system in transport systems

Objective. The purpose of the work is to develop universal logistics approaches to modeling the optimal supply chain management system in the conditions of a multimodal service center.Method. The methodological basis of the study is theoretical and experimental research in the field of creating an effective model of supply chain management in transport systems.Result. A process approach to the strategic management of supply chains for the logistics of transport systems has been formalized, taking into account the influence of a group of factors; developed a methodology for managing relationships with suppliers, taking into account their integration in supply chain management; rational parameters for evaluating the effectiveness of the logistics management system for logistics under given working conditions were determined; a methodology has been developed for managing supply chains of logistics in the conditions of operation of a multimodal service center based on the application of the theory of experiment planning.Conclusion. Optimization of supply chain management (Supply Chain Management, SCM) allows you to effectively plan, form, control and regulate material, financial, information flows throughout the entire life cycle of goods. The resulting regression equations are recommended to be used as a mechanism for analyzing the work with the selected supplier when organizing the optimal management of supply and procurement processes in the transport system in the conditions of operation of the multimodal service center of GRAND MOTORS LLC.

Airport Access Mode Choice: Analysis of Passengers’ Behavior in European Countries

Transportation systems require many challenges in providing seamless door-to-door mobility. The main initiatives are encouraging a shift from private to other transport modes by providing a fully integrated multimodal service in which the coordination and data sharing among different stakeholders are required. The idea of this paper is to analyze the mode choice, as well as the variables that affect the travelers’ airport access mode choice. For that purpose, we used multinomial logistics (MNL) regression to determine probability of mode choice given various multimodal chain alternatives. The inputs of the proposed model were based on the answers from the participants of the online survey which was disseminated in Europe. Through more than 2000 answers to the survey, we collected the data related to the factors that influence the airport access mode choice, travelers’ attitude, motives for traveling, as well as the socio-demographics of participants. Afterwards, we investigated the influence of the main factors that have an impact on the non-coordination in the multimodal travel chain. The obtained results highlight the impact of the factors “reliability” and “waiting time” in making mode choice.

Cross-Modal Transmission Strategy

Multi-modal services, typically integrated by visual, audio, and haptic signals, have been considered as promising killer services in 5G and beyond 5G era. However, due to essential difference among these signals, how to guarantee quality of multi-modal services is a significant technical challenge. Existing transmission schemes, which deliver and process each modality signal separately, cannot meet such requirements as low latency, high reliability, and high throughput. To get over the dilemma, this paper proposes a general cross-modal transmission strategy by taking advantage of the potential correlation among modalities, which consists of a delivery mechanism at the sender and a signal restoration procedure at the receiver. On the one hand, a visual-aided haptic content compression method is designed for the delivery mechanism. By utilizing the category correlation among modalities, haptic signals with similar visual content can be effectively compressed, reducing transmission burden. On the other hand, a fine-grained haptic to image synthesis (FHIS) approach is proposed for realizing signal restoration. Through exploring strong matching properties among modalities, the FHIS can restore the impaired, missing, delayed visual images from the received haptic signals. Experiments on a standard visual-haptic database and a practical platform evaluate the performance of the constructed cross-modal transmission strategy.

Virtual Prehabilitation in Patients With Cancer Undergoing Surgery During the COVID-19 Pandemic: Protocol for a Prospective Feasibility Study.

BackgroundSince the beginning of the COVID-19 pandemic, preoperative care, also termed prehabilitation, has become increasingly relevant due to the decreasing functional and psychosocial health of patients with cancer, which is a result of the pandemic restrictions. Concurrently, access to telehealth has improved; telehealth comprises all remote care delivery facilitated by information technologies (ie, virtually).ObjectiveThe aim of this protocol is to describe the rationale and methodology for a major trial investigating the feasibility and safety of multimodal virtual prehabilitation services (ie, teleprehabilitation).MethodsThis single-arm feasibility trial aims to recruit 100 patients with cancer to receive teleprehabilitation throughout their preoperative period. The inclusion criteria are as follows: (1) 18 years of age or older, (2) scheduled for elective cancer surgery and referred by a surgeon, (3) medically cleared by the referring physician to engage in physical activity, and (4) have a good comprehension of the English or French language. Feasibility will be assessed by documenting recruitment, adherence, and retention rates, in addition to patients’ motives for not participating in the trial, low participation, or discontinuation. The secondary outcome of safety will be assessed by reporting program-related adverse events.ResultsThe Montreal General Hospital Foundation funded the project in August 2020. The protocol was then approved by the Research Ethics Board of the McGill University Health Centre in January 2021 (ID No. 2021-6730). The first patient was recruited in March 2021, and recruitment is expected to end in September 2022. As of March 2022, 36 patients have been recruited, including 24 who have completed their participation. No adverse events have been reported. Data collection is expected to conclude in November 2022. Data analysis will be performed, and the results will be published by the beginning of 2023.ConclusionsThis trial will provide guidance on the use of telehealth in the administration of prehabilitation services. The trial will provide a large amount of information that will respond to gaps in the literature, as there are minimal reports on the use of telehealth rehabilitation and prehabilitation services among elderly populations and in acute contexts, such as the preoperative period.Trial RegistrationClinicalTrials.gov NCT0479956; https://clinicaltrials.gov/ct2/show/NCT04799561International Registered Report Identifier (IRRID)DERR1-10.2196/29936

Measuring the Service Quality at Multimodal Public Transport Interchanges: A Needs-Driven Approach

Multimodal interchanges are key components of integrated public transport systems. A better understanding of the perceptions, needs, and priorities of public transport users as they travel through transport interchanges is needed to provide insight into the design of future integrated interchanges that will ultimately benefit users. This paper aims to help fill this gap by describing the development of a tool that can be used to measure the service quality of multimodal interchanges from a passenger’s viewpoint, taking into account a set of predefined criteria that can easily and cost effectively be measured on site. A mixed methods approach is discussed, and demonstrated for a case study at three public transport facilities in Durban, South Africa. The results show that users of different genders and age groups have different requirements of interchange facilities, and that knowledge of local preferences is important for effective management and design. The external validity of the multimodal service quality index is demonstrated as it ranks the three facilities in the same order of quality as independent satisfaction ratings obtained from users.

Evidence-Based Smart Transition Strategies for Long-Distance Commuters in Beijing

Building a sustainable and eco-friendly transport system is crucial to tackling global challenges such as climate change, as transport can be seen as one of the main sources of air pollution and CO2 emissions in megacities, particularly in developing countries. By bringing together multiple modes of travel, and combining different transport provider options into a single service, Mobility-as-a-Service (MaaS) could offer an effective way to help build a sustainable city by improving public transport services. However, the strategies used to develop MaaS, vary in different cities based on their specific multimodal transport facilities and service. Many residents of Beijing have to contend with long-distance commuting, which may adversely affect individuals’ travel experience and satisfaction and is, therefore, a key issue for transport development in Beijing. Using Beijing as a case study, we carried out in-depth interviews and thereby captured long-distance commuters’ concerns and needs concerning their commuting experiences. Our findings show that long-distance commuters are primarily concerned about the following multimodal commuting scenarios: “Underground + bicycle”, “Underground + taxi”, “Underground + private car”, and “Underground + bus”. Therefore, we suggest that the priority should be to develop a MaaS model for Beijing that focuses on the integration of multimodal transport connected to the underground rail system.

Strategic management of airports efficiency and safety in the process of air and multimodal logistics transportations development

The airport is a complex system that focuses on the intersection of the interests of different players in the transport market and different activities related to the performance of air transportation. The airport has an independent position in relation to partners and users of air transport. Therefore, the role of the airport, above all, is to unite the efforts of the air transport system and find a balance between their economic interests in order to maximize the efficient and safe use of air transport. In addition to effective cooperation with its partners, the airport must meet the requirements of international standards for aviation security, the appropriate state of infrastructure and efficiency of technological processes. States are also interested in developing, increasing the capacity of airports, expanding opportunities in the process of operation, as the airport contributes to the revival of ties between countries and the development of the global economy, tourism. Bringing the air transport infrastructure in line with international requirements is an important component of Ukraine's strategy aimed at ensuring the competitiveness of the national economy on the world market. Solving the problems of airport development is a nationwide problem that hinders the use of the high transit potential of the state. The complex and diverse challenges facing airports require rational, integrated and flexible solutions in business process management and active cooperation of air transport system participants, which is manifested in the coordination of strategic, tactical and operational decisions. The article is devoted to improving approaches to strategic management of airport efficiency and safety in the process of ensuring the implementation of international air and multimodal logistics services.

Unpacking the Role of Artificial Intelligence for a Multimodal Service System Design

Since requirements of service demands are becoming increasingly complex and diversified, one of the success factors of a multimodal service system is its capability to design a specific service instance satisfying a specific set of requirements. This capability is further highlighted in Ad Hoc Multimodal Service Systems (AHMSSs), where service instances rarely follow a standard form of service delivery and exist only for a limited time. However, due to the increasing scale and frequency of services in many business and public sectors, meeting the desired level of capability has become troublesome. A well-designed Artificial Intelligence (AI) approach can be a solution to the difficulty by addressing the underlying complexity and uncertainty of the AHMSS design process. To conceptualize and foster AI applications to an AHMSS, this study identifies key decision-making problems in the AHMSS design process and discusses the role of AI in the process. The results will form the basis for AI development and implementation for an AHMSS and relevant service systems.

Феномен мультимодального сервісного медіатексту комунікаційної платформи

<p><strong><em>The aim</em></strong><em> of the article is to investigate the issue of multimodal media text, which is now popular among authors and creators of content communication platforms.</em><em></em></p><p><strong><em>Research methodology</em></strong><em>. In the course of the research a descriptive method was used to outline modern current scientific approaches to the study of the issue, comparative method – to compare scientific views and derive the author’s definition.</em></p><p><strong><em>Results.</em></strong><em> The definition of modern researchers is given and the author’s own vision of the «multimodal media text» concept and its derivative – «multimodal service media text» is offered. Emphasis is placed on the need to identify and classify the term for the «Applied Social Communication Technologies» field, due to the spread of Internet communications and the growing popularity of communication platforms. The need to study multimodal service media text in view of its versatility, polythematic, diverse structure and other features inherent in the media texts of the network is noted. Attention is paid to different categories of mechanisms of perception of the visual component of the service media text. Multimodal service media text is presented as an applied unit of modern communication processes. Its informative and communicative properties are revealed. The ways of formation of communication modality of modern Internet platforms are outlined. The modern phenomenon of multimodality is described, which is formed taking into account two factors: communication processes and hypertext. The necessity of correct construction of the communication platforms’ content taking into account the multimedia nature of the text is substantiated. The mechanism of functioning of communication platforms capable of synthesizing different types of modal content (video, audio, photo, animation, textual) is studied.</em></p><p><strong><em>Scientific</em></strong><em> <strong>novelty.</strong> Communication multimodality is positioned in the article as a complex formation of symbols and code synthesis of different categories: linguistic and extralinguistic factors. All this gives grounds to study multimodal media text in social communications of the Internet sphere, where the text is now multimedia. It is offered to classify and structure the types of multimodal content according to the thematic orientation of the communication platform: culture, sports, politics, beauty industry, travel topics, etc.</em></p><p><strong><em>Practical meaning.</em></strong><em> The results of the study can be used in the educational process in the «Journalism» specialty.</em></p><p><strong><em>Key words:</em></strong><em> communication platform, multimodal media text, service media text, internet communications.</em></p><div><div><p> </p></div></div>

Haptic Signal Reconstruction for Cross-Modal Communications

The emerging multi-modal services, characterized as the integration of audio, visual, and haptic signals, will become the killer applications in 5 G and beyond 5 G era. In order to support multi-modal services, cross-modal communications come into being. However, when adopting cross-modal communications to haptic-dominant multi-modal services, there still face several technical challenges. On the one hand, haptic signals are very sensitive to interference and easy to be damaged or even missing during transmission. On the other hand, it needs to generate virtual haptic signals when real touch sensory information is hard to be gathered. To get over the dilemma, this paper proposes a haptic signal reconstruction strategy for cross-modal communications. First, a cloud-edge collaboration-based cross-modal communication architecture is constructed. Then, an audio-visual-aided haptic signal reconstruction (AVHR) approach under this architecture is designed by leveraging the potential correlation among modalities. It can be further divided into three components: feature extraction by cloud-edge transfer, shared semantic learning by multi-modal fusion, and haptic signal generation by semantic constraints. Finally, experiments on a standard audio-visual-haptic dataset and a practical cross-modal communication platform show that the proposed AVHR approach has better reconstruction performance when compared with the competing schemes.