Multimodal Semantics Research Articles

"That thing in New York": Impaired naming vs. preserved recognition of unique entities following an anterior temporal lobe lesion

"That thing in New York": Impaired naming vs. preserved recognition of unique entities following an anterior temporal lobe lesion

Toward a pluri-component, multimodal, and dynamic organization of the ventral semantic stream in humans: lessons from stimulation mapping in awake patients

OPINION article Front. Syst. Neurosci., 26 August 2013 https://doi.org/10.3389/fnsys.2013.00044

Multimodal Semantics Extraction from User-Generated Videos

User-generated video content has grown tremendously fast to the point of outpacing professional content creation. In this work we develop methods that analyze contextual information of multiple user-generated videos in order to obtain semantic information about public happenings (e.g., sport and live music events) being recorded in these videos. One of the key contributions of this work is a joint utilization of different data modalities, including such captured by auxiliary sensors during the video recording performed by each user. In particular, we analyze GPS data, magnetometer data, accelerometer data, video- and audio-content data. We use these data modalities to infer information about the event being recorded, in terms of layout (e.g., stadium), genre, indoor versus outdoor scene, and the main area of interest of the event. Furthermore we propose a method that automatically identifies the optimal set of cameras to be used in a multicamera video production. Finally, we detect the camera users which fall within the field of view of other cameras recording at the same public happening. We show that the proposed multimodal analysis methods perform well on various recordings obtained in real sport events and live music performances.

Modality-Specific Operations in Semantic Dementia

A patient suffering from semantic dementia is described who consistently demonstrated the preserved ability to support specific types of semantic judgements from visual, but not from verbal, input. In addition the representations accessed from visual input were found to trigger complex behavioural schemata, while with verbal materials the patient performed almost invariably at chance level. A preliminary description is given of the nature of visual semantic representations, and the privileged relationship between this modality of input and some aspects of semantic knowledge is also explored. The richness of the semantic representations accessed from visual input can be accommodated within the “Multimodal Semantics” framework; alternative views, derived from the Identification Semantics and the Organized Unitary Content Hypothesis, are also discussed.

Prior and rennie on times and tenses

One of Arthur Prior’s constructions of the relational calculus for times within tense logic plus propositional quantifiers is considered using Malcolm Rennie’s multimodal semantics and found wanting in two respects: Prior’s use of an independent necessity operator was both incorrect and unnecessary, and his system imported an objectionable metaphysics namely temporal relationism.When these defects are repaired it is shown that temporal relationism is not entailed by the modified system.Finally, there is considered the possibility of describing an inconsistent temporal structure in which distinct, classically incompatible conceptions of time can be unified within an inconsistent framework