Frame-level Analysis Research Articles

Low-complexity lung ultrasound video scoring by means of intensity projection-based video compression

Since the outbreak of COVID-19, efforts have been made towards semi-quantitative analysis of lung ultrasound (LUS) data to assess the patient’s condition. Several methods have been proposed in this regard, with a focus on frame-level analysis, which was then used to assess the condition at the video and prognostic levels. However, no extensive work has been done to analyze lung conditions directly at the video level. This study proposes a novel method for video-level scoring based on compression of LUS video data into a single image and automatic classification to assess patient’s condition. The method utilizes maximum, mean, and minimum intensity projection-based compression of LUS video data over time. This enables to preserve hyper- and hypo-echoic data regions, while compressing the video down to a maximum of three images. The resulting images are then classified using a convolutional neural network (CNN). Finally, the worst predicted score given among the images is assigned to the corresponding video. The results show that this compression technique can achieve a promising agreement at the prognostic level (81.62%), while the video-level agreement remains comparable with the state-of-the-art (46.19%). Conclusively, the suggested method lays down the foundation for LUS video compression, shifting from frame-level to direct video-level analysis of LUS data.

Neuro-symbolic interpretable AI for automatic COVID-19 patient-stratification based on standardised lung ultrasound data

In the current pandemic, being able to efficiently stratify patients depending on their probability to develop a severe form of COVID-19 can improve the outcome of treatments and optimize the use of the available resources. To this end, recent studies proposed to use deep-networks to perform automatic stratification of COVID-19 patients based on lung ultrasound (LUS) data. In this work, we present a novel neuro-symbolic approach able to provide video-level predictions by aggregating results from frame-level analysis made by deep-networks. Specifically, a decision tree was trained, which provides direct access to the decision process and a high-level explainability. This approach was tested on 1808 LUS videos acquired from 100 patients diagnosed as COVID-19 positive by a RT-PCR swab test. Each video was scored by LUS experts according to a 4-level scoring system specifically developed for COVID-19. This information was utilised for both the training and testing of the algorithms. A five-folds cross-validation process was utilised to assess the performance of the presented approach and compare it with results achieved by deep-learning models alone. Results show that this novel approach achieves better performance (82% of mean prognostic agreement) than a threshold-based ensemble of deep-learning models (78% of mean prognostic agreement).

Anomalous entities detection and localization in pedestrian flows

Abstract We propose a novel Gaussian kernel based integration model (GKIM) for anomalous entities detection and localization in pedestrian flows. The GKIM integrates spatio-temporal features for efficient and robust motion representation to capture the distinctive and meaningful information about the anomalous entities. We next propose a block based detection framework by training a recurrent conditional random field (R-CRF) using the GKIM features. The trained R-CRF model is then used to detect and localize the anomalous entities during the online testing stage. We conduct comprehensive experiments on three benchmark datasets and compare the performance of the proposed method with the state-of-the-art anomalous entities detection methods. Our experiments show that the proposed GKIM outperforms the compared methods in terms of equal error rate (EER) and detection rate (DR) in both frame-level and pixel-level comparisons. The frame-level analysis detects the presence of an anomalous entity in a frame regardless of its location. The pixel-level analysis localizes the anomalous entity in term of its pixels.

Relationship between peri-strut low intensity areas and vascular healing response after everolimus-eluting bioresorbable scaffold implantation: An optical coherence tomography study

BackgroundPeri-strut low intensity areas (PLIA) surrounding metallic coronary stent struts on optical coherence tomography (OCT) images have been histologically related to delayed healing and inflammation, and have been associated with neointimal proliferation. The relationship between PLIA and vascular healing response after bioresorbable scaffold (BRS) implantation remains unclear. MethodsThis study includes 38 consecutive patients (50 scaffolds) evaluated using OCT 12 months after BRS implantation. Mean and percent neointimal area were quantified. A PLIA was defined as a peri-strut region with an homogenous lower intensity appearance than the surrounding tissue on OCT images without significant signal attenuation. Cross sections were scored as follows: score 0, no PLIA; score 1, <1 quadrant; score 2, ≥1 but <2 quadrants; score 3, ≥2 quadrants but <3 quadrants; and score 4, ≥3 quadrants. Scaffolds were divided into two groups (PLIA+ and PLIA−) based on the presence or absence of any PLIA in the scaffold segment. ResultsThe frequency of any PLIA within the scaffold segment was 70.0%. The median PLIA score per scaffold was 0.51 (interquartile range 0–1.07). Using both scaffold- and frame-level analysis, a significant positive correlation was observed between PLIA score and both mean and percent neointimal area. Mean and percent neointimal area were significantly higher in the PLIA+ group than in the PLIA− group (1.95±0.65mm2 vs. 1.51±0.27mm2, p<0.01 and 24.0±7.0% vs. 17.4±3.6%, p<0.01, respectively). ConclusionThe presence and extent of PLIA on OCT imaging after BRS implantation appears to be significantly associated with neointimal formation.

Detecting video frame rate up-conversion based on frame-level analysis of average texture variation

Frame rate up-conversion (FRUC) refers to frame interpolation between adjacent video frames to increase the motion continuity of low frame rate video, which can improve the visual quality on hand-held displays. However, FRUC can also be used for video forgery purposes such as splicing two videos with different frame-rates. We found that most FRUC approaches introduce visual artifacts into texture regions of interpolated frames. Based on this observation, a two-stage blind detection approach is proposed for video FRUC based on the frame-level analysis of average texture variation (ATV). First, the ATV value is computed for each frame to obtain an ATV curve of candidate video. Second, the ATV curve is further processed to highlight its periodic property, which indicates the existence of FRUC operation and further estimates the original frame rate. Thus, the positions of interpolated frames can be inferred as well. Extensive experimental results show that the proposed forensics approach is efficient and effective for the detection of existing typical FRUC approaches such as linear frame averaging and motion-compensated interpolation (MCI). The detection performance is superior to the existing approaches in terms of time efficiency and detection accuracy.

Modeling phonetic pattern variability in favor of the creation of robust emotion classifiers for real-life applications

The role of automatic emotion recognition from speech is growing continuously because of the accepted importance of reacting to the emotional state of the user in human–computer interaction. Most state-of-the-art emotion recognition methods are based on turn- and frame-level analysis independent from phonetic transcription. Here, we are interested in a phoneme-based classification of the level of arousal in acted and spontaneous emotions. To start, we show that our previously published classification technique which showed high-level results in the Interspeech 2009 Emotion Challenge cannot provide sufficiently good classification in cross-corpora evaluation (a condition close to real-life applications). To prove the robustness of our emotion classification techniques we use cross-corpora evaluation for a simplified two-class problem; namely high and low arousal emotions. We use emotion classes on a phoneme-level for classification. We build our speaker-independent emotion classifier with HMMs, using GMMs-based production probabilities and MFCC features. This classifier performs equally well when using a complete phoneme set, as it does in the case of a reduced set of indicative vowels (7 out of 39 phonemes in the German SAM-PA list). Afterwards we compare emotion classification performance of the technique used in the Emotion Challenge with phoneme-based classification within the same experimental setup. With phoneme-level emotion classes we increase cross-corpora classification performance by about 3.15% absolute (4.69% relative) for models trained on acted emotions (EMO-DB dataset) and evaluated on spontaneous emotions (VAM dataset); within vice versa experimental conditions (trained on VAM, tested on EMO-DB) we obtain 15.43% absolute (23.20% relative) improvement. We show that using phoneme-level emotion classes can improve classification performance even with comparably low speech recognition performance obtained with scant a priori knowledge about the language, implemented as a zero-gram for word-level modeling and a bi-gram for phoneme-level modeling. Finally we compare our results with the state-of-the-art cross-corpora evaluations on the VAM database. For training our models, we use an almost 15 times smaller training set, consisting of 456 utterances (210 low and 246 high arousal emotions) instead of 6820 utterances (4685 high and 2135 low arousal emotions). We are yet able to increase cross-corpora classification performance by about 2.25% absolute (3.22% relative) from UA=69.7% obtained by Zhang et al. to UA=71.95%.

Multimodal Physical Activity Recognition by Fusing Temporal and Cepstral Information

A physical activity (PA) recognition algorithm for a wearable wireless sensor network using both ambulatory electrocardiogram (ECG) and accelerometer signals is proposed. First, in the time domain, the cardiac activity mean and the motion artifact noise of the ECG signal are modeled by a Hermite polynomial expansion and principal component analysis, respectively. A set of time domain accelerometer features is also extracted. A support vector machine (SVM) is employed for supervised classification using these time domain features. Second, motivated by their potential for handling convolutional noise, cepstral features extracted from ECG and accelerometer signals based on a frame level analysis are modeled using Gaussian mixture models (GMMs). Third, to reduce the dimension of the tri-axial accelerometer cepstral features which are concatenated and fused at the feature level, heteroscedastic linear discriminant analysis is performed. Finally, to improve the overall recognition performance, fusion of the multimodal (ECG and accelerometer) and multidomain (time domain SVM and cepstral domain GMM) subsystems at the score level is performed. The classification accuracy ranges from 79.3% to 97.3% for various testing scenarios and outperforms the state-of-the-art single accelerometer based PA recognition system by over 24% relative error reduction on our nine-category PA database.