Speech Summarization Research Articles

A Comparative Study for Speech Summarization Based on Machine Learning: A Survey

The most important aspect of human communication is speech. Lengthy media such as speech takes a long time to read and understand. This difficulty is solved by providing a reduced summary with semantics. Speech summarization can either convert speech to text using automated speech recognition (ASR) and then build the summary, or it can process the speech signal directly and generate the summary. This survey will look at a various of recent studies that have used machine and deep learning algorithms to summarize speech. it discusses the speech summarizing literatures in terms of time restrictions, research methodology, and lack of interest in particular databases for literature searches. As newer deep learning approaches were not included in earlier surveys, this is a new survey in this discipline where different approaches with various datasets were explored for speech summarization and evaluated using subjective or objective methods.

Report on the SIGDial 2021 special session on summarization of dialogues and multi-party meetings (SummDial)

The SummDial special session on summarization of dialogues and multi-party meetings was held virtually within the SIGDial 2021 conference on July 29, 2021. SummDial @ SIGDial 2021 aimed to bring together the speech, dialogue, and summarization communities to foster cross-pollination of ideas and fuel the discussions/collaborations to attempt this crucial and timely problem. When the pandemic has restricted most of our in-person interactions, the current scenario has forced people to go virtual, resulting in an information overload from frequent dialogues and meetings in the virtual environment. Summarization could help reduce the cognitive burden on the participants; however, multi-party speech summarization comes with its own set of challenges. The SummDial special session aimed to leverage the community intelligence to find effective solutions while also brainstorming the future of AI interventions in meetings and dialogues. We report the findings of the special session in this article. We organized the SummDial special session under the aegis of the EU-funded H2020 European Live Translator (ELITR) project. 1 Date : 29 July, 2021. Website : https://elitr.github.io/automatic-minuting/summdial.html.

Symbolic and Statistical Learning Approaches to Speech Summarization: A Scoping Review

Speech summarization techniques take human speech as input and then output an abridged version as text or speech. Speech summarization has applications in many domains from information technology to health care, for example improving speech archives or reducing clinical documentation burden. This scoping review maps close to 2 decades of speech summarization literature, spanning from the early machine learning works up to ensemble models, with no restrictions on the language summarized, research method, or paper type. We reviewed a total of 110 papers out of a set of 188 found through a literature search and extracted speech features used, methods, scope, and training corpora. Most studies employ one of four speech summarization architectures: (1) Sentence extraction and compaction; (2) Feature extraction and classification or rank-based sentence selection; (3) Sentence compression and compression summarization; and (4) Language modelling. We also discuss the strengths and weaknesses of these different methods and speech features. Overall, supervised methods (e.g. Hidden Markov support vector machines, Ranking support vector machines, Conditional random fields) performed better than unsupervised methods. As supervised methods require manually annotated training data which can be costly, there was more interest in unsupervised methods. Recent research into unsupervised methods focusses on extending language modelling, for example by combining Uni-gram modelling with deep neural networks. This review does not include recent work in deep learning.

Voice Controlled Mini CNC Plotter

The (CNC) stands for Computer Numerical Control.It is an advanced technology that has automated many manufacturing applications concerning parts machining as well as other applications. This work presents the implementation of a low-cost desktop size 3-axis CNC machine. It has been built from free software and cheap components. It is able for light duty cutting, painting, carving, and drawing. The control system of the machine consists of a PC and a microcontroller running Grbl software, which is an open source free program. The PC has the task of loading files written in a special programming language for CNCs, which is called G-code. It sends the lines of this file to the microcontroller, which parses the code, interpolates the motion commands, and drives the stepper motors of the machine. The system is built and tested experimentally to prove its operation, robustness, and accuracy. Also this paper presents techniques for speech-to-text and speech-to-speech automatic summarization based on speech unit extraction and concatenation. For the latter case, sentences, words, and between-filler units are investigated as units to be extracted from original speech. These methods are applied to the summarization of unrestricted-domain spontaneous presentations and evaluated by objective and subjective measures. It was confirmed that proposed methods are effective in spontaneous speech summarization.

Enhanced Language Modeling with Proximity and Sentence Relatedness Information for Extractive Broadcast News Summarization

The primary task of extractive summarization is to automatically select a set of representative sentences from a text or spoken document that can concisely express the most important theme of the original document. Recently, language modeling (LM) has been proven to be a promising modeling framework for performing this task in an unsupervised manner. However, there still remain three fundamental challenges facing the existing LM-based methods, which we set out to tackle in this article. The first one is how to construct a more accurate sentence model in this framework without resorting to external sources of information. The second is how to take into account sentence-level structural relationships, in addition to word-level information within a document, for important sentence selection. The last one is how to exploit the proximity cues inherent in sentences to obtain a more accurate estimation of respective sentence models. Specifically, for the first and second challenges, we explore a novel, principled approach that generates overlapped clusters to extract sentence relatedness information from the document to be summarized, which can be used not only to enhance the estimation of various sentence models but also to render sentence-level structural relationships within the document, leading to better summarization effectiveness. For the third challenge, we investigate several formulations of proximity cues for use in sentence modeling involved in the LM-based summarization framework, free of the strict bag-of-words assumption. Furthermore, we also present various ensemble methods that seamlessly integrate proximity and sentence relatedness information into sentence modeling. Extensive experiments conducted on a Mandarin broadcast news summarization task show that such integration of proximity and sentence relatedness information is indeed beneficial for speech summarization. Our proposed summarization methods can significantly boost the performance of an LM-based strong baseline (e.g., with a maximum ROUGE-2 improvement of 26.7% relative) and also outperform several state-of-the-art unsupervised methods compared in the article.

Abstractive Summarization with NLP Makes Easier to Memorize

Summarization is used to extract most relevant content from a huge content. It can be extractive and abstractive. But different people may have different scope of relevance in different area. Content can be in the form of opinions, news, judgments and ideas etc. Extractive Summarization extracts most focusing content without any change in the original content. Abstractive summarization is a knowledgebase extraction with some modification in original content. In this paper first we discuss various algorithm for abstractive summarization then on the basis of merits of various algorithm discussed in conclusions, we propose a new algorithm for abstractive summarization with the help of NLP to generate shortcuts from the content that make easier to memorize the content. Challenges in Abstractive summarization:  Repetition or duplicity of the content  Arrangement of words  Different people have different meaning for the same content  Number of times a content repeated  Different people have different interest  Nosiness in content  Information diversity Methods for Abstractive summarization: 1. integer linear programming-based summarization framework [3]. 2. Improved Semantic Graph Approach [4]. 3. novel speech act-guided summarization approach [5]. 4. Neural Abstractive Summarization with Diverse Decoding(NASDD) [6]. 5. Supervised and unsupervised approach [7]. 6. Neural network based representation [8]. 7. Integer linear optimization [9]. 8. Maximum L∞-norm and minimum entropy regularization [10]. 9. Opizer-E and Opizer-A [11] 10. LSTM-CNN based ATSDL model [12]. 11. novel concept-level approach [13].

Application of Different Algorithms to Optimize Abstractive Summarization

Summarization is used to extract most relevant content from a huge content. It can be extractive and abstractive. But different people may have different scope of relevance in different area. Content can be in the form of opinions, news, judgments and ideas etc. Extractive Summarization extracts most focusing content without any change in the original content. Abstractive summarization is a knowledgebase extraction with some modification in original content. In this paper first we discuss various algorithms for abstractive summarization then on the basis of merits of various algorithms we discuss, how various algorithm may help to optimize Abstractive Summarization. Challenges in Abstractive summarization:  Repetition or duplicity of the content  Arrangement of words  Different people have different meaning for the same content  Number of times a content repeated  Different people have different interest  Nosiness in content  Information diversity Methods for Abstractive summarization: 1. integer linear programming-based summarization framework [3] 2. Improved Semantic Graph Approach [4] 3. novel speech act-guided summarization approach [5] 4. Neural Abstractive Summarization with Diverse Decoding(NASDD) [6] 5. Supervised and unsupervised approach [7] 6. Neural network based representation [8] 7. Integer linear optimization [9] 8. Maximum L∞-norm and minimum entropy regularization [10] 9. Opizer-E and Opizer-A [11] 10. LSTM-CNN based ATSDL model [12] 11. novel concept-level approach [13]

On the Effects of Automatic Transcription and Segmentation Errors in Hungarian Spoken Language Processing

Emerging Artificial Intelligence (AI) technology has brought machines to reach an equal or even superior level compared to human capabilities in several fields; nevertheless, among many other fields, making a computer able to understand human language still remains a challenge. When dealing with speech understanding, Automatic Speech Recognition (ASR) is used to generate transcripts, which are processed with text-based tools targeting Spoken Language Understanding (SLU). Depending on the ASR quality (which further depends on speech quality, the complexity of the topic, environment etc.), transcripts contain errors, which propagate further into the processing pipeline. Subjective tests show on the other hand, that humans understand quite well ASR-closed captions, despite the word and punctuation errors. Through word embedding based semantic parsing, the present paper is interested in quantifying the semantic bias introduced by ASR error propagation. As a special use case, speech summarization is also evaluated with regard to ASR error propagation. We show, that despite the higher word error rates seen with the highly inflectional Hungarian, the semantic space suffers least impact than the difference in Word Error Rate would suggest.

A comparison of machine learning techniques for Persian Extractive Speech to Speech Summarization without Transcript

A comparison of machine learning techniques for Persian Extractive Speech to Speech Summarization without Transcript

Using Generic Summarization to Improve Music Information Retrieval Tasks

In order to satisfy processing time constraints, many MIR tasks process only a segment of the whole music signal. This practice may lead to decreasing performance, since the most important information for the tasks may not be in those processed segments. In this paper, we leverage generic summarization algorithms, previously applied to text and speech summarization, to summarize items in music datasets. These algorithms build summaries, that are both concise and diverse, by selecting appropriate segments from the input signal which makes them good candidates to summarize music as well. We evaluate the summarization process on binary and multiclass music genre classification tasks, by comparing the performance obtained using summarized datasets against the performances obtained using continuous segments (which is the traditional method used for addressing the previously mentioned time constraints) and full songs of the same original dataset. We show that GRASSHOPPER, LexRank, LSA, MMR, and a Support Sets-based Centrality model improve classification performance when compared to selected 30-second baselines. We also show that summarized datasets lead to a classification performance whose difference is not statistically significant from using full songs. Furthermore, we make an argument stating the advantages of sharing summarized datasets for future MIR research.

Exploring the use of unsupervised query modeling techniques for speech recognition and summarization

Statistical language modeling (LM) that intends to quantify the acceptability of a given piece of text has long been an interesting yet challenging research area. In particular, language modeling for information retrieval (IR) has enjoyed remarkable empirical success; one emerging stream of the LM approach for IR is to employ the pseudo-relevance feedback process to enhance the representation of an input query so as to improve retrieval effectiveness. This paper presents a continuation of such a general line of research and the major contributions are three-fold. First, we propose a principled framework which can unify the relationships among several widely-cited query modeling formulations. Second, on top of this successfully developed framework, two extensions have been proposed. On one hand, we present an extended query modeling formulation by incorporating critical query-specific information cues to guide the model estimation. On the other hand, a word-based relevance modeling has also been leveraged to overcome the obstacle of time-consuming model estimation when the framework is being utilized for practical applications. In addition, we further adopt and formalize such a framework to the speech recognition and summarization tasks. A series of experiments reveal the empirical potential of such an LM framework and the performance merits of the deduced models on these two tasks.

Extractive Broadcast News Summarization Leveraging Recurrent Neural Network Language Modeling Techniques

Extractive text or speech summarization manages to select a set of salient sentences from an original document and concatenate them to form a summary, enabling users to better browse through and understand the content of the document. A recent stream of research on extractive summarization is to employ the language modeling (LM) approach for important sentence selection, which has proven to be effective for performing speech summarization in an unsupervised fashion. However, one of the major challenges facing the LM approach is how to formulate the sentence models and accurately estimate their parameters for each sentence in the document to be summarized. In view of this, our work in this paper explores a novel use of recurrent neural network language modeling (RNNLM) framework for extractive broadcast news summarization. On top of such a framework, the deduced sentence models are able to render not only word usage cues but also long-span structural information of word co-occurrence relationships within broadcast news documents, getting around the need for the strict bag-of-words assumption. Furthermore, different model complexities and combinations are extensively analyzed and compared. Experimental results demonstrate the performance merits of our summarization methods when compared to several well-studied state-of-the-art unsupervised methods.

Combining Relevance Language Modeling and Clarity Measure for Extractive Speech Summarization

Extractive speech summarization, which purports to select an indicative set of sentences from a spoken document so as to succinctly represent the most important aspects of the document, has garnered much research over the years. In this paper, we cast extractive speech summarization as an ad-hoc information retrieval (IR) problem and investigate various language modeling (LM) methods for important sentence selection. The main contributions of this paper are four-fold. First, we explore a novel sentence modeling paradigm built on top of the notion of relevance, where the relationship between a candidate summary sentence and a spoken document to be summarized is discovered through different granularities of context for relevance modeling. Second, not only lexical but also topical cues inherent in the spoken document are exploited for sentence modeling. Third, we propose a novel clarity measure for use in important sentence selection, which can help quantify the thematic specificity of each individual sentence that is deemed to be a crucial indicator orthogonal to the relevance measure provided by the LM-based methods. Fourth, in an attempt to lessen summarization performance degradation caused by imperfect speech recognition, we investigate making use of different levels of index features for LM-based sentence modeling, including words, subword-level units, and their combination. Experiments on broadcast news summarization seem to demonstrate the performance merits of our methods when compared to several existing well-developed and/or state-of-the-art methods.

A Novel Real-Time Speech Summarizer System for the Learning of Sustainability

As the number of speech and video documents increases on the Internet and portable devices proliferate, speech summarization becomes increasingly essential. Relevant research in this domain has typically focused on broadcasts and news; however, the automatic summarization methods used in the past may not apply to other speech domains (e.g., speech in lectures). Therefore, this study explores the lecture speech domain. The features used in previous research were analyzed and suitable features were selected following experimentation; subsequently, a three-phase real-time speech summarizer for the learning of sustainability (RTSSLS) was proposed. Phase One involved selecting independent features (e.g., centrality, resemblance to the title, sentence length, term frequency, and thematic words) and calculating the independent feature scores; Phase Two involved calculating the dependent features, such as the position compared with the independent feature scores; and Phase Three involved comparing these feature scores to obtain weighted averages of the function-scores, determine the highest-scoring sentence, and provide a summary. In practical results, the accuracies of macro-average and micro-average for the RTSSLS were 70% and 73%, respectively. Therefore, using a RTSSLS can enable users to acquire key speech information for the learning of sustainability.

On the Application of Generic Summarization Algorithms to Music

Several generic summarization algorithms were developed in the past and successfully applied in fields such as text and speech summarization. In this paper, we review and apply these algorithms to music. To evaluate this summarization's performance, we adopt an extrinsic approach: we compare a Fado Genre Classifier's performance using truncated contiguous clips against the summaries extracted with those algorithms on 2 different datasets. We show that Maximal Marginal Relevance (MMR), LexRank and Latent Semantic Analysis (LSA) all improve classification performance in both datasets used for testing.

Blending Sentence Optimization Weights of Unsupervised Approaches for Extractive Speech Summarization

Abstract This paper evaluates the performance of two unsupervised approaches, Maximum Marginal Relevance (MMR) and concept-based global optimization framework for speech summarization. Automatic summarization is very useful techniques that can help the users browse a large amount of data. This study focuses on automatic extractive summarization on multi-dialogue speech corpus. We propose improved methods by blending each unsupervised approach at sentence level. Sentence level information is leveraged to improve the linguistic quality of selected summaries. First, these scores are used to filter sentences for concept extraction and concept weight computation. Second, we pre-select a subset of candidate summary sentences according to their sentence weights. Last, we extend the optimization function to a joint optimization of concept and sentence weights to cover both important concepts and sentences. Our experimental results show that these methods can improve the system performance comparing to the concept-based optimization baseline for both human transcripts and ASR output. The best scores are achieved by combining all three approaches, which are significantly better than the baseline system.

Generic speech summarization of transcribed lecture videos: Using tags and their semantic relations

We propose a tag‐based framework that simulates human abstractors' ability to select significant sentences based on key concepts in a sentence as well as the semantic relations between key concepts to create generic summaries of transcribed lecture videos. The proposed extractive summarization method uses tags (viewer‐ and author‐assigned terms) as key concepts. Our method employs Flickr tag clusters and WordNet synonyms to expand tags and detect the semantic relations between tags. This method helps select sentences that have a greater number of semantically related key concepts. To investigate the effectiveness and uniqueness of the proposed method, we compare it with an existing technique, latent semantic analysis (LSA), using intrinsic and extrinsic evaluations. The results of intrinsic evaluation show that the tag‐based method is as or more effective than the LSA method. We also observe that in the extrinsic evaluation, the grand mean accuracy score of the tag‐based method is higher than that of the LSA method, with a statistically significant difference. Elaborating on our results, we discuss the theoretical and practical implications of our findings for speech video summarization and retrieval.

A Semi-automatic Approach for Annotating Extractive Reference Summaries of Spontaneous Speech

We propose a semi-automatic approach to generate gold standard reference summaries for spontaneous speech by using Relaxed Dynamic Time Warping (RDTW) alignment between spontaneous speech and its accompanied rhetorical units. Most state-of-the-art speech summarization systems are trained by a large amount of document and human reference summaries. However, there is no clear guideline for compiling a gold standard reference summary. We show that, by following the rhetorical structure of speech, our semi-automatic approach can produce “gold standard” reference summaries with very high inter-labeler agreement.

Towards Abstractive Speech Summarization: Exploring Unsupervised and Supervised Approaches for Spoken Utterance Compression

Most previous studies on speech summarization focus on the extractive approaches. Yet directly concatenating the extracted speech utterances may not form a good summary due to the presence of disfluencies and redundancy in the unplanned spontaneous speech. In this paper, we proposed to generate compressed speech summaries by coupling the sentence level compression and summarization approaches, as a viable step towards generating abstractive summaries. We compared two utterance compression approaches: an unsupervised approach based on the Integer Linear Programming (ILP) framework, and a supervised method using conditional random fileds (CRF) that formulates the utterance compression problem as a sequence labeling task. We evaluated the compression performance using both human and ASR transcripts from the ICSI meeting corpus, and performed both automatic and human evaluation. Our results show that we can achieve reasonable utterance compression performance, and that the CRF-based method generally performs better. By coupling the compression and summarization approaches, we generated compressed speech summaries that cover more important information within the given length limit, yielding 5% absolute performance gain on both human and ASR transcripts as evaluated by the ROUGE-1 F-scores.

Comparing the use of semantic relations between tags versus latent semantic analysis for generic speech summarization

We proposed an advanced tag method in which original tags are expanded and the semantic relations between original or expanded tags are used to extract key sentences from lecture speech transcripts. Then, to evaluate our proposed method, we compared it with two other methods, a tag method of using only original tags and a latent semantic analysis (LSA) method. As a result, the F measure mean (0.31) of the advanced tag method is higher than that (0.29) of LSA method and it is also higher than that (0.27) of the tag method, although there is no significant difference between them respectively.