Farsi Text Research Articles

Persian Automatic Text Summarization Based on Named Entity Recognition

In this paper, we propose an unsupervised method for summarizing Farsi texts based on our neural named entity recognition (NER) system. This method consists of three phases: training a supervised NER model, recognizing named entities of the text, and generating a summary. The proposed method is an unsupervised extractive single-document summarization method. Although the proposed method is language independent, we focus on Farsi text summarization in this work. Firstly, we produce a word embedding based on Hamshahri2 corpus. Secondly, we train a neural network on Arman NER corpus. Then, the proposed algorithm ranks the sentences of the text based on the named entities in each sentence and produces the summary. Finally, the proposed method is evaluated on Pasokh single-document data set using the ROUGE evaluation measure. Without using any handcrafted features, our proposed method achieves state-of-the-art results. We compared our unsupervised method with the best supervised Farsi methods, and we achieved an overall improvement of ROUGE-2 recall score of 10.2%.

Corpus based coreference resolution for Farsi text

Corpus based coreference resolution for Farsi text

Farsi Font Recognition Based On The Fonts Of Text Samples Extracted By Som

A Farsi font recognition algorithm based on the fonts of some frequent text samples is proposed. Some features are extracted from the connected components of a text image. The feature vectors are clustered by using a Self-Organizing Map (SOM) clustering method. The clusters with more members determine the most frequent connected components (MFCCs). A number of members of these big clusters are extracted from the input image. This procedure is applied to both training and test images. Since the frequent samples in different Farsi texts are very similar, it can be guaranteed that a large number of samples of the detected MFCCs for a test image surely are in the extracted training samples set. The font type and font style of the extracted test samples are recognized by matching between them and the training samples. The most frequent recognized font of the extracted samples is considered as the font of the input text. To achieve a more accurate algorithm with lower complexity, the font size is determined in the second phase after the phase of the font type and style recognition. Using a lexicon reduction procedure reduces the complexities and processing time. The font size estimation is carried out based on the size of a particular MFCC in a text image. Experiments show that the proposed method outperforms other font recognition methods.

A New Experience in Persian Text Clustering using FarsNet Ontology

Clustering through organizing large text corpora has a key role in an easy navigation and browsing of massive amounts of text data and in particular in search engines. The documents comparison using the conventional clustering techniques is based on the surface similarities of words or extracted morphemes. This leads to non-semantic clusters usually. In this paper, Farsi, also known as Persian, has been taken into account with regards to the fact that the amount of electronic Farsi texts are growing rapidly. The documents are enriched by using semantic relationships-synonymy, hypernymy and hyponymy- extracted from FarsNet lexical ontology. A WSD procedure is proposed to decrease uncertainty. After preprocessing routines, three clustering algorithms including Bisecting K-means, LSI and PLSI based clustering is applied on the pre-categorized Persian Hamshahri corpus. Experimental results show the improvement of clustering quality when text data is enriched by the semantic relations especially using PLSI based approach.

The comparative impact of editing texts translated into Farsi with and without the original English texts

This research sought to determine whether there was a significant difference in the quality of the Farsi texts edited by professional editors with and without referring to their original English texts. To this end, copies of one text translated from English into Farsi were given to 40 professional editors, half of whom received the original English text as well. The editors edited the text as if this was a real-life professional assignment and their texts were subsequently assessed by two raters who were both well-experienced professors of Farsi. To respond to the research question, the means of the scores given by each rater to the two groups, that is, those receiving the English text and those who did not, were calculated for an independent samples t-test to be run. The conclusion was that there was no significant difference in the editing quality of the two groups. In other words, receiving the original English text was not a determining factor on the quality of the editors' work. Furthermore, a questionnaire focusing on both certain professional aspects of the editors' work in general and their working relationship with translators was administered to all 40 editors, the results of which are discussed in the paper.

A Hybrid Approach to Localize Farsi Text in Natural Scene Images

Text in scene images can provide useful and vital information for content-based image analysis. Therefore, localization of text in images is an important task. In this paper, we present a hybrid approach to localize Farsi text in natural scene images. Complex background, variations of text font, size and line orientation and non-uniform illumination are the problems of this method. The Language of text localization in the past works is almost limited to English or Chinese. In this paper we consider Farsi/Arabic language for text localization. Due to the specific features of this language challenges of text localization are numerous. In this paper, in the first step a new color based method is proposed for extracting candidate regions, then the texts in natural scene images are detected by combining edge and color features. Variation due to text size and orientation, are resolved by a new pyramid of images. The candidate texts are verified by combination of two features, wavelet histogram and histogram of oriented gradient. Experimental results using our large dataset have demonstrated that the proposed method is effective and promising.

Designing a Farsi text for the assessment of adult voice features and determining its validity and reliability in measuring the fundamental frequency and intensity of speech

Introduction: The every possible attempt in voice disorders assessment is to gather samples which represent individual s’ actual voice in the daily speech. The researchers have found that voice characteristics in reading a passage are closer to daily speech. Making texts has been started since early 20th century. The lack o f suitable text in Farsi language makes it difficult to conduct requisite researches and present documentary results of interventions. Researchers believe that Sara lip reading test which had been used for voice studies in Farsi language is not appropriate. The goal of this study is to make an appropriate text for assessing the voice characteristics which is also effective for both research and clinical goals. Materials and Methods: The required text was made by the study of the characteristics of used texts in other languages, considering the Dr. Speech software limitations in analyzing duration of stimuli and the results from investigations used Sara lip reading test. The content validity of this text was assessed. Its reliability was determined by test-retest method. Results: The content validity of the text was confirmed by the university professors. In examining the reliability, it was indicated that measuring the studied characteristics has a high reliability. Conclusion: There was a general consensus among university professors about the text validity. In examining the reliability, Cronbach′s alpha was more than 0.7 in studied variables. Considering the confirmed validity and reliability, it can be concluded that this text can be used for assessing voice characteristics in voice studies in normal samples

Farsi lexical analysis and stop word list

PurposeThe purpose of this article is to present an aggregated methodology for construction of the stop word list in Farsi language and generate a generic Farsi stop word list.Design/methodology/approachThe stop word list is extracted based on: syntactic classes, domain dependent, corpus statistic and expert judgments. Some of the main challenges that arise in the Farsi automatic text processing are outlined as well.FindingsResults from the techniques are aggregated and a general Farsi stop word list containing 927 words is generated.Practical implicationsThe created stop word list can affect the efficiency and effectiveness of retrieval and indexing process in Farsi information retrieval system, moreover, it can play an important role during Farsi text segmentation.Originality/valueOur stop word extraction algorithm is a promising technique; it could be applied into other languages that they have ambiguities in automatic text segmentation.

A blackboard approach towards integrated Farsi OCR system

An integrated OCR system for Farsi text is proposed. The system uses information from several knowledge sources (KSs) and manages them in a blackboard approach. Some KSs like classifiers are acquired a priori through an offline training process while others like statistical features are extracted online while recognizing. An arbiter controls the interactions between the solution blackboard and KSs. The system has been tested on 20 real-life scanned documents with ten popular Farsi fonts and a recognition rate of 97.05% in word level and 99.03% in character level has been achieved.

An efficient hybrid solution for pronouncing Farsi text

This paper presents a hybrid Artificial Neural Network (ANN)/rule-based phoneme synthesizer for Farsi text. A new variable called Vowel State (VS) is also introduced which efficiently decreases the size of computations. The proposed system uses VS along with the written form of the words to determine their pronunciation. Some rules are also applied to the system to decrease network interconnections. These rules are performed in the Normalization Unit and the Phoneme Extraction Unit before and after the ANN. A multilayer perceptron with 48, 90 and 3 neurons in the input layer, hidden layer and output layer respectively is chosen to determine the Vowel State of each letter. Each letter in a word enters the ANN in the heart of a window in order to recognize the VS of the letter. An asymmetric 8-letter windowing is suggested; 3 letters before the letter along with 4 letters after it enter the ANN as the input. For training and testing the system, 2359 words are extracted from some online available resources. The results of test show that using 65% of the corpus as training set and testing over the entire corpus gives the result of 84% correct answers for the ANN, and 90% correct answer for the entire system considering additional rules. The performance index of the ANN increases to 88, 92, 98 and 99 percent when the percentage of the training set increases to 75, 85, 95 and 100, respectively. The system also shows the performance of 93% correct answers on an independent word database. The proposed hybrid ANN/rule-based method imposes small size of calculation in comparison with similar systems. In this system, the output of ANN is a 5-value variable Vowel State instead of numerous phonemes. This fact decreases the neurons in the output layer of ANN considerably. Besides, adding some rules before and after the network helps to increase performance. The overall system shows a decrement of 70% in the size of computations comparing with other reported approaches.

Authorship attribution and statistical text analysis

In the study of ancient literature, a major problem is to deal with uncertain authorship. Ambiguity about authorship is not limited to the works from remote era. Different reasons cause uncertainty in authorship, such as reproduction of books by hand, prestige, having good sells for works with forged reputable names on them, and sometimes social or political pressures. Whatever the reason, authorship case studies offer the statistician an interesting opportunity to deal with various applied problems, where many standard statistical techniques have been introduced. In statistical analysis of literary texts one tries to apply an objective methodology to works that have received impressionistic treatment for a long time. In subjective analysis of literary style, experts use literary style of the text, which is not quantifiable, as an important criterion in their judgments. Subjective approach can rarely lead to a unique solution acceptable to all the scholars. Statistical quantitative methods provide objective components for judgments. In the quantitative approach, by carefully analyzing the style of the text one tries to find out how to characterize the style of an author numerically and determine sets of features (variables) in a text that most accurately describe the author's style. Much work has been done covering different aspects of this field. Different variables are proposed as distinguishing characteristics of writers, a wide range of mathematical methods is employed, and there is still a lot to be done in the future. The paper presents a brief history and a review of the statistical analysis of literary style, looks at several variables that have been used as stylistic criteria of authors, as well as the methods used. This is followed by some illustrations on Farsi text, implying that there are some general rules that hold for different languages.

A new segmentation technique for omnifont Farsi text

A new segmentation algorithm based on the conditional labeling of the upper contour is presented. A pre-processing technique is proposed that adjusts the local base line for each subword. The algorithm was tested on a data set of printed Farsi texts in 20 fonts. 98.5% of the connected characters were correctly segmented.

Automatic recognition of printed Farsi texts

The automatic recognition of printed Farsi (Persian) texts is complicated by several properties of the Farsi script: (a) connectivity of symbols, (b) similarity of groups of symbols, (c) highly variable widths, (d) subword overlap, and (e) line overlap. In this paper, a technique for the automatic recognition of printed Farsi texts is presented and its steps are discussed as follows: (1) digitization, (2) editing, (3) line separation, (4) subword separation, (5) symbol separation, (6) recognition, and (7) postprocessing. The most notable contributions of this work are in algorithms for steps (5) and (6) above. Practical application of the technique to Farsi newspaper headlines has been 100% successful. However, smaller type fonts, which could not be handled by the coarse digitization hardware used, will no doubt result in less than perfect recognition. The technique is also applicable with little or no modification to printed Arabic and Urdu texts which use the same alphabet as Farsi.