Handwritten Content Research Articles

Enhancing Study Experience using Handwritten Character and Digit Recognition and Text Summarization

The integration of Handwritten characters and, Digit Recognition and Deep Learning in education heralds a transformative era in learning methodologies. This abstract delves into the multifaceted benefits derived from the amalgamation of these technologies, redefining the educational landscape. Handwritten characters and Digit Recognition technology facilitates the seamless digitization of handwritten content, transcending the limitations of manual note-taking. Its introduction into educational frameworks enhances accessibility, promotes organization, and augments the searchability of diverse educational materials. Deep Learning, acting in tandem with Handwriting Recognition, amplifies these advantages manifold. Deep learning powered study assistants offer personalized learning experiences tailored to individual needs, adapting to varied learning styles. Additionally, these assistants facilitate collaborative opportunities, providing real time feedback and evaluation tools that revolutionize the learning process. Key Words: Image recognition, CNN, RNN, LSTM, Neural Networks, SVM, Deep Learning, Convolutional layer, PreLU, ReLU, Text Summarization, Extractive text summarization, Tokenization.

Segmentation techniques for Arabic handwritten: a review

Image segmentation refers to the process of partitioning a page into distinct sections. This technique aims to improve and transform the image's representation into a more coherent and user-friendly format. Its common application involves identifying objects and boundaries (such as lines and curves) within images. However, this paper focuses on discussing segmentation methods specifically tailored for Arabic handwritten content. Dealing with the segmentation of Arabic handwritten material poses a significant challenge due to the diverse handwriting styles and the interconnection between Arabic letters. The paper will also touch on the classification of segmentation algorithms originally designed for modern documents, illustrating their adaptation in document processing. Furthermore, the paper will address the difficulties associated with segmenting Arabic handwritten content, including variations in writing style, the connected nature of Arabic characters, the complexity of Arabic cursive writing and as well as the diacritics challenges. Lastly, a concise overview of previously widely used segmentation techniques in various research endeavors will be provided.

Highly elastic, frost-resistant and antimicrobial flexible sensor inspired by ramen noodles for human motion detection and deep learning of handwritten content

Biomaterial-based hydrogels have gained significant attention in the development of flexible multifunctional electronic devices for the future. However, it is a great challenge to prepare biomaterial-based hydrogels with high toughness, electrical conductivity and frost-resistant by a simple and low-cost way. Meanwhile, starch bio-based hydrogels have emerged as a popular area of research due to their abundant sources and degradability. Unfortunately, these hydrogels often exhibit poor mechanical properties and freeze resistance, which restrict their practical applications. Taking inspiration from the traditional food ramen, where a simple mixture of noodles can become remarkably stretchable and resilient, our study aims to enhance the performance of hydrogels by leveraging their toughening mechanism. We achieve this by combining polyvinyl alcohol (PVA), branched chain starch (Amy), wheat protein (WP), ethylene glycol (EG) with saline. The Amy/PVA/EG/WP/S (APEWS) organohydrogels demonstrated excellent mechanical properties (1.12 MPa) due to the combination of PVA as a hard backbone, a rich protein network, diverse functional groups, and saline components. Additionally, the binary solvent consisting of water and EG allowed the hydrogel to function effectively even at low temperatures (-20 °C). The APEWS organohydrogel exhibited promising sensing capability (GF = 1.46) and sensing stability (>500 times) as a human wearable strain sensor, making it suitable for detecting human motion and the hydrogel showed excellent antimicrobial properties against Escherichia coli (Gram-negative bacteria). Furthermore, we employed a Long Short-Term Memory (LSTM) artificial neural network for deep learning-based recognition of written content. The network achieved an accuracy of over 95.5% for different English letters and 100% for symbols. These results highlight the potential of this technology in applications such as writing sensing and information encryption. In conclusion, with the simple preparation method and low cost, APEWS organohydrogels are expected to be the next generation of green and sustainable bio-based strain sensors.

Offline HWR Accuracy Enhancement with Image Enhancement and Deep Learning Techniques

Handwriting recognition (HWR) is the ability of a machine to recognize handwritten text present in images, scanned or photographed. While much of the work reported in this field deals with scanned images, recognizing handwritten content from camera-captured images is still a challenge. While researchers are focusing their efforts to improve on the recognition process of such characters, the current work focuses efforts outside the recognition process.This work employs image enhancement techniques, noise removal, and binarization with adaptive thresholding & segmentation, to increase HWR accuracy. Additionally, the work of employs deep learning-based language models, such as BERT & BART, to further enhance the accuracy of recognized text. It is observed that HWR accuracy increases when smaller images such as word images are used as input to the HWR engine. On the post-processing front, BART is demonstrated to be superior in enhancing the accuracy of HWR-recognized text documents in a comparative study presented in this work. An accuracy improvement of 20+% is achieved with the BART model enhanced with transfer learning through a localized dataset to handle HWR-specific error patterns.

Machine-Learning Assisted Handwriting Recognition Using Graphene Oxide-Based Hydrogel.

Machine-learning assisted handwriting recognition is crucial for development of next-generation biometric technologies. However, most of the currently reported handwriting recognition systems are lacking in flexible sensing and machine learning capabilities, both of which are essential for implementation of intelligent systems. Herein, assisted by machine learning, we develop a new handwriting recognition system, which can be applied as both a recognizer for written texts and an encryptor for confidential information. This flexible and intelligent handwriting recognition system combines a printed circuit board with graphene oxide-based hydrogel sensors. It offers fast response and good sensitivity and allows high-precision recognition of handwritten content from a single letter to words and signatures. By analyzing 690 acquired handwritten signatures obtained from seven participants, we successfully demonstrate a fast recognition time (less than 1 s) and a high recognition rate (∼91.30%). Our developed handwriting recognition system has great potential in advanced human-machine interactions, wearable communication devices, soft robotics manipulators, and augmented virtual reality.

Document image analysis and recognition: a survey

This paper analyzes the problems of document image recognition and the existing solutions. Document recognition algorithms have been studied for quite a long time, but despite this, currently, the topic is relevant and research continues, as evidenced by a large number of associated publications and reviews. However, most of these works and reviews are devoted to individual recognition tasks. In this review, the entire set of methods, approaches, and algorithms necessary for document recognition is considered. A preliminary systematization allowed us to distinguish groups of methods for extracting information from documents of different types: single-page and multi-page, with text and handwritten contents, with a fixed template and flexible structure, and digitalized via different ways: scanning, photographing, video recording. Here, we consider methods of document recognition and analysis applied to a wide range of tasks: identification and verification of identity, due diligence, machine learning algorithms, questionnaires, and audits. The groups of methods necessary for the recognition of a single page image are examined: the classical computer vision algorithms, i.e., keypoints, local feature descriptors, Fast Hough Transforms, image binarization, and modern neural network models for document boundary detection, document classification, document structure analysis, i.e., text blocks and tables localization, extraction and recognition of the details, post-processing of recognition results. The review provides a description of publicly available experimental data packages for training and testing recognition algorithms. Methods for optimizing the performance of document image analysis and recognition methods are described.

Student-led development of a lightboard to enhance future student learning

Lightboards are emerging technologies, helping to optimize online and flipped classroom environments by enabling the instructor to face the audience while writing on a transparent board. Although both commercial and do-it-yourself designs of lightboards are currently available, we have conducted a student-led engineering research and design project to optimize a lightboard design for broader use on campus to enhance the learning of future students. Through systematic side-by-side comparison, the team determined that Starphire Ultra-Clear™ glass is more appropriate than standard window glass for larger lightboard designs, despite its higher cost. The fully built and validated solution is housed in the campus Library for broad implementation into classroom environments throughout all institution outlets. The effectiveness of the lightboard is evaluated through a student comparison of short online videos generated using three separate formats: narrated slides, whiteboard, and the new lightboard. Students in a hybrid in-person and online class favored having the instructor facing the camera (76%), hand-written content (64%), and the use of more advanced technology (67%). Furthermore, through a pairwise polling comparison, students consistently favored the lightboard over narrated slides or simple whiteboard presentation formats.

Online recognition of handwritten characters from scalp-recorded brain activities during handwriting

Objective. Brain–computer interfaces aim to build an efficient communication with the world using neural signals, which may bring great benefits to human society, especially to people with physical impairments. To date, the ability to translate brain signals to effective communication outcome remains low. This work explores whether the handwriting process could serve as a potential interface with high performance. To this end, we first examined how much the scalp-recorded brain signals encode information related to handwriting and whether it is feasible to precisely retrieve the handwritten content solely from the scalp-recorded electrical data. Approach. Five participants were instructed to write the sentence ‘HELLO, WORLD!’ repeatedly on a tablet while their brain signals were simultaneously recorded by electroencephalography (EEG). The EEG signals were first decomposed by independent component analysis for extracting features to be used to train a convolutional neural network (CNN) to recognize the written symbols. Main results. The accuracy of the CNN-based classifier trained and applied on the same participant (training and test data separated) ranged from 76.8% to 97.0%. The accuracy of cross-participant application was more diverse, ranging from 14.7% to 58.7%. These results showed the possibility of recognizing the handwritten content directly from the scalp level brain signal. A demonstration of the recognition system in an online mode was presented. The major factor that grounded the recognition was the close association between the rich dynamics of electroencephalogram source activities and the kinematic information during the handwriting movements. Significance. This work revealed an explicit and precise mapping between scalp-level electrophysiological signals and linguistic information conveyed by handwriting, which provided a novel approach to developing brain computer interfaces that focus on semantic communication.

FCN-LectureNet: Extractive Summarization of Whiteboard and Chalkboard Lecture Videos

Recording and sharing of educational or lecture videos has increased in recent years. Within these recordings, we find a large number of math-oriented lectures and tutorials which attract students of all levels. Many of the topics covered by these recordings are better explained using handwritten content on whiteboards or chalkboards. Hence, we find large numbers of lecture videos that feature the instructor writing on a surface. In this work, we propose a novel method for extraction and summarization of the handwritten content found in such videos. Our method is based on a fully convolutional network, FCN-LectureNet, which can extract the handwritten content from the video as binary images. These are further analyzed to identify the unique and stable units of content to produce a spatial-temporal index of handwritten content. A signal which approximates content deletion events is then built using information from the spatial-temporal index. The peaks of this signal are used to create temporal segments of the lecture based on the notion that sub-topics change when large portions of content are deleted. Finally, we use these segments to create an extractive summary of the handwritten content based on key-frames. This will facilitate content-based search and retrieval of these lecture videos. In this work, we also extend the AccessMath dataset to create a novel dataset for benchmarking of lecture video summarization called LectureMath. Our experiments on both datasets show that our novel method can outperform existing methods especially on the larger and more challenging dataset. Our code and data are publicly available.

Basic research on a handwritten note image recognition system that combines two OCRs

In this paper, we propose an OCR system that recognizes the contents of manual writing in handouts. Since the target handouts contain a mixture of Japanese and mathematical formulas, we thought that one type of OCR systems alone would not provide sufficient accuracy. Therefore, in this proposed system, we will consider combining two types of OCR systems. The OCRs to be combined are Tesseract and Mathpix, which are good at recognizing Japanese and mathematical formulas, respectively. When the two OCRs are combined, the final plausible result must be selected, but we wondered if the OCR score could be used as an indicator of selection. The OCR score is a self-evaluation value that is output together with the recognition result. However, there is no guarantee that the OCR score and the actual recognition result show the same tendency. Therefore, in this paper, after classifying the written contents into three categories, words, sentences, and mathematical formulas, we verified whether the recognition results of each OCR can be appropriately selected based on the OCR score. At this point, the result is that further improvement is needed to make the selection based on the OCR score alone. However, if it becomes possible to classify handwritten contents with the proposal system by additional learning etc., it will be possible to analyze the handwritten contents efficiently, and it is expected that it will be possible to improve the handouts and provide feedback to the lesson plan.

Generation of Synthetic Data for Handwritten Word Alteration Detection

Fraudsters often alter handwritten contents in a document in order to achieve illicit purposes. At times, this may result in financial and mental loss to an individual or an organization. Hence, ink analysis is necessary to identify such an alteration. Convolution Neural Network (CNN) can be used to identify such cases of alteration, as CNN has emerged as a monumental success in the field of computer vision for varieties of classification tasks. But, CNN requires large amount of labeled data for training. Hence, there is a need to generate a large dataset for the experiments relating to handwritten word alteration detection. Collection, digitization, and cropping of a large number of altered and unaltered handwritten words are tedious and time consuming. To overcome such an issue, an approach for synthetic word data generation is presented in this paper for handwritten word alteration detection experiments. This scheme is designed in such a way that the synthetically generated words are very similar to the original ones. In order to achieve this, handwritten character data set is prepared using 10 blue and 10 black pens. These handwritten characters are used for creating synthetic word alteration data set. The presented approach uses relatively less number of handwritten character images to create a huge word alteration data set. Further, deep learning models are trained on the synthetically generated data set for word alteration detection.

Enhancing Predictability of Handwritten Document Content using HTR and Word Substitution

Handwritten Text Recognition (HTR) can become progressively abysmal when the documents are damaged with smudges, blemishes and blurs. Recognition of such documents is a challenging task. We, therefore propose a system to identify textual handwritten content in documents where the state-of-the-art Optical Character Recognition (OCR) existing at its full extent performs with low accuracy. By introducing word substitution using character and distance analysis for spell checking and word completion in such areas for giving out more accurate results using a word corpus, we improved our prediction results especially in cases where the OCR is prone to predict false positives on the smudge areas predominantly. Blur detection on every word before segmentation is also substituted with a new word by our OCR algorithm to avoid false positive results and are instead substituted with suitable words. This methodology is far more convenient and reliable since even state-of-the-art HTR technologies do not have more than 71% accuracy. The accuracy of the predicted test is measured using the text similarity metric - Fuzzy Token Set Ratio (FTSR).

The Semantic Field Book Annotator

Biodiversity research expeditions to the globe’s most biodiverse areas have been conducted for several hundred years. Natural history museums contain a wealth of historical materials from such expeditions, but they are stored in a fragmented way. As a consequence links between the various resources, e.g., specimens, illustrations and field notes, are often lost and are not easily re-established. Natural history museums have started to use persistent identifiers for physical collection objects, such as specimens, as well as associated information resources, such as web pages and multimedia. As a result, these resources can more easily be linked, using Linked Open Data (LOD), to information sources on the web. Specimens can be linked to taxonomic backbones of data providers, e.g., the Encyclopedia Of Life (EOL), the Global Biodiversity Information Facility (GBIF), or publications with Digital Object Identifiers (DOI). For the content of biodiversity expedition archives, (e.g. field notes), no such formalisations exist. However, linking the specimens to specific handwritten notes taken in the field can increase their scientific value. Specimens are generally accompanied by a label containing the location of the site where the specimen was collected, the collector’s name and the classification. Field notes often augment the basic metadata found with specimens with important details concerning, for instance, an organism’s habitat and morphology. Therefore, inter-collection interoperability of multimodal resources is just as important as intra-collection interoperability of unimodal resources. The linking of field notes and illustrations to specimens entails a number of challenges: historical handwritten content is generally difficult to read and interpret, especially due to changing taxonomic systems, nomenclature and collection practices. It is vital that: the content is structured in a similar way as the specimens, so that links can more easily be re-established either manually or in an automated way; for consolidation, the content is enriched with outgoing links to semantic resources, such as Geonames or Virtual International Authority File (VIAF); and this process is a transparent one: how links are established, why and by whom, should be stored to encourage scholarly discussions and to promote the attribution of efforts. the content is structured in a similar way as the specimens, so that links can more easily be re-established either manually or in an automated way; for consolidation, the content is enriched with outgoing links to semantic resources, such as Geonames or Virtual International Authority File (VIAF); and this process is a transparent one: how links are established, why and by whom, should be stored to encourage scholarly discussions and to promote the attribution of efforts. In order to address some of these issues, we have built a tool, the Semantic Field Book Annotator (SFB-A), that allows for the direct annotation of digitised (scanned) pages of field books and illustrations with Linked Open Data (LOD). The tool guides the user through the annotation process, so that semantic links are automatically generated in a formalised way. These annotations and links are subsequently stored in an RDF triplestore. As the use of the Darwin Core standard is considered best practice among collection managers for the digitisation of their specimens, our tool is equipped with an ontology based on Darwin Core terms, the NHC-Ontology, which extends the Darwin Semantic Web (DSW) ontology. The tool can annotate any image, be it an image of a specimen with a textual label, an illustration with a textual label or a handwritten species description. Interoperability of annotations between the various resources within a collection is therefore ensured. Terms in the ontology are structured using OWL web ontology language. This allows for more complex tasks such as OWL reasoning and semantic queries, and facilitates the creation of a richer knowledge base that is more amenable to research.

Generalized framework for summarization of fixed-camera lecture videos by detecting and binarizing handwritten content

We propose a framework to extract and binarize handwritten content in lecture videos. The extracted content could potentially be used to index video collections powering content-based search and navigation within lecture videos helping students and educators across the world. A deep learning pipeline is used to detect handwritten text, formulae and sketches and then binarize the extracted content. We exploit the spatio-temporal structure of our binarized detections to compute associativity information of content across all video frames. This information is later used to segment the video. Experiments are conducted to compare the performance of key components of our framework in isolation, as well as the impact on overall performance, with respect to existing methods. We evaluate our framework on the publicly available AccessMath lecture video dataset obtaining an f-measure of $$94.32\%$$ for binary connected components. Code for the framework (including trained weights) and summarization will be released.

Forensic study on the effect of age and illness (Parkinsonism) on handwriting characteristics

BackgroundHandwriting is a neuromuscular task. Old age and illness may produce significant deteriorating effects on the quality of handwriting/signatures of an individual. In some cases, the affected writings may be suspected of being forged due to poor line quality of handwriting. Important documents such as wills, receipts, agreements, contracts, notes, and drafts involve handwritten contents that are often questioned to prove their identity and genuineness. The document examiners may face difficulty to get an adequate number of contemporary standard samples while handling questioned handwriting samples of old age persons and Parkinsonism patients. Therefore, we conducted a qualitative and statistical study to analyze the changes in various handwriting features due to old age and Parkinsonism.MethodologyHandwriting samples from 50 healthy old age persons and 25 individuals suffering from Parkinsonism were collected with their consent. Handwriting features such as line quality defects (tremors, retouchings, and pen lifts), letter formations, abbreviation/simplification of letters or their parts, nature of initial and terminal strokes, nature of connecting strokes, writing speed, overall size, and alignment were analyzed qualitatively and statistically through paired t test using SPSS software.ResultsAccording to results, marked variations were evident in most of the handwriting features in case of old age persons and in case of individuals suffering from Parkinsonism. Two features, namely retouchings and alignment, were least affected in the handwritings of patients suffering from Parkinsonism.ConclusionAccording to the results of this study, old age and illness affect the handwriting of an individual in a considerable manner. Marked deterioration in the quality of handwriting/signatures can be observed in case of old age persons and in individuals suffering from Parkinsonism, which makes their writing appear different altogether as compared to their corresponding normal writing. The document examiner must be aware of this fact while examining such cases.

Attentional bias and the Suicide Status Form: Behavioral perseveration of written responses

Given the vast public health problem of suicide, the need for more effective assessment of suicidal risk is clear. The major approaches applied to this challenge include various direct approaches (e.g., suicide-focused interviews) and indirect approaches (e.g., implicit methodologies or “occult” assessments) that tend to assess an attentional bias for suicidal risk, the latter of which the present investigation sought to study. Using the Suicide Status Form (SSF)—the central multi-purpose tool that is used within a collaborative assessment process with suicidal patients who are engaged in the “Collaborative Assessment and Management of Suicidality” (CAMS; Jobes, 2016)—we aimed to investigate the influence of perseverative, hand-written content responses pertaining to potential suicidal risk. Specifically, we explored whether repeating certain topic content might reflect a perseverative response style; we thus compared written content results of first session SSFs taken from a sample of suicidal U.S. Army Soldiers (Study 1) and a sample of suicidal college students (Study 2). Across the two studies, patients who repeated the same content (“1-Topic Repeaters”) had significantly higher ratings related to suicidal ideation in comparison to those with more heterogeneous response styles. This replicated finding perhaps reveals a form of behavioral perseveration that is potentially related to increased suicidal risk with possible implications for successful treatment.

A Survey on using Neural Network based Algorithms for Hand Written Digit Recognition

The detection and recognition of handwritten content is the process of converting non-intelligent information such as images into machine edit-able text. This research domain has become an active research area due to vast applications in a number of fields such as handwritten filing of forms or documents in banks, exam form filled by students, users’ authentication applications. Generally, the handwritten content recognition process consists of four steps: data preprocessing, segmentation, the feature¬ extraction and selection, application of supervised learning algorithms. In this paper, a detailed survey of existing techniques used for Hand Written Digit Recognition(HWDR) is carried out. This review is novel as it is focused on HWDR and also it only discusses the application of Neural Network (NN) and its modified algorithms. We discuss an overview of NN and different algorithms which have been adopted from NN. In addition, this research study presents a detailed survey of the use of NN and its variants for digit recognition. Each existing work, we elaborate its steps, novelty, use of dataset and advantages and limitations as well. Moreover, we present a Scientometric analysis of HWDR which presents top journals and sources of research content in this research domain. We also present research challenges and potential future work.

Handwriting verification – Comparison of a multi-algorithmic and a multi-semantic approach

In this paper, a comparison of an existing multi-algorithmic and a new multi-semantic fusion approach for biometric online handwriting user verification is presented. First, in order to improve the authentication performance of a biometric online handwriting system four classification algorithms are combined using several weighting strategies for matching score level fusion. Second, based on the best two algorithms and the best weighting strategy found during the test of the multi-algorithmic approach, a new multi-semantic fusion approach using a pair wise combination of four semantics on matching score level is proposed. As semantics we understand alternative handwritten contents (e.g. symbols) in addition to signatures. We show that both fusion approaches, multi-algorithmic and multi-semantic, can lead to a fusion result which is better than the result of the best single algorithm or semantics involved. While the improvement for the multi-algorithmic system yields 19%, we observe more than 57% for the multi-semantic approach.

Extracting content from instructional videos by statistical modelling and classification

This paper presents a robust approach to extracting content from instructional videos for handwritten recognition, indexing and retrieval, and other e-learning applications. For the instructional videos of chalkboard presentations, retrieving the handwritten content (e.g., characters, drawings, figures) on boards is the first and prerequisite step towards further exploration of instructional video content. However, content extraction in instructional videos is still challenging due to video noise, non-uniformity of the color in board regions, light condition changes in a video session, camera movements, and unavoidable occlusions by instructors. To solve this problem, we first segment video frames into multiple regions and estimate the parameters of the board regions based on statistical analysis of the pixels in dominant regions. Then we accurately separate the board regions from irrelevant regions using a probabilistic classifier. Finally, we combine top-hat morphological processing with a gradient-based adaptive thresholding technique to retrieve content pixels from the board regions. Evaluation of the content extraction results on four full-length instructional videos shows the high performance of the proposed method. The extraction of content text facilitates the research on full exploitation of instructional videos, such as content enhancement, indexing, and retrieval.