SPEG—Semiotics-Based Panel Extraction from Graphic Novel

Abstract

The researchers of the graphic novels face challenges coping with the graphic novel images as they are associated with the range of designs, layout, text, and actions. These challenges make the content learning and object detection task much more difficult. To overcome this, deep learning approaches are incorporated in several domains through the use of machine learning techniques. A graphic novel is the composition of a text and graphic. To fully analyze the content of the graphic novel, understanding of the story, dialogs, line drawings, characters, and their location is required. Especially in comic analysis, detection of comic characters has been an interesting area as it inculcates adequate understanding of comics. The comparisons between graphic communication and languages are standardized in visual language theory (VLT). The visual language consists of signs that are highly conventionalized, but they vary according to the distribution of where they are positioned inside the panel of the graphic novel strip. The visual morphology uses semiotic references such as motion lines, scopic lines, radial lines, focal lines, spikes, twirls, spirals, and the shapes such as heart and stars. Depending on the location they are placed, the meaning varies. The research studies are focused to identify these conventions and how these signs interact and modify others. In this work, to identify the semiotics at different locations from a graphic novel strip, a custom YOLOv3 detector model is trained followed by the panel extraction. The individual panels are extracted using contour analysis. The trained model could detect the semiotics from the graphic novel images when they are placed around the character of interest with the mean average precision of 75.8%. The proposed method SPEGYOLO extracts the semiotics in the panels of graphic novels and further analysis based on their location and orientation will help the users to apprehend the meaning associated with it.