Text Based Knowledge Discovery with Information Flow Analysis

Abstract

Information explosion has led to diminishing awareness: disciplines are becoming increasingly specialized; individuals and groups are becoming ever more insular. This paper considers how awareness can be enhanced via text-based knowledge discovery. Knowledge representation is motivated from a socio-cognitive perspective. Concepts are represented as vectors in a high dimensional semantic space automatically derived from a text corpus. Information flow computation between vectors is proposed as a means of discovering implicit associations between concepts. The potential of information flow analysis in text based knowledge discovery has been demonstrated by two case studies: literature-based scientific discovery by attempting to simulate Swanson’s Raynaud-fish oil discovery in medical texts; and automatic category derivation from document titles. There is some justification to believe that the techniques create awareness of new knowledge.KeywordsInformation FlowLatent Semantic AnalysisImplicit AssociationLatent Semantic IndexingDocument TitleThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.