Mining open source text documents for intelligence gathering

Abstract

Intelligence collection and analysis always play a major role in a company's growth. Traditional intelligence management process was most concealed and required massive human effort. It also has the disadvantages of rarity and danger. Therefore open source intelligence (OSINT) emerged as a major intelligence collection and analysis approach. Differing from traditional approach, the sources of OSINT are publicly accessible and have the properties of openness and massiveness which may result in disadvantages of inconsistency and lack of validation. For now, most of the OSINT processing is conducted manually which requires massive human effort and time cost. Automatic processing of OSINT is then unavoidable for modern applications. Although there exists software services to aid such automatic processing, the functionality and degree of automation are still immature and limited. In this work we developed an automatic processing approach for OSINT based on proposed text mining techniques. This approach may automatically identify interesting events from various aspects from which business could benefit. The major contribution of this work is that we have developed high-order mining techniques for OSINT, which will benefit domains like national security, personal knowledge management, business intelligence, e-learning, etc.