Content coding and search for risk assessment: Problems and solutions

Abstract

Risk assessment depends on data from problem reports (PRs), yet analysis of these reports is difficult. Finding all the pertinent reports is a challenge. Full-text search is commonly used to retrieve groups of similar PRs, but this method typically misses many reports because of variations in spelling, synonyms and technical jargon. Metadata codes are used to classify the content of each PR according to features such as the type of failure mode, defect or cause. However, manual PR coding schemes are difficult to follow and there are frequent coding errors. NASA reports have made recommendations to improve the quality of content codes, but these recommendations have not been adequately addressed. During an assessment for the International Space Station (ISS), alternative solutions to metadata coding were explored. Automatic concept extraction from the content of text descriptions was used to assign concept tags to each PR. The concepts were drawn from large ontology of aerospace language. An automatic rule-based approach was developed to generate interoperable “proxy” codes. The rules associated the extracted concepts with manual PR codes. The work to automatically define proxy codes was abandoned because of the poor quality of the manual content codes. Lessons learned and recommended corrective and preventive actions for manual coding errors are discussed. They include improvements in development, definition, evaluation, review and correction of content coding errors and coding schemes. Automated concept extraction is a viable solution to the poor quality of manual PR codes. The benefits of this approach are discussed.