Reports of fatal and nonfatal workplace injuries depict severe circumstances involving health and safety in industries. This leads to workers staying away from work in U.S. for an average of 12 days in 2020, implicating in managerial, financial, and organizational losses. In this context, Vision-Based Deep Learning (VBDL) and Knowledge Representation and Reasoning (KRR) allow real-time data retrieval of situations along with the semantic modeling and expressivity of the real world to mitigate injuries. This article presents a framework that interoperates vision-based deep learning and ontology reasoning to identify adverse working situations, introducing a novel ontology composed of a holistic perspective of workers’ health and safety. Moreover, the article provides multi-agent framework modeling to orchestrate the components’ interoperability, describing the framework’s architecture and deployment in workrooms. As a result, a practical evaluation over five days produced 8395 axioms constituted by 1210 individuals in the ontology, which allowed a temporal analysis of harmful conditions and their multiple overlapping using SPARQL and reasoning rules, particularly relevant to understanding explanations of overexertion, physical and environmental injuries. Therefore, the proposed ontology-based framework corroborates the long-term support in identifying, assessing, and controlling risks in the industries due to a well-defined knowledge model.