The increasing use of the Internet of Things (IoT) has driven the demand for enhanced and robust access control methods to protect resources from unauthorized access. A cloud-based access control approach brings significant challenges in terms of communication overhead, high latency, and complete reliance. In this paper, we propose a Fog-Based Adaptive Context-Aware Access Control (FB-ACAAC) framework for IoT devices, dynamically adjusting access policies based on contextual information to prevent unauthorised resource access. The main purpose of FB-ACAAC is to provide adaptability to changing access behaviors and context by bringing decision-making and information about policies closer to the end nodes of the network. FB-ACAAC improves the availability of resources and reduces the amount of time for information to be processed. FB-ACAAC extends the widely used eXtensible Access Control Markup Language (XACML) to manage access control decisions. Traditional XACML-based methods do not take into account changing environments, different contexts, and changing access behaviors and are vulnerable to certain types of attacks. To address these issues, FB-ACAAC proposes an adaptive context-aware XACML scheme for heterogeneous distributed IoT environments using fog computing and is designed to be context-aware, adaptable, and secure in the face of unauthorised access. The effectiveness of this new scheme is verified through experiments, and it has a low processing time overhead while providing extra features and improved security.