Most IoT devices cannot afford to be a blockchain node due to the high computation and storage loads. Thus, the blockchain is usually deployed on one delegate node, e.g., the edge device or cloud, which may encounters three drawbacks: (1) The delegate node becomes the single failure point when the number of delegate notes are limited. (2) The delegate node replicating the blockchain data can lead to privacy information leak. (3) The delegate node is vulnerable to the Distributed Denial of Service (DDoS) attack. To tackle these drawbacks, we consider to minimize the redundant of blockchain to make the IoT devices as the specialized blockchain nodes. In this paper, we integrate a permissioned blockchain (HLF), an attribute-based access control (ABAC) and an identity-based signature (IBS) to build a security, lightweight, and cross-domain blockchain-based IoT access control system. Specifically, we divided the IoT system into different function domains, named IoT domains. Then, we establish a local blockchain ledger for each IoT domain to enable more IoT devices as blockchain nodes. The local blockchain ledger records the IoT domain entities’ attributes, policy files’ digests, and access decisions. Meanwhile, we use the channel technology of HLF to realize cross-domain access and use the IBS to filter the legal access requests for each IoT domain to prevent DDoS attacks. We also design a policy decision point (PDP) selection algorithm that select multiple IoT devices (blockchain nodes) to achieve the real-time distributed policy decisions (off-chain). Finally, we implement and evaluate the proposed system to demonstrate its practicality.