The datagram transport layer security (DTLS) is a de facto standard for the end-to-end security of the constrained application protocol (CoAP) that defines the following three security modes: preshared key (PSK), raw public key, and certificate. The pros and cons of each security mode are obvious. Even though the PSK mode is the most preferable in terms of the performance of the DTLS handshake, the in-advance distribution of a unique symmetric key for each pair of endpoints is difficult as the number of pairs increases. Alternatively, the certificate mode provides a convenient key-management functionality but its performance is very poor. The focus of most of the previous works is the reduction of the computational load for a single DTLS handshake that is induced by the certificate mode. In this paper, a group-oriented end-to-end security is considered, together with the introduction of a new security mode. Namely, a security association is established between a CoAP client and a group of CoAP servers (sensor devices); however, a fine-grained access control can be enforced so that each CoAP client can access a limited number of CoAP servers in the group. Furthermore, when each CoAP client performs several DTLS handshakes with the CoAP servers in the group, the first DTLS handshake involves a single public-key operation. A public-key operation, however, is not required for the subsequent DTLS handshakes, so the overall computational burden can be reduced. Also, a testbed was established along with the implementation of the proposed security mechanism for the conduction of a performance comparison with the other security mechanisms. Note to Practitioners —Datagram transport layer security (DTLS) is a standards-based security protocol to secure communications in an IoT environment, including home, factory, or building automation systems. However, to develop a practical security solution based on DTLS, several assumptions inherent to DTLS need to address the practical requirements of various IoT scenarios. Furthermore, some barriers associated with implementing DTLS in an IoT environment should be overcome. This paper presents an interface to extend the basic security functionalities provided by DTLS. Through the interface, a general security platform can be constructed and combined with DTLS. Hence, fine-grained access control and group security can be merged with DTLS, together with standards compliance.