The rise of the Internet of Things (IoT) has increased the emphasis on protecting device communication. The Constrained Application Protocol (CoAP) extensively utilizes Datagram Transport Layer Security (DTLS) to provide end-to-end security; however, DTLS has a high connection overhead. This work presents a new Group Oriented Lightweight Payload-based Mutual Authentication (GOLPMA) mechanism that authenticates clients and servers before permitting communication using the Advanced Encryption Standard (AES). GOLPMA combines and improves essential parts of the CoAP architecture, resulting in a low-overhead, dependable, and computationally efficient alternative to DTLS. The payload-based mutual authentication technique only requires two round-trip message exchanges and restricts the payload of each message to 256 bits. GOLPMA surpasses typical DTLS/CoAP techniques based on throughput, latency, packet delivery ratio, average response time, mean jitter, authentication time and energy consumption in two separate network topologies with 15 and 30 nodes when examined using the Cooja simulator on Contiki OS. The results suggest that GOLPMA could be a safe, low-power replacement for DTLS in IoT applications