Abstract This paper introduces an Improved Smell Agent Optimization Algorithm (mSAO), a new and enhanced metaheuristic designed to tackle complex engineering optimization issues by overcoming the shortcomings of the recently introduced Smell Agent Optimization Algorithm. The proposed mSAO incorporates the jellyfish swarm active–passive mechanism and novel random operator in the elementary SAO. The objective of modification is to improve the global convergence speed, exploration–exploitation behaviour, and performance of SAO, as well as provide a problem-free method of global optimization. For numerical validation, the mSAO is examined using 29 IEEE benchmarks with varying degrees of dimensionality, and the findings are contrasted with those of its basic version and numerous renowned recently developed metaheuristics. To measure the viability of the mSAO algorithm for real-world applications, the algorithm was employed to solve to resolve eight challenges drawn from real-world scenarios including cantilever beam design, multi-product batch plant, industrial refrigeration system, pressure vessel design, speed reducer design, tension/compression spring, and three-bar truss problem. The computational analysis demonstrates the robustness of mSAO relatively in finding optimal solutions for mechanical, civil, and industrial design problems. Experimental results show that the suggested modifications lead to an improvement in solution quality by 10–20% of basic SAO while solving constraint benchmarks and engineering problems. Additionally, it contributes to avoiding local optimal stuck, and premature convergence limitations of SAO and simultaneously.