Mobile robots are receiving significant interest in the nuclear energy industry because of their potential intelligence features and efficient operation. Mobile robot path planning (PP) in a radioactive environment can be considered as finding a collision-free path constrained by the path length, cumulative radiation dose rate and energy consumption. To solve this multiobjective path planning (MOPP) problem, we propose a hybrid algorithm based on an improved ant colony optimization (IACO), the A* algorithm and particle swarm optimization (IACO-A*-PSO). First, a modified A* algorithm is presented to find a suboptimal path, which is used to improve the initial ACO pheromone. Next, an improvement of the adaptive heuristic function and pheromone update rule based on an elitist system is proposed to balance the global search ability and convergence speed of the ant colony algorithm. Finally, PSO is used to obtain the optimal ACO control parameters and multiobjective weight coefficients. To evaluate the efficiency of the proposed algorithm, a comparative study has been performed between the proposed IACO-A*-PSO, the A*, ACO, and IACO algorithms and a genetic algorithm (GA) in four radioactive indoor environments with different sizes, radioactive sources and obstacles. The simulation results show that the proposed algorithm has the best comprehensive performance and stronger adaptability to different environments than the A*, ACO, IACO and GA algorithms in terms of the path length, cumulative radiation dose, energy consumption, runtime and success rate simultaneously. This demonstrates that the proposed algorithm is an effective method for solving the MOPP problem of mobile robot in radioactive environment, which is beneficial for improving the safety and reliability of robots in nuclear energy.