Navigation of underactuated wheeled inverted pendulum (WIP) vehicles in unknown environments is still facing great difficulties, especially when the optimal motion is required. This article proposes an optimal trajectory planning method for the navigation of WIP vehicles in unknown environments, where various performance demands, such as security, smoothness, efficiency, etc., are all considered. First, a map-building algorithm based on the improved Rao-Blackwellized particle filter is applied for the WIP vehicle to construct the environmental map. Then, a multiobjective optimization using the genetic algorithm is performed to find an optimized path between the given start and target point with path length, path curvature, and safe distance being taken into consideration simultaneously. Moreover, on the basis of kinematical and dynamical analysis, velocity, and acceleration constraints are parameterized with a path parameter, and the minimum-time trajectory along the optimized path is further planned with a sequence of maximum acceleration and deceleration trajectories. Finally, a WIP vehicle platform based on the robot operating system is designed, and related experiments in a real obstacle environment are conducted to validate the feasibility of the proposed method.