For an internal combustion engine, a large quantity of fuel energy (accounting for approximately 30% of the total combustion energy) is expelled through the exhaust without being converted into useful work. Various technologies including turbocompounding and the pressurized Brayton bottoming cycle have been developed to recover the exhaust heat and thus reduce the fuel consumption and CO2 emission. However, the application of these approaches in small automotive power plants has been relatively less explored because of the inherent difficulties, such as the detrimental backpressure and higher complexity imposed by the additional devices. Therefore, research has been conducted, in which modifications were made to the traditional arrangement aiming to minimize the weaknesses. The turbocharger of the baseline series turbocompounding was eliminated from the system so that the power turbine became the only heat recovery device on the exhaust side of the engine, and operated at a higher expansion ratio. The compressor was separated from the turbine shaft and mechanically connected to the engine via continuous variable transmission (CVT). According to the results, the backpressure of the novel system is significantly reduced comparing with the series turbocompounding model. The power output at lower engine speed was also promoted. For the pressurized Brayton bottoming cycle, rather than transferring the thermal energy from the exhaust to the working fluid, the exhaust gas was directly utilized as the working medium and was simply cooled by ambient coolant before the compressor. This arrangement, which is known as the inverted Brayton cycle (IBC) was simpler to implement. Besides, it allowed the exhaust gasses to be expanded below the ambient pressure. Thereby, the primary cycle was less compromised by the bottoming cycle. The potential of recovering energy from the exhaust was increased as well. This paper analyzed and optimized the parameters (including CVT ratio, turbine and compressor speed and the inlet pressure to the bottoming cycle) that are sensitive to the performance of the small vehicle engine equipped with inverted Brayton cycle and novel turbocompounding system, respectively. The performance evaluation was given in terms of brake power output and specific fuel consumption. Two working conditions, full and partial load (10 and 2 bar brake mean effective pressure (BMEP)) were investigated. Evaluation of the transient performance was also carried out. Simulated results of these two designs were compared with each other as well as the performance from the corresponding baseline models. The system models in this paper were built in GT-Power which is a one dimension (1D) engine simulation code. All the waste heat recovery systems were combined with a 2.0 L gasoline engine.