Recently, damped outrigger structures have shown great potentials in suppressing vibrations of tall buildings subjected to seismic and wind excitations. To gain insight into the seismic performance of damped outrigger structures, this paper proposes a stochastic optimization-based sensitivity analysis procedure (SOSAP) for comparative study of damped outrigger structures implemented with inerter dampers (ID) and negative stiffness dampers (NSD) subjected to stochastic seismic excitation. The novelty of this procedure is it integrates stochastic optimization with seismic excitation modeled using the Clough-Penzien spectrum, Pareto optimal fronts dealing with conflict objectives, and sensitivity analysis of objectives values with respect to both discriminative and common design variables. First, the state-space representations of stochastic seismic excitation and structures implemented either with inerter damped outriggers (IDO) or negative stiffness damped outriggers (NSDO) are combined to yield the representations of the augmented structure-damper-earthquake systems. Pareto optimal fronts are then used to examine the tradeoffs between competing optimization objectives, which are defined as harmful interstory drift and floor absolute acceleration via the solutions of the Lyapunov equations. Subsequently, the stochastic seismic responses of the IDO and NSDO systems are compared by investigating the sensitivity of objective values with respect to inertance, negative stiffness ratio and viscous coefficient. The obtained optimal parameters of comparative IDO and NSDO systems are further examined under real earthquake records from the perspectives of dynamic response and energy balance. These investigations demonstrate the efficacy of the proposed SOSAP, reveal the comparative seismic performance between the IDO and NSDO systems dependent on the optimization objectives and the available amount of ID and NSD, which provides insight into the optimal design of tall buildings with different design strategies and conflicting objectives subjected to stochastic seismic excitation.