Achieving Efficiency in Black-Box Simulation of Distribution Tails with Self-Structuring Importance Samplers

Abstract

Scalable and efficient importance sampling for managing tail risks As the models employed in the realm of risk analytics and optimization become increasingly sophisticated, it is crucial that risk management tools, such as variance reduction techniques, that are typically designed for stylized models on a case by case basis evolve to scale well and gain broader applicability. In the paper titled “Achieving efficiency in black-box simulation of distribution tails with self-structuring importance samplers,” the authors take a step toward this goal by introducing a novel importance sampling scheme for estimating tail risks of objectives modeled with a diverse range of tools, including linear programs, integer linear programs, feature maps specified with neural networks, etc. Instead of explicitly tailoring the change of distribution for each specific model, as conventionally done, the paper identifies an elementary transformation of the samples. This transformation, when applied alike across a wide variety of models, yields a near-optimal reduction in variance for estimating/optimizing over tail expectations.