Simple yet effective adaptive activation functions for physics-informed neural networks

Abstract

Physics-informed neural networks (PINNs) gained widespread advancements in solving differential equations, where the performance tightly hinges on the choice of activation functions that are inefficient when selected manually. To tackle this issue, we propose two straightforward yet powerful adaptive activation functions: a weighted average function that adjusts activation functions by directly manipulating their weights, and a L2-normalization function that compresses learnable parameters. These methods ensure a consistent sum of weights for each activation function, thereby enhancing optimization efficiency. We assess the performance of these approaches across a range of differential equation problems, encompassing Poisson equation, Wave equation, Burgers equation, Navier-Stokes equation, and linear/nonlinear solid mechanics problems. Through comparisons with exact solutions, we demonstrate significant improvements in convergence rate and solution accuracy. Our results underscore the efficacy of these techniques, providing a simple yet promising pathway for augmenting PINN performance across diverse applications. The source codes and software implementation are available at https://github.com/jzhange/AAF-for-PINNs.