General purpose finite element analysis of problems with billions of degrees of freedom

Abstract

The authors are the lead developers in the ParaFEM project. ParaFEM is an open source platform for parallel finite element analysis that builds on the serial codes provided with the text book Smith, Griffiths and Margetts ?Programming the Finite Element Method?, 5th Edition, Wiley, 2014. The latest version of the code provides parallel programs for a range of problem types including linear, nonlinear and dynamic analysis. The philosophy of the project is to provide a foundation for teaching and research. Undergraduate students can use the software together with the book to learn how finite element programs are written. Researchers (and practising engineers) can modify or extend the software to solve their own engineering problems.This paper will focus on capability. ParaFEM has been used to solve problems with more than a billion degrees of freedom and scales on up to 32,000 cores using modern supercomputers. A billion equations is roughly equivalent to a cube of 20-noded hexahedral elements of side 400x400x400. Nowadays, a 400x400 element 2D problem would not be considered large. However in 3D, problem sizes grow rapidly, challenging widely used commercial packages. Assuming perfect scaling, a problem that runs overnight on 32,000 cores (12 hours say) would take 43 years to solve on a single core ? more than the entire working life of a qualified engineer for one run!One of the application areas that we believe can benefit from the ability to solve large problems is ?image-based modelling? whereby a 3D image of a material or structure is converted into a finite element mesh. X-ray tomography systems can produce detailed images with 8000x8000x8000 voxels (3D pixels) or 64 billion hexahedra. An advantage of image-based modelling is that it is possible to compare simulation and experiment for the same specimen, thus providing a unique route for the verification and validation of engineering analyses. The authors will present engineering case studies that illustrate the simulation of high resolution models using tens of thousands of cores.