Mixed-mode implementation of PETSc for scalable linear algebra on multi-core processors

With multi-core processors a ubiquitous building block of modern supercomputers, it is now past time to enable applications to embrace these developments in processor design. To achieve exascale performance, applications will need ways of exploiting the new levels of parallelism that are exposed in modern high-performance computers. A typical approach to this is to use shared-memory programming techniques to best exploit multi-core nodes along with inter-node message passing. In this paper, we describe the addition of OpenMP threaded functionality to the PETSc library. We highlight some issues that hinder good performance of threaded applications on modern processors and describe how to negate them. The OpenMP branch of PETSc was benchmarked using matrices extracted from Fluidity, a CFD application code, which uses the library as its linear solver engine. The overall performance of the mixed-mode implementation is shown to be superior to that of the pure-MPI version.