I am a PhD student working at SCOREC on parallel finite element tools for supercomputers, specifically for communication, field computation, and mesh adaptation. I currently maintain several core SCOREC libraries
The SCOREC core contains many packages for dealing with parallel unstructured finite element meshes. It is currently split into open-source code on Github and a more advanced version coupled to tools from Simmetrix, Inc. is on Bitbucket.Github site
PCU is a fairly well-tested parallel communication layer built on MPI, which provides termination detection of communication rounds and supports passing messages between threads for hybrid programming.
APF is a newly developed system to store finite element field data and apply operators to these fields. It includes an abstraction of mesh databases and extensions for error estimation and linear system assembly.