Abstract
Modern gravitational-wave science demands increasingly accurate and
computationally intensive numerical relativity (NR) simulations. The
Python-based, open-source NRPy framework generates optimized C/C++ code for NR,
including the complete NR code BlackHoles@Home (BH@H), which leverages
curvilinear coordinates well-suited to many astrophysical scenarios.
Historically, BH@H was limited to single-node OpenMP CPU parallelism. To
address this, we introduce superB, an open-source extension to NRPy that
enables automatic generation of scalable, task-based, distributed-memory
Charm++ code from existing BH@H modules. The generated code partitions the
structured grids used by NRPy/BH@H, managing communication between them. Its
correctness is validated through bit-identical results with the standard OpenMP
version on a single node and via a head-on binary black hole simulation in
cylindrical-like coordinates, accurately reproducing quasi-normal modes (up to
$\ell=8$). The superB/NRPy-generated code demonstrates excellent strong
scaling, achieving an $\approx 45$x speedup on 64 nodes (7168 cores) compared
to the original single-node OpenMP code for a large 3D vacuum test. This
scalable infrastructure benefits demanding simulations and lays the groundwork
for future multi-patch grid support, targeting long inspirals, extreme
parameter studies, and rapid follow-ups. This infrastructure readily integrates
with other NRPy/BH@H-based projects, enabling performant scaling for the
general relativistic hydrodynamics code GRoovy, and facilitating future
coupling with GPU acceleration via the NRPy-CUDA project.