PD5: Physics-Based Probabilistic Seismic Hazard Assessment (PSHA) | ChEESE - Centre of Excellence for Exascale Supercomputing in the area of the Solid Earth

Pilot specification

Title

PD5: Physics-Based Probabilistic Seismic Hazard Assessment (PSHA)

Contact Person

Alice-Agnes Gabriel (LMU)

Collaborating institutions

IMO, BSC, TUM, INGV, SCEC, GEM, FAULT2SHA, Icelandic Civil Protection, Italian Civil Protection

Associated Natural hazard

Earthquakes, earthquake coupled tsunamis, earthquake-generated landslides

Software involved

SeisSol (LMU): https://github.com/SeisSol/SeisSol

ExaHyPE (TUM): https://gitlab.lrz.de/exahype/ExaHyPE-Engine

AWP-ODC (SCEC): https://github.com/hzfmer/awp_odc_cpu

SHERIFS (Fault2SHA and GEM): https://github.com/tomchartier/SHERIFS

sam(oa)² (TUM): https://gitlab.lrz.de/samoa/samoa

OpenQuake (GEM): https://github.com/gem/oq-engine

Pre-processing:

Mesh generation tools: Gmsh (open source), Simmetrix/SimModeler (free for academic institution), PUMGen (https://github.com/SeisSol/PUMGen/wiki)

Post-processing & Visualization: Paraview, python tools

Usability

The softwares for the PSHA main workflow are open source codes and can be accessed using the above links. The PSHA results can be used for seismic hazard assessment.

Main objective / mission	The objectives of this PD is to develop general concepts for enabling physics-based seismic hazard assessment with state-of-the-art multi-physics earthquake simulation software (SeisSol, SpecFEM3D, ExaHyPE, AWP-ODC) and conduct 3D physics-based seismic simulations to improve PSHA for validation scenarios provided by IMO (Iceland) and beyond. This PD is expected to be applicable to supplement established methods by stakeholders, for different target regions and varying degrees of complexity.
Workflow description	The workflow of this pilot is shown in Figure 1. To use SeisSol to run fully non-linear dynamic rupture simulations, accounting for various fault geometries, 3D velocity structures, off-fault plasticity, and model parameters uncertainties, to build a fully physics-based dynamic rupture database of mechanically plausible scenarios. Then the linked post-processing python codes (https://github.com/SeisSol/SeisSol/tree/master/postprocessing) are used to extract ground shakings (PGD, PGV, PGA and SA in different periods) from the surface output of SeisSol simulations to build a ground shaking database. SHERIFS uses a logic tree method, with the input of the fault to fault ruptures from dynamic rupture database, converting the slip rate to the annual seismic rate given the geometry of the fault system. With the rupture probability estimation from SHERIFS, and ground shakings from the SeisSol simulations, we can generate the hazard curves for selected site locations and hazard maps for the study region. In addition, the OpenQuake can use the physics-based ground motion models/prediction equations, established with the ground shaking database from fully dynamic rupture simulations. And the Cybershake, which is based on the kinematic simulations, to perform the PSHA and complement the fully physics-based PSHA.
Tested architectures	SuperMUC-NG at LRZ
Target TRL	6-7 View TRL chart
Relevant stakeholders	Civil protection agencies, Insurance companies, etc.
Achievements up to M41	Fault models built up in both north and south Iceland Fully non-linear dynamic simulations accounting for 3-D velocity structures, topography, off-fault plasticity, and model parameter uncertainties and achieved target resolution Cybershake implemented successfully and a demo run for south Iceland Generate the rupture probability using SHERIFS GMPEs based hazard curves and maps with OpenQuake About the code SeisSol: Extended YATeTo DSL to generate GPU GEMM kernels Developed a python library as a GEMM backend for YATeTo Adapted both SeisSol and YATeTO for batched computations Implemented Elastic Solver: time, local, neighbour integrals Both GTS and LTS scheme are working Enabled a distributed Multi-GPU setup Implemented Plasticity kernel (needs to get updated) Tested performance on a multi-GPU distributed cluster: M100 Merged first stage from experimental to the production code
Related work and further information	More detailed information about the software for this Pilot can be accessed using the links above. Some papers or presentations related to this Pilot are available at: ChEESE related publications

Exascale outlook

Exascale target (capability/capacity)

HPC enables the fully physics-based PSHA in this Pilot. It improves our capability to run large and complex simulations with high resolution, and the capacity for hundreds to thousands of simulations for probabilistic hazard analysis and reduce the uncertainties. The parallel strategy expedites the simulations and enables us to get the results in achievable time. The Multi-GUP setup can further speed up simulations and extend the study region to larger domains.

Main benefits expected from large increase in compute resources

There are several main benefits from increasing the computational resources:

To run faster simulations. With more computational resources, we can divide the jobs into more and smaller subdomains, which allows much faster computation for the same quality results.
To run more simulations. Less time for each single simulation means we are able to run more simulations in the same amount of time. Thus we can explore the epistemic uncertainties more completely and generate more stable results.
To run simulations with higher resolution or for larger domains. More computational resources also enable larger memory that allows us to run simulations with smaller meshes to improve the resolution and resolve higher frequencies, which is important for engineering risk analysis. In addition, it also enables us to apply this Pilot for larger regions.

Main challenges in harnessing large compute resources

The scalability of the GPU version is under improvement.

Another challenge is the storage issue, since the probabilistic seismic analysis requires hundreds to thousands of simulations. And the full output set for one simulation can be tens, hundreds of GB or even larger, depending on the resolution and domain size.

Specific actions towards exascale at present time

Multi-GPU setup for the SeisSol and test run.

Computational details

	Number of cores / GPUs:	Memory (GB):	Storage (GB) both temporal and permanent:	#files written both temporal and permanent	I/O data traffic per hour during job
Minimum:	48 cores/node 16 nodes 768 cores	90/node 1440	0.2 - 20	1e0-1e2	~40 GB
Average:
Maximum:	48 cores/node 24 nodes 1152 cores	90/node 2160	0.2 - 100	1e0-1e2	~50 GB