CNRS, ETH, INGV
Earthquake hazard
SPECFEM3D :
fortan compiler
c compiler
cuda
MPI
SALVUS :
c++ compiler
cuda
MPI
LASIF :
obspy
Pre-processing:
python
Visualization tools:
paraview
SPECFEM3D :
https://github.com/geodynamics/specfem3d
LASIF : https://github.com/krischer/LASIF
| Main objective / mission | The aim of the present pilot is s to image the internal structures of the earth at different scales. |
| Workflow description | The seismic tomography workflow can be seen as 3 main steps :
Step 1 consists of selecting, downloading, and preparing the data as well as the model and different inputs required for step 2. We rely on ObsPy (https://docs.obspy.org/) and LASIF python packages for seismological data processing. We have updated LASIF for both SALVUS and SPECFEM3D support and added new tools to help the user for selecting data and build input files for the computation in HPC systems. In addition, we developed a python package, PySpecfem ,that helps to manage all SPECFEM3D input files and directory structure using a unique configuration file. Step 2 is the main tomographic iterations launched in HPC systems and requires a high computing capability. We have optimized SALVUS and SPECFEM3D for those systems. In addition, we have developed within the ChESSE project a new SPECFEM3D branch for gradient computation and models update, SPCEFEM3D_FWI. Step 3 comes after the main iterations have converged towards a model that fits the data. This consists of visualizing the models obtained, storing the synthetics and data used, as well as performing quality control. This general tomographic workflow was adapted for SPECFEM3D_FWI and SALVUS. |
| PD configuration | We have tested the seismic tomography workflow on real data set from exploration geophysics (offshore North Sea). The computation was carried out on Marconi100 GPU computer. We performed a full waveform inversion on offshore real three-component seismic data set recorded by 138 ocean-bottom seismometers (OBS). The dataset is composed of ~70 000 shots per OBS and with 25 meters shot spacing that represents 1 TB of 500Hz sampled time series. The data are very high quality from 3 Hz to 50 Hz. We inverted this data set for low frequency up to 6Hz, performing 2 sets of inversion with 50 iterations each. We used 128 Marconi100 nodes (512 GPU V100) during 24h00. |
| Tested architectures | The workflow has been tested on Marconi100 (CINECA) and Piz Daint (CSCS) supercomputers. |
| Target TRL | 6 View TRL chart |
| Relevant stakeholders | CNRS, ETH, INGV |
| Achievements up to M41 | Improving seismic tomography workflow in CHEESE:
|
We tested the pilot demonstrator on 1000 GPU in Marconi100 in single run. We used nested MPI communicator to split the main communicator in 250 parallel simulations to use 250 seismic sources simulataneously.
By increasing the computational resources we can use more seismic sources, more stations and use higher freqeuncy wavefield. This leads to better resolved subsiurface images.
For the gradient computation, the memory mangement was the main challenge. We developped checkpointing strategy which allows to store sanpshots in memory and also some of them in disk when we reach the memory limit. We also use a subsampling strategy to avoid
reducing the memory demand, the number of files and avoid non necessary IOs.
| Number of cores / GPUs: | Memory (GB): | Storage (GB) both temporal and permanent: | #files written both temporal and permanent | I/O data traffic per hour during job | |
| Minimum: | 1 GPU | 5 | 50 | 1000 | ? |
| Average: | 100 GPU | 3200 | 10000 | 100000 | ? |
| Maximum: | 1000 GPU | 32000 | 100000 | 1000000 | ? |
