Add offline route handle generation scripts

[minghangli-uni]

Background

For the global ACCESS-OM3 configs, the init phase can take O(hours) when the mediator core count goes up, mainly due to computing interpolation weights. This is seen in both:

• 25km config
• 8km config

But this doesnt happen for the regional Pan-Antarctic configs such as 4km regional scaling study.

Goal

To avoid repeatedly recomputing these weights, the new workflow should use offline interpolation weight generation.

Why this happens

ESMF provides a RouteHandle object store precomputed mappings (weights, indices, MPI exchange patterns, etc.). But creating them is expensive - especially patch method + high res grids. ESMF docs also say it's better to write them to file and reuse them rather than regenerate each run.

So the plan is:

• generate RouteHandles offline and save them to files
• update mediator logic to load these files instead of recomputing
• test across the major configs (global + regional)

I'll open a companion PR for the first step.

[DeniseWorthen]

I saw your issue on the CMEPS repo and ended up here. I work at NOAA also we use CMEPS in our UFS-Weather-Model -- completely a different infrastructure, but we share CDEPS, CMEPS, and the MOM6 and CICE6 caps.

We recently ran into the same sorts of issues for the new operational model for UFS (GFSv17) in RH initialization time and also thought that mapfiles would be a solution. Just a point of clarification, after talking w/ the ESMF team about the issue. Mapfiles can provide the mapping weights, but the communication pattern still needs to be established at runtime.

So my understanding is that the fastest solution is to pre-generate the entire RH (via a call to ESMF_RouteHandleWrite); the limitation there is that they are only good for the exact same layouts. Mapfiles that you can provide for CMEPS contain just the weights themselves so the communication pattern still has to be done at runtime, but the advantage is that mapfiles should be layout independent.

We ended up not using pre-generated/stored RHs for GFSv17 because it had an odd side-effect in CMEPS which the ESMF team is looking into right now. We found that the post-ice and post-ocn phases of CMEPS were measurably slower; which is puzzling to say the least.

In any case, right now, I'm working on a very similar issue---to generate the mapfiles for the CDEPS-DATM + 1/12deg MOM/CICE. I'm just starting to test out the impact and/or limitations of the mapfiles for this configuration, so perhaps we should keep aware of each other's efforts?

[minghangli-uni]

Thanks @DeniseWorthen for your quick respond!

Just a point of clarification, after talking w/ the ESMF team about the issue. Mapfiles can provide the mapping weights, but the communication pattern still needs to be established at runtime.

Just to confirm - does this mean that using mapfiles alone cannot fundamentally eliminate the initialisation cost?

We ended up not using pre-generated/stored RHs for GFSv17 because it had an odd side-effect in CMEPS which the ESMF team is looking into right now. We found that the post-ice and post-ocn phases of CMEPS were measurably slower; which is puzzling to say the least.

Is there an existing issue open for this so I can follow the investigation?

In any case, right now, I'm working on a very similar issue---to generate the mapfiles for the CDEPS-DATM + 1/12deg MOM/CICE.

That part seems straightforward to me. As I mentioned in ESCOMP/CMEPS#615, this is what I've done so far to generate the mapfiles:

# ATM -> OCN
ESMF_RegridWeightGen -s $NO_MASK_ESMFMESH -d $ESMF_MESH --method bilinear --src_loc corner --dst_loc corner --ignore_unmapped --ignore_degenerate --netcdf4 --weight a2o_bilnr_$RESOLUTION.nc
ESMF_RegridWeightGen -s $NO_MASK_ESMFMESH -d $ESMF_MESH --method patch --src_loc corner --dst_loc corner --ignore_unmapped --ignore_degenerate --netcdf4 --weight a2o_patch_$RESOLUTION.nc
ESMF_RegridWeightGen -s $NO_MASK_ESMFMESH -d $ESMF_MESH --method conserve --norm_type fracarea --ignore_unmapped --ignore_degenerate --netcdf4 --weight a2o_consf_$RESOLUTION.nc

Feeding these back into the model gives bfb repro with the online calculation for bilinear and conservative, but not for patch or patch_uv3d.

so perhaps we should keep aware of each other's efforts?

Absolutely. I've listed details to the discussion thread at ESCOMP/CMEPS#615. Let me know how your work progresses - we can keep each other updated.

[DeniseWorthen]

@minghangli-uni It is my understanding that yes, the mapfiles only provide the weights, but the RH itself still gets generated at runtime. I believe that makes sense given the code also, since you're still doing the ESMF_FieldSMMStore if the mapfile is not unset.

https://github.com/ESCOMP/CMEPS/blob/6539501ed23482882f4c08f7a23c18d76d21f3e9/mediator/med_map_mod.F90#L461-L468

So generating/storing/reusing the RHs is the fastest option for initialization. If there is wider interest in having the RH save option in CMEPS, we could certainly go ahead and make a PR for that capability, if you haven't already implemented it as a local feature also.

Regarding our odd slow-down issue, I believe there is an internal ESMF ticket for this, but I don't have access to those tickets. It's not clear how much it might be an ESMF issue vs a CMEPS issue.

One question about how you're generating the mapfiles though. You're using corners as the source/destination locations. Is there a reason why? Meshes in CMEPS are, I believe, all set as ESMF_MESHLOC_ELEMENT, so we're using the center points to generate the mapfiles.

Finally, in my testing for our DATM configuration, running w/ the mapfiles vs letting CMEPS generate the RHs at runtime was B4B.

[minghangli-uni]

One question about how you're generating the mapfiles though. You're using corners as the source/destination locations. Is there a reason why? Meshes in CMEPS are, I believe, all set as ESMF_MESHLOC_ELEMENT, so we're using the center points to generate the mapfiles.

@DeniseWorthen Good catch! The snippet above was from an earlier test version. In the actual workflow we also use cell centers instead of corners.

Finally, in my testing for our DATM configuration, running w/ the mapfiles vs letting CMEPS generate the RHs at runtime was B4B.

That’s encouraging! A quick one, for vector remapping are you also using mappatch_uv3d? In our model, scalar bilinear and conservative mappings are bfb, but the vector patch remapping (mappatch_uv3d) has been the most challenging part to reproduce offline so I'm curious how it behaves in your configuration.

---

We had an internal discussion today and agreed:

Rather than attempting full offline routehandle generation, we will adopt a workflow where routehandles are generated online once, written to disk, and then reused in production runs. The reasons:

• It guarantees bfb repro, since the RHs come directly from CMEPS internal routines.
• It avoids the complexity and fragility of re-implementing CMEPS vector mapping logic offline, especially mappatch_uv3d.
• In the production environment, processor layouts rarely change so reusing pre-generated routehandles is both practical and robust.

This workflow has now been validated on a global 8 km MOM6–CICE6 coupled with DATM and DROF configuration using 232 mediator pets.

Below shows some performance results,

# Online generation (CMEPS computes RH at runtime)

Region                                                                                  PETs   PEs    Count    Mean (s)    Min (s)     Min PET Max (s)     Max PET
    [ensemble] Init 1                                                                   4680   4680   1        1554.3685   1554.3446   4669    1554.4459   2413   
      [ESM0001] IPDv02p5                                                                4680   4680   1        1300.4372   1300.4352   4276    1300.4381   450    
        [MED] IPDv03p7                                                                  232    232    2        1300.3079   1300.2776   118     1300.3553   0      
          MED: (med_map_mod: RouteHandles_init)                                         232    232    1        1299.5334   1299.4998   115     1299.5721   22     

# Using pre-written routehandles (online-generated, saved, reused)

Region                                                                                  PETs   PEs    Count    Mean (s)    Min (s)     Min PET Max (s)     Max PET
    [ensemble] Init 1                                                                   4680   4680   1        319.6602    319.6336    52      319.7458    4224   
      [ESM0001] IPDv02p5                                                                4680   4680   1        72.3847     72.3834     1397    72.3864     3977   
        [MED] IPDv03p7                                                                  232    232    2        72.2536     72.2247     140     72.2865     0      
          MED: (med_map_mod: RouteHandles_init)                                         232    232    1        71.3844     71.3418     213     71.4269     97     

So you can see the routehandle generation time drops dramatically from 1300 seconds to 70 seconds (18x speedup) w/ bfb identical results.

We found that the post-ice and post-ocn phases of CMEPS were measurably slower; which is puzzling to say the least.

For the post-ocn and post-ice phases, it appears that supplying RH files does not degrade runtime mapping efficiency.

# Online generation (CMEPS computes RH at runtime)

Region                                                                                  PETs   PEs    Count    Mean (s)    Min (s)     Min PET Max (s)     Max PET
          MED:(med_phases_post_ocn)                                                     232    232    1        0.0939      0.0506      156     0.1185      231    
            MED:(med_phases_post_ocn) map_ocn2ice                                       232    232    1        0.0939      0.0506      156     0.1184      231    
              MED: (med_map_mod:med_map_field_packed)                                   232    232    1        0.0939      0.0506      156     0.1184      231    
                MED: (med_map_mod:med_map_field_packed) map                             232    232    1        0.0787      0.0399      156     0.1100      180    
                MED: (med_map_mod:med_map_field_packed) copy from src                   232    232    1        0.0099      0.0023      180     0.0141      104    
                MED: (med_map_mod:med_map_field_packed) copy to dest                    232    232    1        0.0038      0.0024      119     0.0098      0      

          MED:(med_phases_post_ice)                                                     232    232    1        0.1016      0.0717      180     0.1227      0      
            MED:(med_phases_post_ice) map_ice2ocn                                       232    232    1        0.0894      0.0610      180     0.1109      0      
              MED: (med_map_mod:med_map_field_packed)                                   232    232    1        0.0894      0.0610      180     0.1109      0      
                MED: (med_map_mod:med_map_field_packed) map                             232    232    1        0.0491      0.0298      180     0.0573      221    
                MED: (med_map_mod:med_map_field_packed) copy from src                   232    232    1        0.0241      0.0186      231     0.0354      104    
                MED: (med_map_mod:med_map_field_packed) copy to dest                    232    232    1        0.0134      0.0042      134     0.0274      0      

# Offline-loaded RH (pre-generated)

Region                                                                                  PETs   PEs    Count    Mean (s)    Min (s)     Min PET Max (s)     Max PET
          MED:(med_phases_post_ocn)                                                     232    232    1        0.1723      0.1289      140     0.2061      69     
            MED:(med_phases_post_ocn) map_ocn2ice                                       232    232    1        0.1723      0.1289      140     0.2061      69     
              MED: (med_map_mod:med_map_field_packed)                                   232    232    1        0.1723      0.1288      140     0.2061      69     
                MED: (med_map_mod:med_map_field_packed) map                             232    232    1        0.1562      0.1164      138     0.1945      103    
                MED: (med_map_mod:med_map_field_packed) copy from src                   232    232    1        0.0096      0.0024      180     0.0164      15     
                MED: (med_map_mod:med_map_field_packed) copy to dest                    232    232    1        0.0052      0.0017      183     0.0097      29     
          MED:(med_phases_post_ice)                                                     232    232    1        0.1091      0.0817      68      0.1250      201    
            MED:(med_phases_post_ice) map_ice2ocn                                       232    232    1        0.0985      0.0745      102     0.1139      201    
              MED: (med_map_mod:med_map_field_packed)                                   232    232    1        0.0985      0.0745      102     0.1139      201    
                MED: (med_map_mod:med_map_field_packed) map                             232    232    1        0.0593      0.0431      103     0.0718      229    
                MED: (med_map_mod:med_map_field_packed) copy from src                   232    232    1        0.0243      0.0186      228     0.0350      104    
                MED: (med_map_mod:med_map_field_packed) copy to dest                    232    232    1        0.0122      0.0032      191     0.0212      0      

We’ll continue testing this workflow as we scale to higher med core counts. Based on our current experience, the routehandle generation cost can increase dramatically reaching O(hours) at large PET counts.

[DeniseWorthen]

@minghangli-uni Thanks for the update.

The mappatch "uv3d" variant RH should be identical to the "normal" mappatch RH. It is a separate RH because of Issue ESCOMP/CMEPS#552.

We do not use mapuv3d for the DATM since for us, these fields come from an ATM forecast model (cubed-sphere domain) which does the vector remapping when it creates the output (gaussian) history files. Do you know for sure that you need the uv3d mapping when using your DATM?

Originally, when we looked at doing this for the operational GFSv17, I had thought the patch RH generation might be relatively expensive, time-wise. But it turned out to be not much more time-intensive than bilinear. The real cost for us was the generation of some nstod mappings, which are required because we have a very large unstructured WW3 mesh also.

For your timing results, it looks like this is a single timestep ? (count=1). We were testing it in a 16d operational forecast run. Our ATM timestep is 150s, so the additional cost of especially the post_ice phase cumulatively meant that we almost eliminated the initialization time reduction.

For the GFSv17, we're also using much larger CMEPS PET counts O(1000-1500). Our DataInitialize was never in the O(hours). I would need to dig through some old data, but when we were generating them at runtime, it was ~O(minutes).

[DeniseWorthen]

I managed to find this example for some testing we did using an ~2day run. I've plotted the "mean" PE time from the ESMF_ProfileSummary for various post phases (un-weighted by "count"). This figure shows the 'base' case (ran each only 1x) where CMEPS generates the RHs at runtime and two repeats of using CMEPS pre-generated RHs.

(CMEPS and ATM share tasks in our setup, and we run ATM w/ 4-threads).

This is a screenshot (sorry, I don't know how to share a google sheet w/ you) that was sent to the ESMF team, using a 36 hour run. This is from a colleague's testing, not mine. I'm not sure what his CMEPS tasking was, but I think it was 4800 (4-threads).

[minghangli-uni]

The mappatch "uv3d" variant RH should be identical to the "normal" mappatch RH.

I remember testing the normla patch mapping (mapuv_with_cart3d = .false.), and the results were not identical to those obtained with the uv3d mapping.

Do you know for sure that you need the uv3d mapping when using your DATM?

Ha, I’m not sure - we inherited this setting from an older CESM config.

For your timing results, it looks like this is a single timestep ? (count=1).

Sorry i thought you were referring to the initialisation phase. For the run phase (a 1-day run, with each case ran 1x), the timings do increase when using offline-loaded RHs, but not as significantly as what you observed in your GFSv17.

Phase	Online generation (s)	Offline-loaded RH (s)	Increase (%)
med_phases_post_ocn	13.2973	15.0633	13.28%
med_phases_post_ice	17.6836	18.0515	2.08%

[DeniseWorthen]

The mappatch "uv3d" variant RH should be identical to the "normal" mappatch RH.

I remember testing the normla patch mapping (mapuv_with_cart3d = .false.), and the results were not identical to those obtained with the uv3d mapping.

Sure, with and without uv3d mapping will be different. I just meant that the RHs themselves should be identical. They're created the same way, but they're used differently.

EDIT: Thanks for the timings! I'd like to share this nugget with the ESMF team, since it comes from a completely different setup and Infrastructure (UFS vs CESM).