4. Verify Model Performance
4.1. eagle-tools Overview
We use the eagle-tools package for postprocessing model output and visualizing model performance. The eagle-tools library provides command-line utilities that are configured via YAML files.
This library currently supports:
Running anemoi-inference across many initial conditions at scale, for example over a validation set
Postprocessing inference output into a format ready for the
wxvxpackageComputing aggregated error metrics such as RMSE and MAE while preserving the initial-condition dimension
Visualizing spatial error across lead times
Computing power spectra
Visualizing predictions alongside targets through figures and GIFs
At a high level, eagle-tools enables users to analyze model performance at scale. For more information, see the eagle-tools GitHub repository.
4.2. wxvx Overview
EAGLE uses wxvx to verify model forecasts against either gridded analyses or point observations.
Within the EAGLE workflow, wxvx is responsible for running the underlying
verification tasks and producing the statistics and plots used to evaluate
forecast performance. In practice, this means you can compare both global
and LAM forecasts against:
gridded analyses with MET’s grid_stat
point observations with MET’s point_stat
Before running verification, the driver runs the prewxvx component from eagle-tools to postprocess forecast output from the previous step.
See the wxvx repository for further information about the project itself.
4.2.1. Precipitation Skill Scores
The vx-stageiv-lam target verifies the LAM model’s six-hour accumulated
precipitation field, accum_tp, as the canonical wxvx variable tp.
The default truth source is the hourly Stage IV APCP analysis distributed by
NOMADS.
For each forecast valid time, wxvx downloads the six hourly Stage IV files
ending at that time and uses MET’s pcp_combine to create the six-hour truth
accumulation. MET’s grid_stat then regrids Stage IV to the LAM forecast grid.
The default URL uses NOMADS’ recent operational Stage IV archive, so this target
is intended primarily for NRT forecasts. Data availability may lag the valid
time. For historical verification, override
vx.grid2stageiv.lam.wxvx.wxvx.truth.url with a URL or local-path template for
an archive that contains the required hourly files.
wxvx computes the categorical CSI and HSS scores and the neighborhood FSS
at precipitation thresholds of 1, 5, 10, and 25 kg m-2. FSS is computed
with circular neighborhoods 3, 5, and 11 grid points wide.
Accumulated precipitation is not included in the grid-to-observation targets because the PrepBUFR point-observation truth source does not provide the corresponding field.
4.3. wxvx Quick Tips
Use the
vx-grid-*targets to verify against gridded analyses.Use the
vx-obs-*targets to verify against PrepBUFR observations.Use
vx-stageiv-lamto verify LAM accumulated precipitation against Stage IV.The
globalandlamtargets are independent, so they can be run in parallel.Check the
run/<expname>/vx/*.logfiles if a verification job fails.
4.3.1. Running Verification
EAGLE provides the following standard verification targets:
make vx-grid-global config=eagle.yaml
make vx-grid-lam config=eagle.yaml
make vx-obs-global config=eagle.yaml
make vx-obs-lam config=eagle.yaml
make vx-stageiv-lam config=eagle.yaml # for nested runs only
These commands submit batch jobs for:
grid-global: global forecasts verified against gridded analysesgrid-lam: limited-area forecasts verified against gridded analysesobs-global: global forecasts verified against point observationsobs-lam: limited-area forecasts verified against point observationsstageiv-lam: limited-area accumulated precipitation verified against Stage IV
Because these are separate jobs, they can be launched in quick succession to run in parallel.
4.3.2. Verification Output
When a verification job completes successfully, wxvx writes output beneath
the corresponding verification run directory:
run/<expname>/vx/grid2{grid,obs}/{global,lam}/run/
The most useful outputs are:
stats/for MET.statfilesplots/for generated.pngplotsrun/<expname>/vx/*.logfor verification logs
The output for eagle-tools prewxvx can be found under: run/<expname>/vx/prewxvx/{global,lam}/runscript.prewxvx-*.out.
4.3.3. Additional Visualization
After verification is complete, you can generate additional postprocessed visualization outputs with:
make vis-grid-global config=eagle.yaml
make vis-grid-lam config=eagle.yaml
make vis-obs-global config=eagle.yaml
make vis-obs-lam config=eagle.yaml
These targets run eagle-tools postwxvx to create netCDF summary files and
additional plots under:
run/<expname>/visualization/grid2{grid,obs}/{global,lam}/