Notebook

Run it live: Open in Binder

Download this notebook: 1a_basics.ipynb

Running a Hydrometeorological Ensemble Verification Pipeline with veriflow#

Forecast verification helps us evaluate the quality and behaviour of forecasts. This can include comparing forecasts with observations, assessing skill across lead times, examining forecast uncertainty, or checking whether forecast distributions are realistic under different hydrological conditions.

veriflow is a Python-based framework for running forecast verification workflows in a consistent and reproducible way. It uses a configuration-driven approach to load data, define verification experiments, compute verification metrics and statistical criteria, and return results. It can be used to verify both deterministic and probabilistic ensemble forecasts.

1. What this notebook does#

This notebook focuses on the workflow, with an initial visualization of verification results:

  1. Define a verification configuration

  2. Run the veriflow pipeline

  3. Inspect the returned results

  4. Explore the available data and diagnostics

  5. Visualize the results with a small set of reusable plotting functions

The aim of this notebook is to demonstrate usage of veriflow, not to provide a complete overview of forecast verification theory.


2. Dataset and case study#

This example uses hydrometeorological data for the River Rhine basin. We focus on discharge and compare three discharge forecast datasets. Each dataset consists of a 5-member ensemble reforecast. The discharge forecasts were generated by forcing the HBV hydrological model with raw or post-processed ECMWF ensemble reforecasts of precipitation and temperature.

The forecast datasets are:

  • ``raw_raw``: baseline streamflow forecast driven by raw meteorological ensemble forecasts

  • ``lin_log``: streamflow forecast driven by meteorological ensembles post-processed using a linear-log transformation

  • ``qqt_qqt``: streamflow forecast driven by meteorological ensembles post-processed using a quantile-to-quantile transformation

In this notebook, all forecast datasets are evaluated against the same observation dataset. We focus on a key location, Lobith.

The data are read directly from a remote Zarr store, so no local download is required. For more information, see the References section.

3. Imports and environment setup#

We start by importing the required Python libraries and veriflow components.

The imports include:

  • Required Python tools

  • veriflow components, including configuration objects, Zarr datasource for reading the example data remotely, and pipeline runner

  • A local app for interactive visualization

[1]:
# Add automatic reloading of modules in case of changes
%load_ext autoreload
%autoreload 2

from datetime import datetime, timezone
import logging
import numpy as np
import pandas as pd
import xarray as xr
import matplotlib.pyplot as plt
from pathlib import Path
from veriflow.configuration import Config, GeneralInfoConfig
from veriflow.configuration.utils import (
    LeadTimes,
    Range,
    TimeUnits,
    VerificationPair,
    VerificationPeriod,
)
from veriflow.constants import DataType
from veriflow.datasources import ZarrConfig
from veriflow.pipeline import run_pipeline
from veriflow.scores import CrpsForEnsembleConfig, RankHistogramConfig

import sys

sys.path.append(str(Path("..").resolve()))
from pair_plots import find_crps_variable, forecast_scroller
from tree_plots import (
    plot_score_vs_lead_time,
    get_pair_dataset,
    rank_histogram_3d_plot,
    rank_histogram_plot,
    scatter_plot,
)
Exception in thread Thread-5 (_readerthread):
Traceback (most recent call last):
  File "C:\Users\beunk\AppData\Roaming\uv\python\cpython-3.12.11-windows-x86_64-none\Lib\threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "C:\Users\beunk\AppData\Roaming\uv\python\cpython-3.12.11-windows-x86_64-none\Lib\threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\beunk\AppData\Roaming\uv\python\cpython-3.12.11-windows-x86_64-none\Lib\subprocess.py", line 1599, in _readerthread
    buffer.append(fh.read())
                  ^^^^^^^^^
  File "C:\Users\beunk\AppData\Roaming\uv\python\cpython-3.12.11-windows-x86_64-none\Lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 28536: character maps to <undefined>

4. Defining the verification configuration#

veriflow is a configuration-driven workflow. The configuration tells the pipeline:

  • which data sources to read

  • which forecasts and observations to compare

  • which lead times to evaluate

  • which verification metrics and statistical criteria to compute

We define these parts below and then run the full workflow with a single pipeline call.

4.1. General verification settings#

The general settings define the structure of the verification experiment:

  • the verification period

  • the forecast lead times

  • the verification pairs

Each verification pair represents a single comparison between a forecast dataset and the observations.

[2]:
general = GeneralInfoConfig(
    # Evaluate forecasts issued during this period.
    verification_period=VerificationPeriod(
        start=datetime(1988, 1, 1, tzinfo=timezone.utc),
        end=datetime(2008, 12, 31, tzinfo=timezone.utc),
        dimension="forecast_reference_time",
    ),
    # Define which forecast datasets are compared against observations.
    verification_pairs=[
        VerificationPair(id="raw_raw", observations_source_id="obs", simulations_source_id="raw_raw", variable="Q"),
        VerificationPair(id="lin_log", observations_source_id="obs", simulations_source_id="lin_log", variable="Q"),
        VerificationPair(id="qqt_qqt", observations_source_id="obs", simulations_source_id="qqt_qqt", variable="Q"),
    ],
    # Evaluate lead times from 0 to 10 days.
    lead_times=LeadTimes(
        unit=TimeUnits.day,
        values=Range(start=0, end=10, step=1),
    ),
)

4.2. Data sources#

The example data is stored in remote Zarr datasets. Zarr is well suited for cloud-based, multidimensional data access.

Here we define one observation source and three forecast sources. The source names must match the names used in the verification pairs above.

[3]:
datasources = [
    ZarrConfig(
        general=general,
        import_adapter="zarr",
        source_id="obs",
        data_type=DataType.observed_historical,
        consolidated=True,
        path="s3://deltares-verification-assets/rhine_dataset_verkade_2013/optimized/obs_Q.zarr",
    ),
    ZarrConfig(
        general=general,
        import_adapter="zarr",
        source_id="raw_raw",
        data_type=DataType.simulated_forecast_ensemble,
        consolidated=True,
        path="s3://deltares-verification-assets/rhine_dataset_verkade_2013/optimized/case-raw-raw_Q.zarr",
    ),
    ZarrConfig(
        general=general,
        import_adapter="zarr",
        source_id="lin_log",
        data_type=DataType.simulated_forecast_ensemble,
        consolidated=True,
        path="s3://deltares-verification-assets/rhine_dataset_verkade_2013/optimized/case-lin-log_Q.zarr",
    ),
    ZarrConfig(
        general=general,
        import_adapter="zarr",
        source_id="qqt_qqt",
        data_type=DataType.simulated_forecast_ensemble,
        consolidated=True,
        path="s3://deltares-verification-assets/rhine_dataset_verkade_2013/optimized/case-qqt-qqt_Q.zarr",
    ),
]

4.3. Verification metrics and statistical criteria#

In this notebook, we compute two commonly used criteria for ensemble forecasts:

Continuous Ranked Probability Score (CRPS) assesses how well an ensemble forecast matches the observed outcome.

  • A CRPS of 0 indicates a perfect forecast.

  • Lower values indicate better probabilistic forecast quality, indicating the ensemble is centred around the observation and does not unnecessarily spread out.

  • CRPS increases when the forecast distribution is biased, too wide, or too narrow relative to the observation.

Rank Histogram assesses statistical reliability by showing where observations fall within the ensemble distribution.

  • A roughly uniform histogram indicates a well-calibrated ensemble.

  • Systematic patterns can reveal biases or issues with ensemble spread.

The plots later in this notebook can be used to explore both the resulting CRPS and rank histogram.

[4]:
scores = [
    # CRPS
    CrpsForEnsembleConfig(
        score_adapter="crps_for_ensemble",
        general=general,
        reduce_dims=[],
    ),
    # Rank histogram
    RankHistogramConfig(
        score_adapter="rank_histogram",
        general=general,
        reduce_dims=["forecast_reference_time"],
    ),
]

config = Config(
    fileversion="0.1.0",
    general=general,
    datasources=datasources,
    scores=scores,
)

5. Running the verification pipeline#

With the configuration in place, we can run the veriflow pipeline, which combines the following steps into a workflow:

  1. Load the input datasets

  2. Pair forecasts and observations

  3. Compute the requested verification metrics and statistical criteria

  4. Return everything in an output DataTree

[5]:
import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(name)s: %(message)s",
    handlers=[logging.StreamHandler()],
)

tree = run_pipeline(config)
2026-09-17 16:32:29,701 INFO veriflow.pipeline: Successfully initialized the configuration.
         verification_period_start = 1988-01-01 00:00:00
         verification_period_end = 2008-12-31 00:00:00
2026-09-17 16:32:29,709 INFO veriflow.datasources.base: Starting dataset fetch (source_id=obs) from Zarr.
2026-09-17 16:32:32,287 INFO veriflow.pipeline: Dataset (source_id=obs) successfully loaded and validated.
2026-09-17 16:32:32,289 INFO veriflow.datasources.base: Starting dataset fetch (source_id=raw_raw) from Zarr.
2026-09-17 16:32:34,011 INFO veriflow.pipeline: Dataset (source_id=raw_raw) successfully loaded and validated.
2026-09-17 16:32:34,013 INFO veriflow.datasources.base: Starting dataset fetch (source_id=lin_log) from Zarr.
2026-09-17 16:32:36,202 INFO veriflow.pipeline: Dataset (source_id=lin_log) successfully loaded and validated.
2026-09-17 16:32:36,204 INFO veriflow.datasources.base: Starting dataset fetch (source_id=qqt_qqt) from Zarr.
2026-09-17 16:32:38,285 INFO veriflow.pipeline: Dataset (source_id=qqt_qqt) successfully loaded and validated.
2026-09-17 16:32:38,295 INFO veriflow.pipeline: Successfully loaded all data from sources.
2026-09-17 16:32:39,564 INFO veriflow.pipeline: Successfully computed CrpsForEnsemble for verification pair raw_raw.
2026-09-17 16:32:40,169 INFO veriflow.pipeline: Successfully computed CrpsForEnsemble for verification pair lin_log.
2026-09-17 16:32:40,575 INFO veriflow.pipeline: Successfully computed CrpsForEnsemble for verification pair qqt_qqt.
2026-09-17 16:32:40,814 INFO veriflow.pipeline: Successfully computed RankHistogram for verification pair raw_raw.
2026-09-17 16:32:41,041 INFO veriflow.pipeline: Successfully computed RankHistogram for verification pair lin_log.
2026-09-17 16:32:41,249 INFO veriflow.pipeline: Successfully computed RankHistogram for verification pair qqt_qqt.
2026-09-17 16:32:41,250 INFO veriflow.pipeline: Verification pipeline completed successfully.

6. Inspecting the output DataTree#

The pipeline returns an ``xr.DataTree``, which provides access to the verification results via the .veriflow accessor. Results are organized by verification pair, where each pair defines a comparison between an observation dataset and a forecast dataset (for example, raw_raw versus observations). Each verification pair contains the input data together with the computed verification metrics.

We start by inspecting the output structure and then extract the results for a single verification pair.

[6]:
tree
[6]:
<xarray.DataTree 'veriflow-datatree'>
Group: /
├── Group: /input_data
│   ├── Group: /input_data/obs
│   │       Dimensions:  (time: 7306, station: 88)
│   │       Coordinates:
│   │         * time     (time) datetime64[ns] 58kB 1988-01-01 1988-01-02 ... 2008-01-01
│   │         * station  (station) <U11 4kB 'H-RN-0001' 'H-RN-0024' ... 'H-RN-WURZ'
│   │           lat      (station) float64 704B dask.array<chunksize=(88,), meta=np.ndarray>
│   │           lon      (station) float64 704B dask.array<chunksize=(88,), meta=np.ndarray>
│   │       Data variables:
│   │           Q        (time, station) float32 3MB dask.array<chunksize=(7306, 88), meta=np.ndarray>
│   │       Attributes:
│   │           data_type:     observed_historical
│   │           source_id:     obs
│   │           spatial_type:  point
│   │           crs:           EPSG:4326
│   ├── Group: /input_data/raw_raw
│   │       Dimensions:                  (forecast_reference_time: 2924, lead_time: 10,
│   │                                     station: 85, realization: 5)
│   │       Coordinates:
│   │         * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
│   │         * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
│   │           time                     (forecast_reference_time, lead_time) datetime64[ns] 234kB dask.array<chunksize=(2924, 10), meta=np.ndarray>
│   │         * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
│   │           lat                      (station) float64 680B dask.array<chunksize=(14,), meta=np.ndarray>
│   │           lon                      (station) float64 680B dask.array<chunksize=(14,), meta=np.ndarray>
│   │         * realization              (realization) int64 40B 0 1 2 3 4
│   │       Data variables:
│   │           Q                        (forecast_reference_time, lead_time, station, realization) float32 50MB dask.array<chunksize=(2924, 10, 14, 5), meta=np.ndarray>
│   │       Attributes:
│   │           data_type:     simulated_forecast_ensemble
│   │           source_id:     raw_raw
│   │           spatial_type:  point
│   │           crs:           EPSG:4326
│   ├── Group: /input_data/lin_log
│   │       Dimensions:                  (forecast_reference_time: 2919, lead_time: 10,
│   │                                     station: 85, realization: 5)
│   │       Coordinates:
│   │         * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
│   │         * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
│   │           time                     (forecast_reference_time, lead_time) datetime64[ns] 234kB dask.array<chunksize=(2919, 10), meta=np.ndarray>
│   │         * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
│   │           lat                      (station) float64 680B dask.array<chunksize=(15,), meta=np.ndarray>
│   │           lon                      (station) float64 680B dask.array<chunksize=(15,), meta=np.ndarray>
│   │         * realization              (realization) int64 40B 0 1 2 3 4
│   │       Data variables:
│   │           Q                        (forecast_reference_time, lead_time, station, realization) float32 50MB dask.array<chunksize=(2919, 10, 15, 5), meta=np.ndarray>
│   │       Attributes:
│   │           data_type:     simulated_forecast_ensemble
│   │           source_id:     lin_log
│   │           spatial_type:  point
│   │           crs:           EPSG:4326
│   └── Group: /input_data/qqt_qqt
│           Dimensions:                  (forecast_reference_time: 2919, lead_time: 10,
│                                         station: 85, realization: 5)
│           Coordinates:
│             * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
│             * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
│               time                     (forecast_reference_time, lead_time) datetime64[ns] 234kB dask.array<chunksize=(2919, 10), meta=np.ndarray>
│             * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
│               lat                      (station) float64 680B dask.array<chunksize=(15,), meta=np.ndarray>
│               lon                      (station) float64 680B dask.array<chunksize=(15,), meta=np.ndarray>
│             * realization              (realization) int64 40B 0 1 2 3 4
│           Data variables:
│               Q                        (forecast_reference_time, lead_time, station, realization) float32 50MB dask.array<chunksize=(2919, 10, 15, 5), meta=np.ndarray>
│           Attributes:
│               data_type:     simulated_forecast_ensemble
│               source_id:     qqt_qqt
│               spatial_type:  point
│               crs:           EPSG:4326
├── Group: /raw_raw
│   ├── Group: /raw_raw/aligned_input
│   │   ├── Group: /raw_raw/aligned_input/observations
│   │   │       Dimensions:                  (forecast_reference_time: 2924, lead_time: 10,
│   │   │                                     station: 88)
│   │   │       Coordinates:
│   │   │         * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
│   │   │         * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
│   │   │         * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
│   │   │           lat                      (station) float64 704B dask.array<chunksize=(88,), meta=np.ndarray>
│   │   │           lon                      (station) float64 704B dask.array<chunksize=(88,), meta=np.ndarray>
│   │   │       Data variables:
│   │   │           Q                        (station, forecast_reference_time, lead_time) float32 10MB dask.array<chunksize=(88, 730, 10), meta=np.ndarray>
│   │   └── Group: /raw_raw/aligned_input/simulations
│   │           Dimensions:                  (forecast_reference_time: 2924, station: 85,
│   │                                         lead_time: 10, realization: 5)
│   │           Coordinates:
│   │             * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
│   │             * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
│   │               lat                      (station) float64 680B dask.array<chunksize=(14,), meta=np.ndarray>
│   │               lon                      (station) float64 680B dask.array<chunksize=(14,), meta=np.ndarray>
│   │             * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
│   │               time                     (forecast_reference_time, lead_time) datetime64[ns] 234kB dask.array<chunksize=(2924, 10), meta=np.ndarray>
│   │             * realization              (realization) int64 40B 0 1 2 3 4
│   │           Data variables:
│   │               Q                        (forecast_reference_time, lead_time, station, realization) float32 50MB dask.array<chunksize=(2924, 10, 14, 5), meta=np.ndarray>
│   └── Group: /raw_raw/output
│       ├── Group: /raw_raw/output/crps_for_ensemble
│       │       Dimensions:                  (forecast_reference_time: 2924, lead_time: 10,
│       │                                     station: 85)
│       │       Coordinates:
│       │         * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
│       │         * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
│       │           time                     (forecast_reference_time, lead_time) datetime64[ns] 234kB dask.array<chunksize=(2924, 10), meta=np.ndarray>
│       │         * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
│       │           lat                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
│       │           lon                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
│       │       Data variables:
│       │           crps_for_ensemble        (forecast_reference_time, lead_time, station) float64 20MB dask.array<chunksize=(730, 10, 14), meta=np.ndarray>
│       └── Group: /raw_raw/output/rank_histogram
│               Dimensions:         (station: 85, lead_time: 10, rank: 6)
│               Coordinates:
│                 * station         (station) <U11 4kB 'H-RN-0001' 'H-RN-0024' ... 'H-RN-WURZ'
│                 * lead_time       (lead_time) timedelta64[ns] 80B 1 days 2 days ... 10 days
│                 * rank            (rank) float64 48B 1.0 2.0 3.0 4.0 5.0 6.0
│               Data variables:
│                   histogram_rank  (station, lead_time, rank) int64 41kB dask.array<chunksize=(14, 10, 6), meta=np.ndarray>
├── Group: /lin_log
│   ├── Group: /lin_log/aligned_input
│   │   ├── Group: /lin_log/aligned_input/observations
│   │   │       Dimensions:                  (forecast_reference_time: 2919, lead_time: 10,
│   │   │                                     station: 88)
│   │   │       Coordinates:
│   │   │         * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
│   │   │         * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
│   │   │         * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
│   │   │           lat                      (station) float64 704B dask.array<chunksize=(88,), meta=np.ndarray>
│   │   │           lon                      (station) float64 704B dask.array<chunksize=(88,), meta=np.ndarray>
│   │   │       Data variables:
│   │   │           Q                        (station, forecast_reference_time, lead_time) float32 10MB dask.array<chunksize=(88, 730, 10), meta=np.ndarray>
│   │   └── Group: /lin_log/aligned_input/simulations
│   │           Dimensions:                  (forecast_reference_time: 2919, station: 85,
│   │                                         lead_time: 10, realization: 5)
│   │           Coordinates:
│   │             * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
│   │             * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
│   │               lat                      (station) float64 680B dask.array<chunksize=(15,), meta=np.ndarray>
│   │               lon                      (station) float64 680B dask.array<chunksize=(15,), meta=np.ndarray>
│   │             * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
│   │               time                     (forecast_reference_time, lead_time) datetime64[ns] 234kB dask.array<chunksize=(2919, 10), meta=np.ndarray>
│   │             * realization              (realization) int64 40B 0 1 2 3 4
│   │           Data variables:
│   │               Q                        (forecast_reference_time, lead_time, station, realization) float32 50MB dask.array<chunksize=(2919, 10, 15, 5), meta=np.ndarray>
│   └── Group: /lin_log/output
│       ├── Group: /lin_log/output/crps_for_ensemble
│       │       Dimensions:                  (forecast_reference_time: 2919, lead_time: 10,
│       │                                     station: 85)
│       │       Coordinates:
│       │         * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
│       │         * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
│       │           time                     (forecast_reference_time, lead_time) datetime64[ns] 234kB dask.array<chunksize=(2919, 10), meta=np.ndarray>
│       │         * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
│       │           lat                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
│       │           lon                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
│       │       Data variables:
│       │           crps_for_ensemble        (forecast_reference_time, lead_time, station) float64 20MB dask.array<chunksize=(730, 10, 15), meta=np.ndarray>
│       └── Group: /lin_log/output/rank_histogram
│               Dimensions:         (station: 85, lead_time: 10, rank: 6)
│               Coordinates:
│                 * station         (station) <U11 4kB 'H-RN-0001' 'H-RN-0024' ... 'H-RN-WURZ'
│                 * lead_time       (lead_time) timedelta64[ns] 80B 1 days 2 days ... 10 days
│                 * rank            (rank) float64 48B 1.0 2.0 3.0 4.0 5.0 6.0
│               Data variables:
│                   histogram_rank  (station, lead_time, rank) int64 41kB dask.array<chunksize=(15, 10, 6), meta=np.ndarray>
└── Group: /qqt_qqt
    ├── Group: /qqt_qqt/aligned_input
    │   ├── Group: /qqt_qqt/aligned_input/observations
    │   │       Dimensions:                  (forecast_reference_time: 2919, lead_time: 10,
    │   │                                     station: 88)
    │   │       Coordinates:
    │   │         * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
    │   │         * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
    │   │         * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
    │   │           lat                      (station) float64 704B dask.array<chunksize=(88,), meta=np.ndarray>
    │   │           lon                      (station) float64 704B dask.array<chunksize=(88,), meta=np.ndarray>
    │   │       Data variables:
    │   │           Q                        (station, forecast_reference_time, lead_time) float32 10MB dask.array<chunksize=(88, 730, 10), meta=np.ndarray>
    │   └── Group: /qqt_qqt/aligned_input/simulations
    │           Dimensions:                  (forecast_reference_time: 2919, station: 85,
    │                                         lead_time: 10, realization: 5)
    │           Coordinates:
    │             * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
    │             * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
    │               lat                      (station) float64 680B dask.array<chunksize=(15,), meta=np.ndarray>
    │               lon                      (station) float64 680B dask.array<chunksize=(15,), meta=np.ndarray>
    │             * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
    │               time                     (forecast_reference_time, lead_time) datetime64[ns] 234kB dask.array<chunksize=(2919, 10), meta=np.ndarray>
    │             * realization              (realization) int64 40B 0 1 2 3 4
    │           Data variables:
    │               Q                        (forecast_reference_time, lead_time, station, realization) float32 50MB dask.array<chunksize=(2919, 10, 15, 5), meta=np.ndarray>
    └── Group: /qqt_qqt/output
        ├── Group: /qqt_qqt/output/crps_for_ensemble
        │       Dimensions:                  (forecast_reference_time: 2919, lead_time: 10,
        │                                     station: 85)
        │       Coordinates:
        │         * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
        │         * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
        │           time                     (forecast_reference_time, lead_time) datetime64[ns] 234kB dask.array<chunksize=(2919, 10), meta=np.ndarray>
        │         * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
        │           lat                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
        │           lon                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
        │       Data variables:
        │           crps_for_ensemble        (forecast_reference_time, lead_time, station) float64 20MB dask.array<chunksize=(730, 10, 15), meta=np.ndarray>
        └── Group: /qqt_qqt/output/rank_histogram
                Dimensions:         (station: 85, lead_time: 10, rank: 6)
                Coordinates:
                  * station         (station) <U11 4kB 'H-RN-0001' 'H-RN-0024' ... 'H-RN-WURZ'
                  * lead_time       (lead_time) timedelta64[ns] 80B 1 days 2 days ... 10 days
                  * rank            (rank) float64 48B 1.0 2.0 3.0 4.0 5.0 6.0
                Data variables:
                    histogram_rank  (station, lead_time, rank) int64 41kB dask.array<chunksize=(15, 10, 6), meta=np.ndarray>

The output contains three verification pairs, corresponding to the three forecast datasets defined in the configuration.

We can now select one verification pair and inspect its dataset. The returned object is an xarray.Dataset containing:

  • the forecast data

  • the corresponding observations

  • the computed verification metrics and statistical criteria (in this case, CRPS and rank histogram)

  • metadata and coordinates (e.g., station, lead time, and forecast reference time)

[7]:
print(f"The size of the datatree is: {tree.nbytes / 1024**2} mb")
The size of the datatree is: 375.3161315917969 mb
[8]:
tree.load()
[8]:
<xarray.DataTree 'veriflow-datatree'>
Group: /
├── Group: /input_data
│   ├── Group: /input_data/obs
│   │       Dimensions:  (time: 7306, station: 88)
│   │       Coordinates:
│   │         * time     (time) datetime64[ns] 58kB 1988-01-01 1988-01-02 ... 2008-01-01
│   │         * station  (station) <U11 4kB 'H-RN-0001' 'H-RN-0024' ... 'H-RN-WURZ'
│   │           lat      (station) float64 704B 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0
│   │           lon      (station) float64 704B 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0
│   │       Data variables:
│   │           Q        (time, station) float32 3MB 2.091e+03 2.76 0.58 4.5 ... nan nan nan
│   │       Attributes:
│   │           data_type:     observed_historical
│   │           source_id:     obs
│   │           spatial_type:  point
│   │           crs:           EPSG:4326
│   ├── Group: /input_data/raw_raw
│   │       Dimensions:                  (forecast_reference_time: 2924, lead_time: 10,
│   │                                     station: 85, realization: 5)
│   │       Coordinates:
│   │         * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
│   │         * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
│   │           time                     (forecast_reference_time, lead_time) datetime64[ns] 234kB ...
│   │         * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
│   │           lat                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
│   │           lon                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
│   │         * realization              (realization) int64 40B 0 1 2 3 4
│   │       Data variables:
│   │           Q                        (forecast_reference_time, lead_time, station, realization) float32 50MB ...
│   │       Attributes:
│   │           data_type:     simulated_forecast_ensemble
│   │           source_id:     raw_raw
│   │           spatial_type:  point
│   │           crs:           EPSG:4326
│   ├── Group: /input_data/lin_log
│   │       Dimensions:                  (forecast_reference_time: 2919, lead_time: 10,
│   │                                     station: 85, realization: 5)
│   │       Coordinates:
│   │         * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
│   │         * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
│   │           time                     (forecast_reference_time, lead_time) datetime64[ns] 234kB ...
│   │         * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
│   │           lat                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
│   │           lon                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
│   │         * realization              (realization) int64 40B 0 1 2 3 4
│   │       Data variables:
│   │           Q                        (forecast_reference_time, lead_time, station, realization) float32 50MB ...
│   │       Attributes:
│   │           data_type:     simulated_forecast_ensemble
│   │           source_id:     lin_log
│   │           spatial_type:  point
│   │           crs:           EPSG:4326
│   └── Group: /input_data/qqt_qqt
│           Dimensions:                  (forecast_reference_time: 2919, lead_time: 10,
│                                         station: 85, realization: 5)
│           Coordinates:
│             * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
│             * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
│               time                     (forecast_reference_time, lead_time) datetime64[ns] 234kB ...
│             * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
│               lat                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
│               lon                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
│             * realization              (realization) int64 40B 0 1 2 3 4
│           Data variables:
│               Q                        (forecast_reference_time, lead_time, station, realization) float32 50MB ...
│           Attributes:
│               data_type:     simulated_forecast_ensemble
│               source_id:     qqt_qqt
│               spatial_type:  point
│               crs:           EPSG:4326
├── Group: /raw_raw
│   ├── Group: /raw_raw/aligned_input
│   │   ├── Group: /raw_raw/aligned_input/observations
│   │   │       Dimensions:                  (forecast_reference_time: 2924, lead_time: 10,
│   │   │                                     station: 88)
│   │   │       Coordinates:
│   │   │         * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
│   │   │         * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
│   │   │         * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
│   │   │           lat                      (station) float64 704B 0.0 0.0 0.0 ... 0.0 0.0 0.0
│   │   │           lon                      (station) float64 704B 0.0 0.0 0.0 ... 0.0 0.0 0.0
│   │   │       Data variables:
│   │   │           Q                        (station, forecast_reference_time, lead_time) float32 10MB ...
│   │   └── Group: /raw_raw/aligned_input/simulations
│   │           Dimensions:                  (forecast_reference_time: 2924, station: 85,
│   │                                         lead_time: 10, realization: 5)
│   │           Coordinates:
│   │             * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
│   │             * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
│   │               lat                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
│   │               lon                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
│   │             * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
│   │               time                     (forecast_reference_time, lead_time) datetime64[ns] 234kB ...
│   │             * realization              (realization) int64 40B 0 1 2 3 4
│   │           Data variables:
│   │               Q                        (forecast_reference_time, lead_time, station, realization) float32 50MB ...
│   └── Group: /raw_raw/output
│       ├── Group: /raw_raw/output/crps_for_ensemble
│       │       Dimensions:                  (forecast_reference_time: 2924, lead_time: 10,
│       │                                     station: 85)
│       │       Coordinates:
│       │         * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
│       │         * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
│       │           time                     (forecast_reference_time, lead_time) datetime64[ns] 234kB ...
│       │         * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
│       │           lat                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
│       │           lon                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
│       │       Data variables:
│       │           crps_for_ensemble        (forecast_reference_time, lead_time, station) float64 20MB ...
│       └── Group: /raw_raw/output/rank_histogram
│               Dimensions:         (station: 85, lead_time: 10, rank: 6)
│               Coordinates:
│                 * station         (station) <U11 4kB 'H-RN-0001' 'H-RN-0024' ... 'H-RN-WURZ'
│                 * lead_time       (lead_time) timedelta64[ns] 80B 1 days 2 days ... 10 days
│                 * rank            (rank) float64 48B 1.0 2.0 3.0 4.0 5.0 6.0
│               Data variables:
│                   histogram_rank  (station, lead_time, rank) int64 41kB 1117 295 ... 400 540
├── Group: /lin_log
│   ├── Group: /lin_log/aligned_input
│   │   ├── Group: /lin_log/aligned_input/observations
│   │   │       Dimensions:                  (forecast_reference_time: 2919, lead_time: 10,
│   │   │                                     station: 88)
│   │   │       Coordinates:
│   │   │         * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
│   │   │         * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
│   │   │         * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
│   │   │           lat                      (station) float64 704B 0.0 0.0 0.0 ... 0.0 0.0 0.0
│   │   │           lon                      (station) float64 704B 0.0 0.0 0.0 ... 0.0 0.0 0.0
│   │   │       Data variables:
│   │   │           Q                        (station, forecast_reference_time, lead_time) float32 10MB ...
│   │   └── Group: /lin_log/aligned_input/simulations
│   │           Dimensions:                  (forecast_reference_time: 2919, station: 85,
│   │                                         lead_time: 10, realization: 5)
│   │           Coordinates:
│   │             * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
│   │             * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
│   │               lat                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
│   │               lon                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
│   │             * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
│   │               time                     (forecast_reference_time, lead_time) datetime64[ns] 234kB ...
│   │             * realization              (realization) int64 40B 0 1 2 3 4
│   │           Data variables:
│   │               Q                        (forecast_reference_time, lead_time, station, realization) float32 50MB ...
│   └── Group: /lin_log/output
│       ├── Group: /lin_log/output/crps_for_ensemble
│       │       Dimensions:                  (forecast_reference_time: 2919, lead_time: 10,
│       │                                     station: 85)
│       │       Coordinates:
│       │         * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
│       │         * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
│       │           time                     (forecast_reference_time, lead_time) datetime64[ns] 234kB ...
│       │         * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
│       │           lat                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
│       │           lon                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
│       │       Data variables:
│       │           crps_for_ensemble        (forecast_reference_time, lead_time, station) float64 20MB ...
│       └── Group: /lin_log/output/rank_histogram
│               Dimensions:         (station: 85, lead_time: 10, rank: 6)
│               Coordinates:
│                 * station         (station) <U11 4kB 'H-RN-0001' 'H-RN-0024' ... 'H-RN-WURZ'
│                 * lead_time       (lead_time) timedelta64[ns] 80B 1 days 2 days ... 10 days
│                 * rank            (rank) float64 48B 1.0 2.0 3.0 4.0 5.0 6.0
│               Data variables:
│                   histogram_rank  (station, lead_time, rank) int64 41kB 419 561 ... 493 308
└── Group: /qqt_qqt
    ├── Group: /qqt_qqt/aligned_input
    │   ├── Group: /qqt_qqt/aligned_input/observations
    │   │       Dimensions:                  (forecast_reference_time: 2919, lead_time: 10,
    │   │                                     station: 88)
    │   │       Coordinates:
    │   │         * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
    │   │         * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
    │   │         * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
    │   │           lat                      (station) float64 704B 0.0 0.0 0.0 ... 0.0 0.0 0.0
    │   │           lon                      (station) float64 704B 0.0 0.0 0.0 ... 0.0 0.0 0.0
    │   │       Data variables:
    │   │           Q                        (station, forecast_reference_time, lead_time) float32 10MB ...
    │   └── Group: /qqt_qqt/aligned_input/simulations
    │           Dimensions:                  (forecast_reference_time: 2919, station: 85,
    │                                         lead_time: 10, realization: 5)
    │           Coordinates:
    │             * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
    │             * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
    │               lat                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
    │               lon                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
    │             * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
    │               time                     (forecast_reference_time, lead_time) datetime64[ns] 234kB ...
    │             * realization              (realization) int64 40B 0 1 2 3 4
    │           Data variables:
    │               Q                        (forecast_reference_time, lead_time, station, realization) float32 50MB ...
    └── Group: /qqt_qqt/output
        ├── Group: /qqt_qqt/output/crps_for_ensemble
        │       Dimensions:                  (forecast_reference_time: 2919, lead_time: 10,
        │                                     station: 85)
        │       Coordinates:
        │         * forecast_reference_time  (forecast_reference_time) datetime64[ns] 23kB 19...
        │         * lead_time                (lead_time) timedelta64[ns] 80B 1 days ... 10 days
        │           time                     (forecast_reference_time, lead_time) datetime64[ns] 234kB ...
        │         * station                  (station) <U11 4kB 'H-RN-0001' ... 'H-RN-WURZ'
        │           lat                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
        │           lon                      (station) float64 680B 0.0 0.0 0.0 ... 0.0 0.0 0.0
        │       Data variables:
        │           crps_for_ensemble        (forecast_reference_time, lead_time, station) float64 20MB ...
        └── Group: /qqt_qqt/output/rank_histogram
                Dimensions:         (station: 85, lead_time: 10, rank: 6)
                Coordinates:
                  * station         (station) <U11 4kB 'H-RN-0001' 'H-RN-0024' ... 'H-RN-WURZ'
                  * lead_time       (lead_time) timedelta64[ns] 80B 1 days 2 days ... 10 days
                  * rank            (rank) float64 48B 1.0 2.0 3.0 4.0 5.0 6.0
                Data variables:
                    histogram_rank  (station, lead_time, rank) int64 41kB 871 322 ... 448 554

7. Visualization of verification results#

Now that we have an overview of the observed and ensemble forecast time series and the verification results through the output ``DataTree``, we can visualize the results using a small collection of reusable plotting functions in pair_plots.py (single-dataset, single-figure builders) and tree_plots.py (whole-tree, multi-pair builders).

Each function returns a Plotly figure for a pre-sliced dataset. There is no interactive layer: selections that used to be interactive controls - verification pair, station, and lead time - are now made explicitly by slicing the xarray.Dataset before calling a plot function. This makes the plots easy to reuse in notebooks, scripts, or reports.

The available functions are:

  • scatter_plot - observation vs. simulation scatter for a single station and lead time

  • plot_score_vs_lead_time - CRPS across lead times for one or more pre-selected stations

  • rank_histogram_plot - rank histogram for a single station and lead time

  • rank_histogram_3d_plot - 3D rank histogram showing all ranks across lead times for a single station

  • forecast_scroller - scroll (or auto-play) through individual forecasts against their surrounding observations

The interpretation of these plots and diagnostics is covered in the second notebook.

[9]:
first_ds = tree.veriflow.get_outputs("raw_raw")["crps_for_ensemble"]
stations = first_ds.coords["station"].values
lead_times = first_ds.coords["lead_time"].values
print("Stations:", stations)
Stations: ['H-RN-0001' 'H-RN-0024' 'H-RN-0026' 'H-RN-0028' 'H-RN-0029' 'H-RN-0031'
 'H-RN-0036' 'H-RN-0038' 'H-RN-0039' 'H-RN-0052' 'H-RN-0053' 'H-RN-0659'
 'H-RN-0689' 'H-RN-0693' 'H-RN-0808' 'H-RN-0847' 'H-RN-0888' 'H-RN-0900'
 'H-RN-0908' 'H-RN-0913' 'H-RN-0943' 'H-RN-0947' 'H-RN-0950' 'H-RN-0957'
 'H-RN-0984' 'H-RN-1025' 'H-RN-1026' 'H-RN-1027' 'H-RN-2289' 'H-RN-BFG001'
 'H-RN-BFG002' 'H-RN-BFG003' 'H-RN-BFG005' 'H-RN-BFG007' 'H-RN-BFG008'
 'H-RN-BFG009' 'H-RN-BFG014' 'H-RN-BFG015' 'H-RN-BFG017' 'H-RN-BFG018'
 'H-RN-BFG019' 'H-RN-BFG021' 'H-RN-BFG023' 'H-RN-BFG025' 'H-RN-BFG026'
 'H-RN-BFG028' 'H-RN-BFG029' 'H-RN-BFG031' 'H-RN-BFG033' 'H-RN-BFG035'
 'H-RN-BFG036' 'H-RN-BFG037' 'H-RN-BFG039' 'H-RN-BFG040' 'H-RN-BFG041'
 'H-RN-BFG042' 'H-RN-BFG044' 'H-RN-BFG045' 'H-RN-BFG046' 'H-RN-BFG048'
 'H-RN-BFG049' 'H-RN-BFG050' 'H-RN-BFG052' 'H-RN-BFG058' 'H-RN-BFG059'
 'H-RN-BFG060' 'H-RN-BFG061' 'H-RN-BFG063' 'H-RN-BFG066' 'H-RN-BFG068'
 'H-RN-BFG069' 'H-RN-BFG071' 'H-RN-BFG072' 'H-RN-BFG073' 'H-RN-BFG074'
 'H-RN-BFG075' 'H-RN-BFG076' 'H-RN-BFG077' 'H-RN-BFG078' 'H-RN-BFG079'
 'H-RN-BOLL' 'H-RN-DIET' 'H-RN-PLOC' 'H-RN-TRIE' 'H-RN-WURZ']
[10]:
# Scatter: pre-slice to a single station and a single lead time.
scatter_plot(tree, station=stations[0], lead_time=lead_times[5]).show()
[11]:
plot_score_vs_lead_time(tree, stations=["H-RN-0001"], score_name="crps_for_ensemble").show()
[12]:
# Rank histogram: pre-slice to a single station and a single lead time.
rank_histogram_plot(tree, station="H-RN-0001", lead_time=lead_times[5]).show()
[13]:
# Advanced and abstract.
rank_histogram_3d_plot(tree, station="H-RN-0001").show()

8. Bonus: flood event animation for Dutch Rhine floods in January of 1995#

[15]:
# Get forecasts for
pair_ds = get_pair_dataset(tree, "raw_raw").sel(station=stations[0])
variable = str(pair_ds.attrs["variable"])

# Flood period
time_selection = slice("1995-01-01", "1995-02-10")
pair_ds = pair_ds.sel(forecast_reference_time=time_selection)

# Animate
forecast_scroller(pair_ds, obs_var=f"{variable}_obs", sim_var=f"{variable}_sim", play_speed=1, obs_before=24*7, obs_after=24*7, y_min=0, y_max=15000).show()

Optional: YAML-based configuration#

This notebook uses Python objects to define the configuration because it is convenient for interactive work.

veriflow can also support YAML-based configuration files, which are useful for reproducible and shareable workflows. A YAML example can be added here once the preferred release configuration format is finalized.

References#

The dataset used in the notebook was originally used in: > Verkade et al. (2013), Post-processing ECMWF precipitation and temperature ensemble reforecasts for operational hydrologic forecasting at various spatial scales, Journal of Hydrology. https://doi.org/10.1016/j.jhydrol.2013.07.039

Additional information is available via 4TU.ResearchData: