Download this notebook: 03_elbow_precipitation_ensemble.ipynb
03 - Precipitation Ensemble Forecasts#
Learning goals of this module#
Learn how to do a basic analysis, comparing observed precipitation with forecast precipitation from GEPS, GEFS and IFS.
Learn about ensemble forecast verification.
Learn how to calculate and visualize simple error metrics, such as the rank_histogram and the continuous_ranked_probability_score.
Assumptions#
We assume you are familiar with the concept of a probabilistic forecast.
Reference to ensemble forecast products#
Run imports and set-up logging#
[ ]:
# Reload automatically
%load_ext autoreload
%autoreload 2
import logging
import sys
import warnings
from pathlib import Path
from dotenv import load_dotenv
from veriflow import run_pipeline
from veriflow.constants import VERSION
# add project root (parent of notebook folder) to path
sys.path.append(str(Path("..").resolve()))
from tree_plots import (
plot_score_vs_lead_time,
forecast_timeseries_plot,
get_pair_dataset,
rank_histogram_plot,
scatter_plot,
)
load_dotenv(dotenv_path="tutorial.env", override=True)
base_config = Path("config")
base_config.exists()
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s - %(levelname)s - %(message)s",
handlers=[logging.StreamHandler()],
)
logging.info(f"Running Veriflow version {VERSION}")
Exception in thread Thread-7 (_readerthread):
Traceback (most recent call last):
File "C:\Users\beunk\AppData\Roaming\uv\python\cpython-3.12.11-windows-x86_64-none\Lib\threading.py", line 1075, in _bootstrap_inner
self.run()
File "C:\Users\beunk\AppData\Roaming\uv\python\cpython-3.12.11-windows-x86_64-none\Lib\threading.py", line 1012, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\beunk\AppData\Roaming\uv\python\cpython-3.12.11-windows-x86_64-none\Lib\subprocess.py", line 1599, in _readerthread
buffer.append(fh.read())
^^^^^^^^^
File "C:\Users\beunk\AppData\Roaming\uv\python\cpython-3.12.11-windows-x86_64-none\Lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 364510: character maps to <undefined>
2026-11-04 15:02:39,835 - INFO - Running Veriflow version 0.4.1
Inspecting the veriflow pipeline configuration#
Open the config file in the “config” directory. The name of the file is identical to the name of the notebook.
Inspect each of the sections to gain an understanding of what this configuration is about.
Running the veriflow pipeline#
[2]:
dt = run_pipeline((base_config / "03_elbow_precipitation_ensemble.yaml", "yaml"))
2026-11-04 15:02:40,093 - INFO - Successfully initialized the configuration.
verification_period_start = 2025-05-15 00:00:00
verification_period_end = 2026-06-01 00:00:00
2026-11-04 15:02:40,098 - INFO - Starting dataset fetch (source_id=observed) from FewsWebservice.
2026-11-04 15:02:40,801 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.obs&moduleInstanceIds=ImportMeteoStations&startTime=2025-05-16T00%3A00%3A00Z&endTime=2026-06-07T00%3A00%3A00Z&timeSeriesType=EXTERNAL_HISTORICAL&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:02:40,804 - INFO - Starting dataset fetch (source_id=observed) from FewsNetCDF.
2026-11-04 15:02:41,968 - INFO - Dataset (source_id=observed) successfully loaded and validated.
2026-11-04 15:02:41,968 - INFO - Starting dataset fetch (source_id=simulated_GEPS) from FewsWebservice.
2026-11-04 15:02:42,949 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEPS&startForecastTime=2025-05-15T00%3A00%3A00Z&endForecastTime=2026-06-01T00%3A00%3A00Z&ensembleId=GEPS&forecastCount=1000000&documentFormat=PI_JSON&onlyHeaders=True&omitEmptyTimeSeries=True
2026-11-04 15:02:42,954 - INFO - Configured verification period (2025-05-15 00:00:00 to 2026-06-01 00:00:00) exceeds what source 'simulated_GEPS' has available (2026-05-15 00:00:00 to 2026-06-01 00:00:00); clamping to (2026-05-15 00:00:00 to 2026-06-01 00:00:00).
2026-11-04 15:02:43,023 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEPS&startForecastTime=2026-05-15T00%3A00%3A00Z&endForecastTime=2026-06-01T00%3A00%3A00Z&ensembleId=GEPS&forecastCount=1000000&documentFormat=PI_JSON&onlyHeaders=True&omitEmptyTimeSeries=True
2026-11-04 15:02:43,080 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEPS&startForecastTime=2026-05-15T00%3A00%3A00Z&endForecastTime=2026-05-15T00%3A00%3A00Z&ensembleId=GEPS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:02:43,133 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEPS&startForecastTime=2026-05-21T00%3A00%3A00Z&endForecastTime=2026-05-21T00%3A00%3A00Z&ensembleId=GEPS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:02:43,191 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEPS&startForecastTime=2026-05-16T00%3A00%3A00Z&endForecastTime=2026-05-16T00%3A00%3A00Z&ensembleId=GEPS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:02:43,252 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEPS&startForecastTime=2026-05-18T00%3A00%3A00Z&endForecastTime=2026-05-18T00%3A00%3A00Z&ensembleId=GEPS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:02:43,308 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEPS&startForecastTime=2026-05-28T00%3A00%3A00Z&endForecastTime=2026-05-28T00%3A00%3A00Z&ensembleId=GEPS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:02:43,359 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEPS&startForecastTime=2026-05-30T00%3A00%3A00Z&endForecastTime=2026-05-30T00%3A00%3A00Z&ensembleId=GEPS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:02:43,413 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEPS&startForecastTime=2026-05-20T00%3A00%3A00Z&endForecastTime=2026-05-20T00%3A00%3A00Z&ensembleId=GEPS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:02:43,469 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEPS&startForecastTime=2026-05-29T00%3A00%3A00Z&endForecastTime=2026-05-29T00%3A00%3A00Z&ensembleId=GEPS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:02:43,479 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEPS&startForecastTime=2026-05-31T00%3A00%3A00Z&endForecastTime=2026-05-31T00%3A00%3A00Z&ensembleId=GEPS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:02:43,548 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEPS&startForecastTime=2026-05-19T00%3A00%3A00Z&endForecastTime=2026-05-19T00%3A00%3A00Z&ensembleId=GEPS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:02:43,549 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEPS&startForecastTime=2026-05-22T00%3A00%3A00Z&endForecastTime=2026-05-22T00%3A00%3A00Z&ensembleId=GEPS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:02:43,620 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEPS&startForecastTime=2026-06-01T00%3A00%3A00Z&endForecastTime=2026-06-01T00%3A00%3A00Z&ensembleId=GEPS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:02:43,621 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEPS&startForecastTime=2026-05-24T00%3A00%3A00Z&endForecastTime=2026-05-24T00%3A00%3A00Z&ensembleId=GEPS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:02:43,695 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEPS&startForecastTime=2026-05-17T00%3A00%3A00Z&endForecastTime=2026-05-17T00%3A00%3A00Z&ensembleId=GEPS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:02:43,695 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEPS&startForecastTime=2026-05-26T00%3A00%3A00Z&endForecastTime=2026-05-26T00%3A00%3A00Z&ensembleId=GEPS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:02:43,771 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEPS&startForecastTime=2026-05-23T00%3A00%3A00Z&endForecastTime=2026-05-23T00%3A00%3A00Z&ensembleId=GEPS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:02:43,773 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEPS&startForecastTime=2026-05-25T00%3A00%3A00Z&endForecastTime=2026-05-25T00%3A00%3A00Z&ensembleId=GEPS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:02:43,819 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEPS&startForecastTime=2026-05-27T00%3A00%3A00Z&endForecastTime=2026-05-27T00%3A00%3A00Z&ensembleId=GEPS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:02:43,825 - INFO - Starting dataset fetch (source_id=simulated_GEPS) from FewsNetCDF.
2026-11-04 15:03:14,196 - INFO - Opened dataset for source 'simulated_GEPS' from [WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260515_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260515_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260516_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260516_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260517_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260517_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260518_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260518_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260519_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260519_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260520_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260520_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260521_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260521_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260522_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260522_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260523_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260523_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260524_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260524_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260525_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260525_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260526_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260526_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260527_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260527_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260528_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260528_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260529_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260529_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260530_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260530_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260531_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260531_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260601_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmp52g6fhjy/20260601_000000_PC.nwp_0_externalforecasting_day.nc')] (took 30.37 seconds)
2026-11-04 15:03:14,291 - INFO - Dataset (source_id=simulated_GEPS) successfully loaded and validated.
2026-11-04 15:03:14,293 - INFO - Starting dataset fetch (source_id=simulated_GEFS) from FewsWebservice.
2026-11-04 15:03:14,947 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEFS&startForecastTime=2025-05-15T00%3A00%3A00Z&endForecastTime=2026-06-01T00%3A00%3A00Z&ensembleId=GEFS&forecastCount=1000000&documentFormat=PI_JSON&onlyHeaders=True&omitEmptyTimeSeries=True
2026-11-04 15:03:14,958 - INFO - Configured verification period (2025-05-15 00:00:00 to 2026-06-01 00:00:00) exceeds what source 'simulated_GEFS' has available (2026-05-15 00:00:00 to 2026-06-01 00:00:00); clamping to (2026-05-15 00:00:00 to 2026-06-01 00:00:00).
2026-11-04 15:03:15,006 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEFS&startForecastTime=2026-05-15T00%3A00%3A00Z&endForecastTime=2026-06-01T00%3A00%3A00Z&ensembleId=GEFS&forecastCount=1000000&documentFormat=PI_JSON&onlyHeaders=True&omitEmptyTimeSeries=True
2026-11-04 15:03:15,058 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEFS&startForecastTime=2026-05-15T00%3A00%3A00Z&endForecastTime=2026-05-15T00%3A00%3A00Z&ensembleId=GEFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:15,113 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEFS&startForecastTime=2026-05-21T00%3A00%3A00Z&endForecastTime=2026-05-21T00%3A00%3A00Z&ensembleId=GEFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:15,149 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEFS&startForecastTime=2026-05-16T00%3A00%3A00Z&endForecastTime=2026-05-16T00%3A00%3A00Z&ensembleId=GEFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:15,186 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEFS&startForecastTime=2026-05-18T00%3A00%3A00Z&endForecastTime=2026-05-18T00%3A00%3A00Z&ensembleId=GEFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:15,223 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEFS&startForecastTime=2026-05-28T00%3A00%3A00Z&endForecastTime=2026-05-28T00%3A00%3A00Z&ensembleId=GEFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:15,260 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEFS&startForecastTime=2026-05-30T00%3A00%3A00Z&endForecastTime=2026-05-30T00%3A00%3A00Z&ensembleId=GEFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:15,296 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEFS&startForecastTime=2026-05-20T00%3A00%3A00Z&endForecastTime=2026-05-20T00%3A00%3A00Z&ensembleId=GEFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:15,335 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEFS&startForecastTime=2026-05-31T00%3A00%3A00Z&endForecastTime=2026-05-31T00%3A00%3A00Z&ensembleId=GEFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:15,372 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEFS&startForecastTime=2026-05-22T00%3A00%3A00Z&endForecastTime=2026-05-22T00%3A00%3A00Z&ensembleId=GEFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:15,409 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEFS&startForecastTime=2026-05-19T00%3A00%3A00Z&endForecastTime=2026-05-19T00%3A00%3A00Z&ensembleId=GEFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:15,451 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEFS&startForecastTime=2026-06-01T00%3A00%3A00Z&endForecastTime=2026-06-01T00%3A00%3A00Z&ensembleId=GEFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:15,490 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEFS&startForecastTime=2026-05-24T00%3A00%3A00Z&endForecastTime=2026-05-24T00%3A00%3A00Z&ensembleId=GEFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:15,549 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEFS&startForecastTime=2026-05-17T00%3A00%3A00Z&endForecastTime=2026-05-17T00%3A00%3A00Z&ensembleId=GEFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:15,551 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEFS&startForecastTime=2026-05-29T00%3A00%3A00Z&endForecastTime=2026-05-29T00%3A00%3A00Z&ensembleId=GEFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:15,617 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEFS&startForecastTime=2026-05-25T00%3A00%3A00Z&endForecastTime=2026-05-25T00%3A00%3A00Z&ensembleId=GEFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:15,621 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEFS&startForecastTime=2026-05-26T00%3A00%3A00Z&endForecastTime=2026-05-26T00%3A00%3A00Z&ensembleId=GEFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:15,671 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEFS&startForecastTime=2026-05-27T00%3A00%3A00Z&endForecastTime=2026-05-27T00%3A00%3A00Z&ensembleId=GEFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:15,676 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessGEFS&startForecastTime=2026-05-23T00%3A00%3A00Z&endForecastTime=2026-05-23T00%3A00%3A00Z&ensembleId=GEFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:15,682 - INFO - Starting dataset fetch (source_id=simulated_GEFS) from FewsNetCDF.
2026-11-04 15:03:40,509 - INFO - Opened dataset for source 'simulated_GEFS' from [WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260515_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260515_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260516_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260516_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260517_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260517_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260518_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260518_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260519_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260519_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260520_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260520_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260521_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260521_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260522_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260522_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260523_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260523_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260524_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260524_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260525_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260525_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260526_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260526_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260527_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260527_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260528_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260528_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260529_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260529_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260530_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260530_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260531_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260531_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260601_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmpdr4ipfsd/20260601_000000_PC.nwp_0_externalforecasting_day.nc')] (took 24.82 seconds)
2026-11-04 15:03:40,610 - INFO - Dataset (source_id=simulated_GEFS) successfully loaded and validated.
2026-11-04 15:03:40,611 - INFO - Starting dataset fetch (source_id=simulated_IFS) from FewsWebservice.
2026-11-04 15:03:41,695 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessIFS&startForecastTime=2025-05-15T00%3A00%3A00Z&endForecastTime=2026-06-01T00%3A00%3A00Z&ensembleId=IFS&forecastCount=1000000&documentFormat=PI_JSON&onlyHeaders=True&omitEmptyTimeSeries=True
2026-11-04 15:03:41,717 - INFO - Configured verification period (2025-05-15 00:00:00 to 2026-06-01 00:00:00) exceeds what source 'simulated_IFS' has available (2026-05-15 00:00:00 to 2026-06-01 00:00:00); clamping to (2026-05-15 00:00:00 to 2026-06-01 00:00:00).
2026-11-04 15:03:41,853 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessIFS&startForecastTime=2026-05-15T00%3A00%3A00Z&endForecastTime=2026-06-01T00%3A00%3A00Z&ensembleId=IFS&forecastCount=1000000&documentFormat=PI_JSON&onlyHeaders=True&omitEmptyTimeSeries=True
2026-11-04 15:03:41,940 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessIFS&startForecastTime=2026-05-15T00%3A00%3A00Z&endForecastTime=2026-05-15T00%3A00%3A00Z&ensembleId=IFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:42,008 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessIFS&startForecastTime=2026-05-21T00%3A00%3A00Z&endForecastTime=2026-05-21T00%3A00%3A00Z&ensembleId=IFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:42,062 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessIFS&startForecastTime=2026-05-16T00%3A00%3A00Z&endForecastTime=2026-05-16T00%3A00%3A00Z&ensembleId=IFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:42,100 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessIFS&startForecastTime=2026-05-18T00%3A00%3A00Z&endForecastTime=2026-05-18T00%3A00%3A00Z&ensembleId=IFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:42,137 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessIFS&startForecastTime=2026-05-28T00%3A00%3A00Z&endForecastTime=2026-05-28T00%3A00%3A00Z&ensembleId=IFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:42,175 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessIFS&startForecastTime=2026-05-30T00%3A00%3A00Z&endForecastTime=2026-05-30T00%3A00%3A00Z&ensembleId=IFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:42,211 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessIFS&startForecastTime=2026-05-20T00%3A00%3A00Z&endForecastTime=2026-05-20T00%3A00%3A00Z&ensembleId=IFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:42,250 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessIFS&startForecastTime=2026-05-31T00%3A00%3A00Z&endForecastTime=2026-05-31T00%3A00%3A00Z&ensembleId=IFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:42,309 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessIFS&startForecastTime=2026-05-22T00%3A00%3A00Z&endForecastTime=2026-05-22T00%3A00%3A00Z&ensembleId=IFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:42,348 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessIFS&startForecastTime=2026-05-19T00%3A00%3A00Z&endForecastTime=2026-05-19T00%3A00%3A00Z&ensembleId=IFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:42,388 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessIFS&startForecastTime=2026-06-01T00%3A00%3A00Z&endForecastTime=2026-06-01T00%3A00%3A00Z&ensembleId=IFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:42,432 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessIFS&startForecastTime=2026-05-24T00%3A00%3A00Z&endForecastTime=2026-05-24T00%3A00%3A00Z&ensembleId=IFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:42,470 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessIFS&startForecastTime=2026-05-17T00%3A00%3A00Z&endForecastTime=2026-05-17T00%3A00%3A00Z&ensembleId=IFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:42,525 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessIFS&startForecastTime=2026-05-26T00%3A00%3A00Z&endForecastTime=2026-05-26T00%3A00%3A00Z&ensembleId=IFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:42,531 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessIFS&startForecastTime=2026-05-29T00%3A00%3A00Z&endForecastTime=2026-05-29T00%3A00%3A00Z&ensembleId=IFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:42,568 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessIFS&startForecastTime=2026-05-25T00%3A00%3A00Z&endForecastTime=2026-05-25T00%3A00%3A00Z&ensembleId=IFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:42,581 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessIFS&startForecastTime=2026-05-23T00%3A00%3A00Z&endForecastTime=2026-05-23T00%3A00%3A00Z&ensembleId=IFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:42,604 - INFO - Download successful from URL: https://veriflow-open.fews.deltares.nl/FewsWebServices/rest/fewspiservice/v1/timeseries?locationIds=3031092¶meterIds=PC.nwp&moduleInstanceIds=PreprocessIFS&startForecastTime=2026-05-27T00%3A00%3A00Z&endForecastTime=2026-05-27T00%3A00%3A00Z&ensembleId=IFS&documentFormat=PI_NETCDF&omitEmptyTimeSeries=True
2026-11-04 15:03:42,611 - INFO - Starting dataset fetch (source_id=simulated_IFS) from FewsNetCDF.
2026-11-04 15:04:26,075 - INFO - Opened dataset for source 'simulated_IFS' from [WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260515_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260515_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260516_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260516_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260517_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260517_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260518_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260518_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260519_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260519_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260520_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260520_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260521_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260521_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260522_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260522_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260523_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260523_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260524_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260524_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260525_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260525_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260526_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260526_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260527_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260527_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260528_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260528_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260529_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260529_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260530_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260530_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260531_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260531_000000_PC.nwp_0_externalforecasting_day.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260601_000000_PC.nwp_0_externalforecasting_3hours.nc'), WindowsPath('C:/Users/beunk/AppData/Local/Temp/tmplr0w94q7/20260601_000000_PC.nwp_0_externalforecasting_day.nc')] (took 43.46 seconds)
2026-11-04 15:04:26,188 - INFO - Dataset (source_id=simulated_IFS) successfully loaded and validated.
2026-11-04 15:04:26,192 - INFO - Successfully loaded all data from sources.
2026-11-04 15:04:26,272 - INFO - Successfully computed CrpsForEnsemble for verification pair PC_GEPS.
2026-11-04 15:04:26,298 - INFO - Successfully computed CrpsForEnsemble for verification pair PC_GEFS.
2026-11-04 15:04:26,321 - INFO - Successfully computed CrpsForEnsemble for verification pair PC_IFS.
2026-11-04 15:04:26,358 - INFO - Successfully computed RankHistogram for verification pair PC_GEPS.
2026-11-04 15:04:26,399 - INFO - Successfully computed RankHistogram for verification pair PC_GEFS.
2026-11-04 15:04:26,436 - INFO - Successfully computed RankHistogram for verification pair PC_IFS.
2026-11-04 15:04:26,438 - INFO - Verification pipeline completed successfully.
Evaluating the results in the veriflow output DataTree#
Verification metrics and results can contain a level of abstraction. Although these abstractions can reveal important information about forecast quality, a basic “eyeball verification” is often the best and intuitive way to start your verification exercise. You’ll likely find strengths and weaknesses in your forecasts early on, without directly diving into levels of abstraction. In addition, a solid visual inspection may help you later on in understanding or explaining the more abstract results.
1 - Visual inspection of observed and forecast data#
A good starting point for “eyeball” verification is simple: just looking at your observations and forecasts in a visual way. Use the interactive elements in the plots below to zoom, pan and compare the results of our 3 NWP products.
[3]:
dt
[3]:
<xarray.DataTree 'veriflow-datatree'>
Group: /
├── Group: /input_data
│ ├── Group: /input_data/observed
│ │ Dimensions: (station: 1, time: 388)
│ │ Coordinates:
│ │ * station (station) <U7 28B '3031092'
│ │ station_name (station) |S255 255B b'Calgary International'
│ │ lat (station) float64 8B 51.12
│ │ lon (station) float64 8B -114.0
│ │ y (station) float64 8B 51.12
│ │ x (station) float64 8B -114.0
│ │ z (station) float64 8B 1.099e+03
│ │ * time (time) datetime64[ns] 3kB 2025-05-16 2025-05-17 ... 2026-06-07
│ │ Data variables:
│ │ PC (station, time) float32 2kB 3.7 6.0 14.3 0.2 ... 0.0 0.2 6.0
│ │ Attributes: (12/14)
│ │ originalParameterId: PC.obs
│ │ Conventions: CF-1.6
│ │ coordinate_system: WGS 1984
│ │ featureType: timeSeries
│ │ time_coverage_start: 2025-05-16T00:00:00+0000
│ │ time_coverage_end: 2026-06-07T00:00:00+0000
│ │ ... ...
│ │ geospatial_lat_min: 51.1226
│ │ geospatial_lat_max: 51.1226
│ │ data_type: observed_historical
│ │ source_id: observed
│ │ spatial_type: point
│ │ crs: EPSG:4326
│ ├── Group: /input_data/simulated_GEPS
│ │ Dimensions: (station: 1, forecast_reference_time: 18,
│ │ lead_time: 6, realization: 21)
│ │ Coordinates:
│ │ * station (station) <U7 28B '3031092'
│ │ station_name (station) |S255 255B b'Calgary International'
│ │ lat (station) float64 8B 51.12
│ │ lon (station) float64 8B -114.0
│ │ y (station) float64 8B 51.12
│ │ x (station) float64 8B -114.0
│ │ z (station) float64 8B 1.099e+03
│ │ * forecast_reference_time (forecast_reference_time) datetime64[ns] 144B 20...
│ │ * lead_time (lead_time) timedelta64[ns] 48B 1 days ... 6 days
│ │ time (forecast_reference_time, lead_time) datetime64[ns] 864B ...
│ │ * realization (realization) int32 84B 0 1 2 3 4 ... 17 18 19 20
│ │ Data variables:
│ │ PC (station, forecast_reference_time, lead_time, realization) float32 9kB ...
│ │ Attributes: (12/14)
│ │ originalParameterId: PC.nwp
│ │ Conventions: CF-1.6
│ │ coordinate_system: WGS 1984
│ │ featureType: timeSeries
│ │ time_coverage_start: 2026-05-16T00:00:00+0000
│ │ time_coverage_end: 2026-05-22T00:00:00+0000
│ │ ... ...
│ │ geospatial_lat_min: 51.1226
│ │ geospatial_lat_max: 51.1226
│ │ data_type: simulated_forecast_ensemble
│ │ source_id: simulated_GEPS
│ │ spatial_type: point
│ │ crs: EPSG:4326
│ ├── Group: /input_data/simulated_GEFS
│ │ Dimensions: (station: 1, forecast_reference_time: 18,
│ │ lead_time: 6, realization: 31)
│ │ Coordinates:
│ │ * station (station) <U7 28B '3031092'
│ │ station_name (station) |S255 255B b'Calgary International'
│ │ lat (station) float64 8B 51.12
│ │ lon (station) float64 8B -114.0
│ │ y (station) float64 8B 51.12
│ │ x (station) float64 8B -114.0
│ │ z (station) float64 8B 1.099e+03
│ │ * forecast_reference_time (forecast_reference_time) datetime64[ns] 144B 20...
│ │ * lead_time (lead_time) timedelta64[ns] 48B 1 days ... 6 days
│ │ time (forecast_reference_time, lead_time) datetime64[ns] 864B ...
│ │ * realization (realization) int32 124B 0 1 2 3 4 ... 27 28 29 30
│ │ Data variables:
│ │ PC (station, forecast_reference_time, lead_time, realization) float32 13kB ...
│ │ Attributes: (12/14)
│ │ originalParameterId: PC.nwp
│ │ Conventions: CF-1.6
│ │ coordinate_system: WGS 1984
│ │ featureType: timeSeries
│ │ time_coverage_start: 2026-05-15T00:00:00+0000
│ │ time_coverage_end: 2026-05-21T00:00:00+0000
│ │ ... ...
│ │ geospatial_lat_min: 51.1226
│ │ geospatial_lat_max: 51.1226
│ │ data_type: simulated_forecast_ensemble
│ │ source_id: simulated_GEFS
│ │ spatial_type: point
│ │ crs: EPSG:4326
│ └── Group: /input_data/simulated_IFS
│ Dimensions: (station: 1, forecast_reference_time: 18,
│ lead_time: 6, realization: 51)
│ Coordinates:
│ * station (station) <U7 28B '3031092'
│ station_name (station) |S255 255B b'Calgary International'
│ lat (station) float64 8B 51.12
│ lon (station) float64 8B -114.0
│ y (station) float64 8B 51.12
│ x (station) float64 8B -114.0
│ z (station) float64 8B 1.099e+03
│ * forecast_reference_time (forecast_reference_time) datetime64[ns] 144B 20...
│ * lead_time (lead_time) timedelta64[ns] 48B 1 days ... 6 days
│ time (forecast_reference_time, lead_time) datetime64[ns] 864B ...
│ * realization (realization) int32 204B 0 1 2 3 4 ... 47 48 49 50
│ Data variables:
│ PC (station, forecast_reference_time, lead_time, realization) float32 22kB ...
│ Attributes: (12/14)
│ originalParameterId: PC.nwp
│ Conventions: CF-1.6
│ coordinate_system: WGS 1984
│ featureType: timeSeries
│ time_coverage_start: 2026-05-15T00:00:00+0000
│ time_coverage_end: 2026-05-21T00:00:00+0000
│ ... ...
│ geospatial_lat_min: 51.1226
│ geospatial_lat_max: 51.1226
│ data_type: simulated_forecast_ensemble
│ source_id: simulated_IFS
│ spatial_type: point
│ crs: EPSG:4326
├── Group: /PC_GEPS
│ ├── Group: /PC_GEPS/aligned_input
│ │ ├── Group: /PC_GEPS/aligned_input/observations
│ │ │ Dimensions: (forecast_reference_time: 18, lead_time: 6,
│ │ │ station: 1)
│ │ │ Coordinates:
│ │ │ * forecast_reference_time (forecast_reference_time) datetime64[ns] 144B 20...
│ │ │ * lead_time (lead_time) timedelta64[ns] 48B 1 days ... 6 days
│ │ │ * station (station) <U7 28B '3031092'
│ │ │ station_name (station) |S255 255B b'Calgary International'
│ │ │ lat (station) float64 8B 51.12
│ │ │ lon (station) float64 8B -114.0
│ │ │ y (station) float64 8B 51.12
│ │ │ x (station) float64 8B -114.0
│ │ │ z (station) float64 8B 1.099e+03
│ │ │ Data variables:
│ │ │ PC (station, forecast_reference_time, lead_time) float32 432B ...
│ │ └── Group: /PC_GEPS/aligned_input/simulations
│ │ Dimensions: (station: 1, forecast_reference_time: 18,
│ │ realization: 21, lead_time: 6)
│ │ Coordinates:
│ │ * station (station) <U7 28B '3031092'
│ │ station_name (station) |S255 255B b'Calgary International'
│ │ lat (station) float64 8B 51.12
│ │ lon (station) float64 8B -114.0
│ │ y (station) float64 8B 51.12
│ │ x (station) float64 8B -114.0
│ │ z (station) float64 8B 1.099e+03
│ │ * forecast_reference_time (forecast_reference_time) datetime64[ns] 144B 20...
│ │ * realization (realization) int32 84B 0 1 2 3 4 ... 17 18 19 20
│ │ * lead_time (lead_time) timedelta64[ns] 48B 1 days ... 6 days
│ │ time (forecast_reference_time, lead_time) datetime64[ns] 864B ...
│ │ Data variables:
│ │ PC (station, forecast_reference_time, lead_time, realization) float32 9kB ...
│ └── Group: /PC_GEPS/output
│ ├── Group: /PC_GEPS/output/crps_for_ensemble
│ │ Dimensions: (station: 1, forecast_reference_time: 18,
│ │ lead_time: 6)
│ │ Coordinates:
│ │ * station (station) <U7 28B '3031092'
│ │ station_name (station) |S255 255B b'Calgary International'
│ │ lat (station) float64 8B 51.12
│ │ lon (station) float64 8B -114.0
│ │ y (station) float64 8B 51.12
│ │ x (station) float64 8B -114.0
│ │ z (station) float64 8B 1.099e+03
│ │ * forecast_reference_time (forecast_reference_time) datetime64[ns] 144B 20...
│ │ * lead_time (lead_time) timedelta64[ns] 48B 1 days ... 6 days
│ │ time (forecast_reference_time, lead_time) datetime64[ns] 864B ...
│ │ Data variables:
│ │ crps_for_ensemble (station, forecast_reference_time, lead_time) float64 864B ...
│ └── Group: /PC_GEPS/output/rank_histogram
│ Dimensions: (lead_time: 6, station: 1, rank: 22)
│ Coordinates:
│ * lead_time (lead_time) timedelta64[ns] 48B 1 days 2 days ... 6 days
│ * station (station) <U7 28B '3031092'
│ station_name (station) |S255 255B b'Calgary International'
│ lat (station) float64 8B 51.12
│ lon (station) float64 8B -114.0
│ y (station) float64 8B 51.12
│ x (station) float64 8B -114.0
│ z (station) float64 8B 1.099e+03
│ * rank (rank) int64 176B 1 2 3 4 5 6 7 8 ... 16 17 18 19 20 21 22
│ Data variables:
│ rank_histogram (station, lead_time, rank) float64 1kB 0.1607 ... 0.0
├── Group: /PC_GEFS
│ ├── Group: /PC_GEFS/aligned_input
│ │ ├── Group: /PC_GEFS/aligned_input/observations
│ │ │ Dimensions: (forecast_reference_time: 18, lead_time: 6,
│ │ │ station: 1)
│ │ │ Coordinates:
│ │ │ * forecast_reference_time (forecast_reference_time) datetime64[ns] 144B 20...
│ │ │ * lead_time (lead_time) timedelta64[ns] 48B 1 days ... 6 days
│ │ │ * station (station) <U7 28B '3031092'
│ │ │ station_name (station) |S255 255B b'Calgary International'
│ │ │ lat (station) float64 8B 51.12
│ │ │ lon (station) float64 8B -114.0
│ │ │ y (station) float64 8B 51.12
│ │ │ x (station) float64 8B -114.0
│ │ │ z (station) float64 8B 1.099e+03
│ │ │ Data variables:
│ │ │ PC (station, forecast_reference_time, lead_time) float32 432B ...
│ │ └── Group: /PC_GEFS/aligned_input/simulations
│ │ Dimensions: (station: 1, forecast_reference_time: 18,
│ │ realization: 31, lead_time: 6)
│ │ Coordinates:
│ │ * station (station) <U7 28B '3031092'
│ │ station_name (station) |S255 255B b'Calgary International'
│ │ lat (station) float64 8B 51.12
│ │ lon (station) float64 8B -114.0
│ │ y (station) float64 8B 51.12
│ │ x (station) float64 8B -114.0
│ │ z (station) float64 8B 1.099e+03
│ │ * forecast_reference_time (forecast_reference_time) datetime64[ns] 144B 20...
│ │ * realization (realization) int32 124B 0 1 2 3 4 ... 27 28 29 30
│ │ * lead_time (lead_time) timedelta64[ns] 48B 1 days ... 6 days
│ │ time (forecast_reference_time, lead_time) datetime64[ns] 864B ...
│ │ Data variables:
│ │ PC (station, forecast_reference_time, lead_time, realization) float32 13kB ...
│ └── Group: /PC_GEFS/output
│ ├── Group: /PC_GEFS/output/crps_for_ensemble
│ │ Dimensions: (station: 1, forecast_reference_time: 18,
│ │ lead_time: 6)
│ │ Coordinates:
│ │ * station (station) <U7 28B '3031092'
│ │ station_name (station) |S255 255B b'Calgary International'
│ │ lat (station) float64 8B 51.12
│ │ lon (station) float64 8B -114.0
│ │ y (station) float64 8B 51.12
│ │ x (station) float64 8B -114.0
│ │ z (station) float64 8B 1.099e+03
│ │ * forecast_reference_time (forecast_reference_time) datetime64[ns] 144B 20...
│ │ * lead_time (lead_time) timedelta64[ns] 48B 1 days ... 6 days
│ │ time (forecast_reference_time, lead_time) datetime64[ns] 864B ...
│ │ Data variables:
│ │ crps_for_ensemble (station, forecast_reference_time, lead_time) float64 864B ...
│ └── Group: /PC_GEFS/output/rank_histogram
│ Dimensions: (lead_time: 6, station: 1, rank: 32)
│ Coordinates:
│ * lead_time (lead_time) timedelta64[ns] 48B 1 days 2 days ... 6 days
│ * station (station) <U7 28B '3031092'
│ station_name (station) |S255 255B b'Calgary International'
│ lat (station) float64 8B 51.12
│ lon (station) float64 8B -114.0
│ y (station) float64 8B 51.12
│ x (station) float64 8B -114.0
│ z (station) float64 8B 1.099e+03
│ * rank (rank) int64 256B 1 2 3 4 5 6 7 8 ... 26 27 28 29 30 31 32
│ Data variables:
│ rank_histogram (station, lead_time, rank) float64 2kB 0.06497 ... 0.0
└── Group: /PC_IFS
├── Group: /PC_IFS/aligned_input
│ ├── Group: /PC_IFS/aligned_input/observations
│ │ Dimensions: (forecast_reference_time: 18, lead_time: 6,
│ │ station: 1)
│ │ Coordinates:
│ │ * forecast_reference_time (forecast_reference_time) datetime64[ns] 144B 20...
│ │ * lead_time (lead_time) timedelta64[ns] 48B 1 days ... 6 days
│ │ * station (station) <U7 28B '3031092'
│ │ station_name (station) |S255 255B b'Calgary International'
│ │ lat (station) float64 8B 51.12
│ │ lon (station) float64 8B -114.0
│ │ y (station) float64 8B 51.12
│ │ x (station) float64 8B -114.0
│ │ z (station) float64 8B 1.099e+03
│ │ Data variables:
│ │ PC (station, forecast_reference_time, lead_time) float32 432B ...
│ └── Group: /PC_IFS/aligned_input/simulations
│ Dimensions: (station: 1, forecast_reference_time: 18,
│ realization: 51, lead_time: 6)
│ Coordinates:
│ * station (station) <U7 28B '3031092'
│ station_name (station) |S255 255B b'Calgary International'
│ lat (station) float64 8B 51.12
│ lon (station) float64 8B -114.0
│ y (station) float64 8B 51.12
│ x (station) float64 8B -114.0
│ z (station) float64 8B 1.099e+03
│ * forecast_reference_time (forecast_reference_time) datetime64[ns] 144B 20...
│ * realization (realization) int32 204B 0 1 2 3 4 ... 47 48 49 50
│ * lead_time (lead_time) timedelta64[ns] 48B 1 days ... 6 days
│ time (forecast_reference_time, lead_time) datetime64[ns] 864B ...
│ Data variables:
│ PC (station, forecast_reference_time, lead_time, realization) float32 22kB ...
└── Group: /PC_IFS/output
├── Group: /PC_IFS/output/crps_for_ensemble
│ Dimensions: (station: 1, forecast_reference_time: 18,
│ lead_time: 6)
│ Coordinates:
│ * station (station) <U7 28B '3031092'
│ station_name (station) |S255 255B b'Calgary International'
│ lat (station) float64 8B 51.12
│ lon (station) float64 8B -114.0
│ y (station) float64 8B 51.12
│ x (station) float64 8B -114.0
│ z (station) float64 8B 1.099e+03
│ * forecast_reference_time (forecast_reference_time) datetime64[ns] 144B 20...
│ * lead_time (lead_time) timedelta64[ns] 48B 1 days ... 6 days
│ time (forecast_reference_time, lead_time) datetime64[ns] 864B ...
│ Data variables:
│ crps_for_ensemble (station, forecast_reference_time, lead_time) float64 864B ...
└── Group: /PC_IFS/output/rank_histogram
Dimensions: (lead_time: 6, station: 1, rank: 52)
Coordinates:
* lead_time (lead_time) timedelta64[ns] 48B 1 days 2 days ... 6 days
* station (station) <U7 28B '3031092'
station_name (station) |S255 255B b'Calgary International'
lat (station) float64 8B 51.12
lon (station) float64 8B -114.0
y (station) float64 8B 51.12
x (station) float64 8B -114.0
z (station) float64 8B 1.099e+03
* rank (rank) int64 416B 1 2 3 4 5 6 7 8 ... 46 47 48 49 50 51 52
Data variables:
rank_histogram (station, lead_time, rank) float64 2kB 0.1463 ... 0.0[4]:
ds = dt.veriflow.get_aligned_input(dt.veriflow.verification_pairs[0])["observations"].to_dataset()
ds
[4]:
<xarray.Dataset> Size: 947B
Dimensions: (station: 1, forecast_reference_time: 18,
lead_time: 6)
Coordinates:
* station (station) <U7 28B '3031092'
station_name (station) |S255 255B b'Calgary International'
lat (station) float64 8B 51.12
lon (station) float64 8B -114.0
y (station) float64 8B 51.12
x (station) float64 8B -114.0
z (station) float64 8B 1.099e+03
* forecast_reference_time (forecast_reference_time) datetime64[ns] 144B 20...
* lead_time (lead_time) timedelta64[ns] 48B 1 days ... 6 days
Data variables:
PC (station, forecast_reference_time, lead_time) float32 432B ...[10]:
stations = ds.coords["station"].values
lead_times = ds.coords["lead_time"].values
lead_times_hours = [str(lt.astype("timedelta64[h]")) for lt in lead_times]
print(f"Stations: {stations}")
print(f"GDPS Lead times (hours): {lead_times_hours}")
Stations: ['3031092']
GDPS Lead times (hours): ['24 hours', '48 hours', '72 hours', '96 hours', '120 hours', '144 hours']
[11]:
forecast_timeseries_plot(dt, show_members=False)
2 - Visual inspection with a scatter plot per lead time#
Another great tool for “eyeball” verification is the scatter plot. The scatter plot is relatively easy to understand, but is slightly more abstract than the visualization above.
For all forecasts in our output DataTree, we collect all realizations of the ensemble at a specific lead time. You can use the lead_times variable (a list of np.timedelta64 instances) to slice the data at a specific lead time. For that given slice, we can make the scatter plot.
[12]:
scatter_plot(dt, lead_time=lead_times[1])
3 - Looking into the Continuous Ranked Probability Score#
Next, we’ll look into the results of the continuous ranked probability score. Before you continue to the next section, we’ll look into the documentation and do a tutorial on the CRPS for ensemble forecasts. The veriflow package relies on scores (developed by the Bureau of Meteorology, Australia) for computation of various scores.
Read the API documentation the crps_for_ensemble function, which is used under the hood in veriflow.
Run through the scores tutorial on the crps_for_ensemble function. You can run it in Binder (link on top of the page), or view the static view.
Q1: what attribute(s) of forecast quality can be measured by the CRPS?
Show suggested answers for Q1
The CRPS captures multiple attributes: accuracy, reliability and sharpness. Can you reason why?
Q2: what is the best possible CRPS score?
Show suggested answers for Q2
The best possible outcome of CRPS is 0. In this case the forecast has maximum sharpness: all ensemble members exactly predict the observed outcome.
Q3: computing the CRPS over just one realization (i.e. a deterministic forecast) yields the exact same result as computing the … for a deterministic score?
Show suggested answers for Q3
The absolute error. The CRPS is a probabilistic generalization of the absolute error. When taking the mean of the CRPS over all forecasts, it is equal to the mean absolute error when the number of realizations is 1.
[8]:
plot_score_vs_lead_time(dt)
4 - Looking into the Rank Histogram#
Next, we’ll look into the results of the rank histogram. Before you continue to the next section, we’ll look into the documentation and do a tutorial on the rank histogram for ensemble forecasts. The veriflow package relies on scores (developed by the Bureau of Meteorology, Australia) for computation of various scores.
Read the API documentation the rank_histogram function, which is used under the hood in veriflow.
Run through the scores tutorial on the rank_histogram function. You can run it in Binder (link on top of the page), or view the static view.
Q1: what attribute(s) of forecast quality can be measured by the Rank Histogram?
Show suggested answers for Q1
The rank histogram primarily measures reliability, although bias in forecasts will show up in the rank histogram as well.
Q2: what does a perfect Rank Histogram look like?
Show suggested answers for Q2
The rank histogram shows a uniform (flat) distribution, indicating forecast and observed frequencies are equal.
Q3: does a perfectly uniform/flat rank histogram always correspond to a good forecast? Can you come up with a hypothetical forecast that has a perfect rank histogram, but is still bad?
Show suggested answers for Q3
No, although a perfect rank histogram indicates statistical reliability, the forecast may still have poor resolution (the ability to discriminate between situations that have different observed outcomes).
Imagine an ensemble forecast that ignores current conditions and simply samples from the historical distribution of streamflow for that day of year.
For example, every day in June the ensemble consists of random draws from the last 30 years of June observations.
This forecast can also produce a nearly uniform rank histogram because the observations come from the same climatological distribution. However, it has zero ability to distinguish today’s conditions from any other June day.
[9]:
rank_histogram_plot(dt, lead_time=lead_times[1])