veriflow.datatree.datatree#

The veriflow data tree structure and accessor.

The veriflow DataTree is an instance of xarray.DataTree and thus has all standard methods and attributes of an xarray.DataTree. The VeriflowAccessor extends this functionality with methods specific to the veriflow DataTree structure. It can be used as follows:

import xarray as xr

from veriflow import run_pipeline

# The output of the veriflow pipeline is an instance of ``xr.DataTree``.
dt: xr.DataTree = run_pipeline(...)

# It has all the standard methods and attributes of an ``xarray.DataTree``.
dt.keys()  # Example of a standard xarray.DataTree method.

# The ``dt.veriflow`` accessor provides additional methods to interact with the veriflow
# DataTree.

# Example 1: Get the list of verification pair IDs.
dt.veriflow.verification_pairs

# Example 2: List scores for a given verification_pair
dt.veriflow.list_scores("example_verification_pair_id")

The VeriflowAccessor extends xarray.DataTree via the xr.register_datatree_accessor decorator, as recommended by the xarray documentation.

Classes

VeriflowAccessor(dt)

Accessor for managing the internal datatree for veriflow.

class veriflow.datatree.datatree.VeriflowAccessor(dt)[source]#

Accessor for managing the internal datatree for veriflow.

The DataTree layout is as follows:

  • The root node is veriflow-datatree.

  • The the root node’s child nodes are input_data and one or more verification pairs.

  • The input_data node holds the validated input datasets. These are instances of xr.Dataset`as fetched from the configured datasources. They are instances of `xr.Dataset keyed by their source_id.

  • Each verification_pair node represents a verification pair, identified by its unique ID, and has two main child nodes: aligned_input and output.

    • The aligned_input node contains data prepared for verification. The observation and simulation data are aligned and ready for verification. For example, when a forecast is verified against observations, the observations are mapped into forecast space along the forecast_reference_time and lead_time dimensions.

    • The output node contains the results of the verification. Each child under the output node is a dataset containing one or more data variables corresponding to different aspects of the output.

An example of the DataTree structure:

veriflow-datatree
├── input_data
│   ├── source_id_a
│   └── source_id_b
│   └── source_id_c
├── verification_pair_1
│   ├── aligned_input
│   │   ├── observations
│   │   │   └── waterlevel
│   │   └── simulations
│   │       └── waterlevel
│   └── output
│       ├── crps
│       │   ├── crps
│       │   ├── overforecast_penalty
│       │   ├── underforecast_penalty
│       │   └── forecast_spread_term
│       └── rank_histogram
│           └── rank_histogram
└── verification_pair_2
    ├── aligned_input
    │   └── ...
    └── output
        └── ...
Parameters:

dt (DataTree)

property dt: DataTree#

Get the underlying DataTree.

property verification_pairs: list[str]#

Get a list of verification pair IDs in the DataTree.

static map_historical_into_forecast_space(obs, sim)[source]#

Transform array of historical data into forecast structure.

Given an observation array with dimension ‘time’ and a simulation array with dimensions ‘forecast_reference_time’ and ‘lead_time’, project the observed values onto the simulation array.

This method is called at runtime when the pipeline starts a score computation on forecast data. On the fly, the observation array is mapped to the forecast structure, so data are aligned along the same dimensions.

Parameters:
Return type:

DataArray

get_pair(verification_pair)[source]#

Return observations and simulations for a given verification pair.

Selects verification_pair.variable from each source’s dataset and returns the resulting DataArrays. This method is called by the verification pipeline at runtime to retrieve the correct data for one of the configured verification pairs.

Parameters:

verification_pair (VerificationPair)

Return type:

tuple[DataArray, DataArray]

get_thresholds_array(variable)[source]#

Get the thresholds array for a given variable from the input dataset.

Parameters:

variable (str)

Return type:

DataArray

get_verification_pair_node(verification_pair_id)[source]#

Get the DataTree node corresponding to a specific verification pair.

Parameters:

verification_pair_id (str)

Return type:

DataTree

get_aligned_input(verification_pair_id)[source]#

Get the input dataset for a specific verification pair.

Parameters:

verification_pair_id (str)

Return type:

DataTree

get_outputs(verification_pair_id)[source]#

Get the output dataset for a specific verification pair.

Parameters:

verification_pair_id (str)

Return type:

DataTree

list_scores(verification_pair_id)[source]#

Get a list of available scores for a specific verification pair.

Parameters:

verification_pair_id (str)

Return type:

list[str]

add_score(verification_pair, result, name)[source]#

Add a score results to the datatree.

Parameters:
Return type:

None

add_staged_input_data(verification_pair, obs, sim)[source]#

Add input data to the datastore.

Parameters:
Return type:

None

add_input_data(data)[source]#

Add raw input datasets to the datatree, keyed by their source_id.

Parameters:

data (Iterable[Dataset])

Return type:

None

filter_nodes(*, include_input_data, include_aligned_input_data, include_output)[source]#

Filter the datatree based on the specified inclusion flags.

Builds a new DataTree containing only the parts of this DataTree selected by config: the raw input_data node, and/or the aligned_input and output nodes under each verification pair, in any combination.

Parameters:
  • include_input_data (bool)

  • include_aligned_input_data (bool)

  • include_output (bool)

Return type:

DataTree