veriflow.datatree.datatree#
The veriflow data tree structure and accessor.
The veriflow DataTree is an instance of xarray.DataTree and thus has all standard methods and
attributes of an xarray.DataTree. The VeriflowAccessor extends this functionality with methods
specific to the veriflow DataTree structure. It can be used as follows:
import xarray as xr
from veriflow import run_pipeline
# The output of the veriflow pipeline is an instance of ``xr.DataTree``.
dt: xr.DataTree = run_pipeline(...)
# It has all the standard methods and attributes of an ``xarray.DataTree``.
dt.keys() # Example of a standard xarray.DataTree method.
# The ``dt.veriflow`` accessor provides additional methods to interact with the veriflow
# DataTree.
# Example 1: Get the list of verification pair IDs.
dt.veriflow.verification_pairs
# Example 2: List scores for a given verification_pair
dt.veriflow.list_scores("example_verification_pair_id")
The VeriflowAccessor extends xarray.DataTree via the xr.register_datatree_accessor
decorator, as recommended by the xarray documentation.
Classes
|
Accessor for managing the internal datatree for veriflow. |
- class veriflow.datatree.datatree.VeriflowAccessor(dt)[source]#
Accessor for managing the internal datatree for veriflow.
The DataTree layout is as follows:
The root node is
veriflow-datatree.The the root node’s child nodes are
input_dataand one or more verification pairs.The
input_datanode holds the validated input datasets. These are instances of xr.Dataset`as fetched from the configured datasources. They are instances of `xr.Dataset keyed by theirsource_id.Each
verification_pairnode represents a verification pair, identified by its unique ID, and has two main child nodes:aligned_inputandoutput.The
aligned_inputnode contains data prepared for verification. The observation and simulation data are aligned and ready for verification. For example, when a forecast is verified against observations, the observations are mapped into forecast space along theforecast_reference_timeandlead_timedimensions.The
outputnode contains the results of the verification. Each child under theoutputnode is a dataset containing one or more data variables corresponding to different aspects of the output.
An example of the DataTree structure:
veriflow-datatree ├── input_data │ ├── source_id_a │ └── source_id_b │ └── source_id_c ├── verification_pair_1 │ ├── aligned_input │ │ ├── observations │ │ │ └── waterlevel │ │ └── simulations │ │ └── waterlevel │ └── output │ ├── crps │ │ ├── crps │ │ ├── overforecast_penalty │ │ ├── underforecast_penalty │ │ └── forecast_spread_term │ └── rank_histogram │ └── rank_histogram └── verification_pair_2 ├── aligned_input │ └── ... └── output └── ...- Parameters:
dt (DataTree)
- static map_historical_into_forecast_space(obs, sim)[source]#
Transform array of historical data into forecast structure.
Given an observation array with dimension ‘time’ and a simulation array with dimensions ‘forecast_reference_time’ and ‘lead_time’, project the observed values onto the simulation array.
This method is called at runtime when the pipeline starts a score computation on forecast data. On the fly, the observation array is mapped to the forecast structure, so data are aligned along the same dimensions.
- get_pair(verification_pair)[source]#
Return observations and simulations for a given verification pair.
Selects
verification_pair.variablefrom each source’s dataset and returns the resulting DataArrays. This method is called by the verification pipeline at runtime to retrieve the correct data for one of the configured verification pairs.- Parameters:
verification_pair (VerificationPair)
- Return type:
- get_thresholds_array(variable)[source]#
Get the thresholds array for a given variable from the input dataset.
- get_verification_pair_node(verification_pair_id)[source]#
Get the DataTree node corresponding to a specific verification pair.
- get_aligned_input(verification_pair_id)[source]#
Get the input dataset for a specific verification pair.
- list_scores(verification_pair_id)[source]#
Get a list of available scores for a specific verification pair.
- add_score(verification_pair, result, name)[source]#
Add a score results to the datatree.
- Parameters:
verification_pair (VerificationPair)
name (str)
- Return type:
None
- add_staged_input_data(verification_pair, obs, sim)[source]#
Add input data to the datastore.
- Parameters:
verification_pair (VerificationPair)
obs (DataArray)
sim (DataArray)
- Return type:
None
- filter_nodes(*, include_input_data, include_aligned_input_data, include_output)[source]#
Filter the datatree based on the specified inclusion flags.
Builds a new DataTree containing only the parts of this DataTree selected by
config: the rawinput_datanode, and/or thealigned_inputandoutputnodes under each verification pair, in any combination.