veriflow.datasources.zarr#

Read Zarr stores from local disk or S3.

Classes

Zarr(config)

A datasource for reading Zarr stores compatible with the internal datatree.

ZarrConfig(*, path[, group, auth_config, ...])

A Zarr config element.

class veriflow.datasources.zarr.Zarr(config)[source]#

A datasource for reading Zarr stores compatible with the internal datatree.

Wraps xarray.open_zarr() and supports both local filesystem paths and remote URLs (currently s3:// is exercised). For S3 stores, credentials are taken from a S3AuthConfig instance loaded from environment variables prefixed with S3_. Additional storage_options configured on the ZarrConfig are merged on top and forwarded to xr.open_zarr.

Note

The dataset must carry a data_type attribute that matches one of the supported data types; if the attribute is missing it will be set from the configuration.

Parameters:

config (ZarrConfig)

kind: str = 'zarr'#
config_class#

alias of ZarrConfig

supported_data_specs: ClassVar[set[tuple[DataType, SpatialType]]] = {(DataType.observed_historical, SpatialType.gridded), (DataType.observed_historical, SpatialType.point), (DataType.simulated_forecast_ensemble, SpatialType.gridded), (DataType.simulated_forecast_ensemble, SpatialType.point), (DataType.simulated_forecast_probabilistic, SpatialType.point), (DataType.simulated_forecast_single, SpatialType.gridded), (DataType.simulated_forecast_single, SpatialType.point), (DataType.threshold, SpatialType.point)}#
config: ZarrConfig#
property configured_stations: set[str] | None#

Return the internal station identifiers configured for this datasource.

This is needed for standardization of station identifiers across sources.

property cache: ZarrCache | None#

Return the cache instance if caching is enabled, otherwise None.

clamp_verification_period_on_frt_to_available_period(available)#

Clamp the configured verification period to the source’s currently available period.

Parameters:

available (TimePeriod)

Return type:

None

property configured_variables: set[str] | None#

Return the internal variable identifiers configured for this datasource.

This is needed for standardization of variable identifiers across sources.

property configured_variables_internal: set[str] | None#

Return the standardized internal variable identifiers configured for this datasource.

This standardized format is needed for caching across different datasources. If your datasource implementation does not have any configurable variables, return None.

create_cache_request(cached_dataset)#

Get the cache request based on the cached dataset and the config.

Parameters:

cached_dataset (Dataset)

Return type:

CacheRequest

property data_type: DataType#

Whether the instance represents simulations or observations data.

fetch_available_forecast_reference_time_period()#

Return the period actually available from the source right now, if resolvable.

Used to clamp an open-ended/rolling configured verification period (e.g. “up to now”) to what the source can currently serve, so operational runs don’t fail by requesting data that hasn’t been produced (yet) or has fallen outside the source’s retention. Returns None by default, meaning the configured verification period is used as-is.

When implemented, this method is used during caching. In a standard veriflow run, the period for which to fetch data is defined in the configuration. However, the actual available period from the datasource may differ, and this method allows Veriflow to adjust the verification period accordingly. As a result, we never request data outside the source’s actual availability and the cache is only updated with new data if it is available in the datasource.

Return type:

TimePeriod | None

fetch_validate_filter_cache(*, clear_cache=False)#

High-level wrapper to fetch, validate, filter and apply id mapping to the dataset.

Parameters:

clear_cache (bool)

Return type:

Self

filter_dataset(dataset)#

Filter the dataset on lead times and times outside the verification period.

Parameters:

dataset (Dataset)

Return type:

Dataset

classmethod from_config(raw_config)#

Initialize class from config dict.

Parameters:

raw_config (dict[str, Any])

Return type:

Self

static get_cached_data(cached_dataset, datasource)#

Get the dataset from the cache based on the datasource configuration.

Parameters:
Return type:

Dataset

get_data()#

Get data and make use of cache if configured.

Return type:

Self

property spatial_type: SpatialType#

Spatial structure (point/gridded) of the datasource’s data.

validate_fetched_data()#

Validate that the dataset is consistent with the config.

Return type:

None

dataset: Dataset#
fetch_data()[source]#

Retrieve the configured Zarr store as an xarray Dataset.

Return type:

Self

class veriflow.datasources.zarr.ZarrConfig(*, path, group=None, auth_config=None, storage_options=None, consolidated=None, import_adapter, source_id, data_type, spatial_type=SpatialType.point, general, id_mapping=None, crs=None, variables=None, stations=None, **extra_data)#

A Zarr config element.

Reads a single Zarr store via xarray.open_zarr(). The path may point to a local directory or a remote location (e.g. s3://bucket/key/store.zarr). When the path uses an s3:// URL, credentials and connection details are taken from auth_config (an S3AuthConfig), which is populated from environment variables prefixed with S3_. Additional storage_options are merged on top of the ones derived from auth_config and forwarded to xr.open_zarr.

Parameters:
  • path (Annotated[str, MinLen(min_length=1)])

  • group (str | None)

  • auth_config (S3AuthConfig | None)

  • storage_options (dict[str, str] | None)

  • consolidated (bool | None)

  • import_adapter (Literal[DataSourceKind.ZARR])

  • source_id (Annotated[str, StringConstraints(strip_whitespace=None, to_upper=None, to_lower=None, strict=None, min_length=1, max_length=None, pattern=^[A-Za-z][A-Za-z0-9_]*$, ascii_only=None), MinLen(min_length=1)])

  • data_type (DataType)

  • spatial_type (SpatialType)

  • general (Annotated[GeneralInfoConfig, SkipJsonSchema()])

  • id_mapping (Annotated[IdMappingConfig, SkipJsonSchema()] | None)

  • crs (Annotated[str, FieldInfo(annotation=NoneType, required=True, description="A coordinate reference system as a string compatible with pyproj.CRS.from_string, i.e. a PROJ string, a CRS WKT string, or an authority string such as 'EPSG:4326'. The string is not parsed at configuration time so that pyproj remains an optional dependency; it is only parsed when an actual coordinate transformation is required.", metadata=[MinLen(min_length=1)])] | None)

  • variables (Annotated[list[str] | None, MinLen(min_length=1)])

  • stations (Annotated[list[str] | None, MinLen(min_length=1)])

  • extra_data (Any)

is_remote_path()#

Return True if path looks like a remote/fsspec URL (e.g. s3://).

Return type:

bool

property lead_times: LeadTimes | None#
property resolved_storage_options: dict[str, object] | None#

Build the final storage_options for xr.open_zarr/to_zarr.

Merges ‘auth_config’ (auto-populated from S3_-prefixed environment variables for remote paths) with the explicit ‘storage_options’ override. Returns None for local paths so that xarray opens the store directly from the local filesystem. This is the single place that should ever build storage_options from a BaseZarrConfig - duplicating this logic elsewhere risks silently dropping ‘auth_config’ (and thus falling back to boto3’s default credential discovery instead of the configured S3_-prefixed credentials).

validate_zarr_path_accessible()#

Check that a local cache dir exists, or initialize S3 auth for remote paths.

Return type:

Self

property verification_period: TimePeriod#
property verification_period_on_frt: TimePeriod#
property verification_period_on_time: TimePeriod#
import_adapter: Literal[DataSourceKind.ZARR]#
crs: Annotated[Annotated[str, FieldInfo(annotation=NoneType, required=True, description="A coordinate reference system as a string compatible with pyproj.CRS.from_string, i.e. a PROJ string, a CRS WKT string, or an authority string such as 'EPSG:4326'. The string is not parsed at configuration time so that pyproj remains an optional dependency; it is only parsed when an actual coordinate transformation is required.", metadata=[MinLen(min_length=1)])] | None, FieldInfo(annotation=NoneType, required=False, default=None, description="Coordinate reference system of the data in the store. Optional: when omitted, the CRS is read from the dataset's 'crs' attribute if present. When set, it takes precedence and is written onto the dataset's 'crs' attribute. Required for gridded data that does not already carry a 'crs' attribute.")]#
variables: Annotated[list[str] | None, FieldInfo(annotation=NoneType, required=True, metadata=[MinLen(min_length=1)])]#
stations: Annotated[list[str] | None, FieldInfo(annotation=NoneType, required=True, metadata=[MinLen(min_length=1)])]#
source_id#
data_type#
spatial_type#
general#
id_mapping#
path#
group#
auth_config#
storage_options#
consolidated#