veriflow.datasources.zarr#
Read Zarr stores from local disk or S3.
Classes
|
A datasource for reading Zarr stores compatible with the internal datatree. |
|
A Zarr config element. |
- class veriflow.datasources.zarr.Zarr(config)[source]#
A datasource for reading Zarr stores compatible with the internal datatree.
Wraps
xarray.open_zarr()and supports both local filesystem paths and remote URLs (currentlys3://is exercised). For S3 stores, credentials are taken from aS3AuthConfiginstance loaded from environment variables prefixed withS3_. Additionalstorage_optionsconfigured on theZarrConfigare merged on top and forwarded toxr.open_zarr.Note
The dataset must carry a
data_typeattribute that matches one of the supported data types; if the attribute is missing it will be set from the configuration.- Parameters:
config (ZarrConfig)
- config_class#
alias of
ZarrConfig
- supported_data_specs: ClassVar[set[tuple[DataType, SpatialType]]] = {(DataType.observed_historical, SpatialType.gridded), (DataType.observed_historical, SpatialType.point), (DataType.simulated_forecast_ensemble, SpatialType.gridded), (DataType.simulated_forecast_ensemble, SpatialType.point), (DataType.simulated_forecast_probabilistic, SpatialType.point), (DataType.simulated_forecast_single, SpatialType.gridded), (DataType.simulated_forecast_single, SpatialType.point), (DataType.threshold, SpatialType.point)}#
- config: ZarrConfig#
- property configured_stations: set[str] | None#
Return the internal station identifiers configured for this datasource.
This is needed for standardization of station identifiers across sources.
- clamp_verification_period_on_frt_to_available_period(available)#
Clamp the configured verification period to the source’s currently available period.
- Parameters:
available (TimePeriod)
- Return type:
None
- property configured_variables: set[str] | None#
Return the internal variable identifiers configured for this datasource.
This is needed for standardization of variable identifiers across sources.
- property configured_variables_internal: set[str] | None#
Return the standardized internal variable identifiers configured for this datasource.
This standardized format is needed for caching across different datasources. If your datasource implementation does not have any configurable variables, return None.
- create_cache_request(cached_dataset)#
Get the cache request based on the cached dataset and the config.
- Parameters:
cached_dataset (Dataset)
- Return type:
- fetch_available_forecast_reference_time_period()#
Return the period actually available from the source right now, if resolvable.
Used to clamp an open-ended/rolling configured verification period (e.g. “up to now”) to what the source can currently serve, so operational runs don’t fail by requesting data that hasn’t been produced (yet) or has fallen outside the source’s retention. Returns None by default, meaning the configured verification period is used as-is.
When implemented, this method is used during caching. In a standard veriflow run, the period for which to fetch data is defined in the configuration. However, the actual available period from the datasource may differ, and this method allows Veriflow to adjust the verification period accordingly. As a result, we never request data outside the source’s actual availability and the cache is only updated with new data if it is available in the datasource.
- Return type:
TimePeriod | None
- fetch_validate_filter_cache(*, clear_cache=False)#
High-level wrapper to fetch, validate, filter and apply id mapping to the dataset.
- filter_dataset(dataset)#
Filter the dataset on lead times and times outside the verification period.
- classmethod from_config(raw_config)#
Initialize class from config dict.
- static get_cached_data(cached_dataset, datasource)#
Get the dataset from the cache based on the datasource configuration.
- Parameters:
cached_dataset (Dataset)
datasource (BaseDatasource)
- Return type:
- property spatial_type: SpatialType#
Spatial structure (point/gridded) of the datasource’s data.
- validate_fetched_data()#
Validate that the dataset is consistent with the config.
- Return type:
None
- class veriflow.datasources.zarr.ZarrConfig(*, path, group=None, auth_config=None, storage_options=None, consolidated=None, import_adapter, source_id, data_type, spatial_type=SpatialType.point, general, id_mapping=None, crs=None, variables=None, stations=None, **extra_data)#
A Zarr config element.
Reads a single Zarr store via
xarray.open_zarr(). Thepathmay point to a local directory or a remote location (e.g.s3://bucket/key/store.zarr). When the path uses ans3://URL, credentials and connection details are taken fromauth_config(anS3AuthConfig), which is populated from environment variables prefixed withS3_. Additionalstorage_optionsare merged on top of the ones derived fromauth_configand forwarded toxr.open_zarr.- Parameters:
group (str | None)
auth_config (S3AuthConfig | None)
consolidated (bool | None)
import_adapter (Literal[DataSourceKind.ZARR])
source_id (Annotated[str, StringConstraints(strip_whitespace=None, to_upper=None, to_lower=None, strict=None, min_length=1, max_length=None, pattern=^[A-Za-z][A-Za-z0-9_]*$, ascii_only=None), MinLen(min_length=1)])
data_type (DataType)
spatial_type (SpatialType)
general (Annotated[GeneralInfoConfig, SkipJsonSchema()])
id_mapping (Annotated[IdMappingConfig, SkipJsonSchema()] | None)
crs (Annotated[str, FieldInfo(annotation=NoneType, required=True, description="A coordinate reference system as a string compatible with pyproj.CRS.from_string, i.e. a PROJ string, a CRS WKT string, or an authority string such as 'EPSG:4326'. The string is not parsed at configuration time so that pyproj remains an optional dependency; it is only parsed when an actual coordinate transformation is required.", metadata=[MinLen(min_length=1)])] | None)
variables (Annotated[list[str] | None, MinLen(min_length=1)])
stations (Annotated[list[str] | None, MinLen(min_length=1)])
extra_data (Any)
- is_remote_path()#
Return True if
pathlooks like a remote/fsspec URL (e.g.s3://).- Return type:
- property resolved_storage_options: dict[str, object] | None#
Build the final storage_options for
xr.open_zarr/to_zarr.Merges ‘auth_config’ (auto-populated from
S3_-prefixed environment variables for remote paths) with the explicit ‘storage_options’ override. ReturnsNonefor local paths so that xarray opens the store directly from the local filesystem. This is the single place that should ever build storage_options from aBaseZarrConfig- duplicating this logic elsewhere risks silently dropping ‘auth_config’ (and thus falling back to boto3’s default credential discovery instead of the configuredS3_-prefixed credentials).
- validate_zarr_path_accessible()#
Check that a local cache dir exists, or initialize S3 auth for remote paths.
- Return type:
- property verification_period: TimePeriod#
- property verification_period_on_frt: TimePeriod#
- property verification_period_on_time: TimePeriod#
- crs: Annotated[Annotated[str, FieldInfo(annotation=NoneType, required=True, description="A coordinate reference system as a string compatible with pyproj.CRS.from_string, i.e. a PROJ string, a CRS WKT string, or an authority string such as 'EPSG:4326'. The string is not parsed at configuration time so that pyproj remains an optional dependency; it is only parsed when an actual coordinate transformation is required.", metadata=[MinLen(min_length=1)])] | None, FieldInfo(annotation=NoneType, required=False, default=None, description="Coordinate reference system of the data in the store. Optional: when omitted, the CRS is read from the dataset's 'crs' attribute if present. When set, it takes precedence and is written onto the dataset's 'crs' attribute. Required for gridded data that does not already carry a 'crs' attribute.")]#
- variables: Annotated[list[str] | None, FieldInfo(annotation=NoneType, required=True, metadata=[MinLen(min_length=1)])]#
- stations: Annotated[list[str] | None, FieldInfo(annotation=NoneType, required=True, metadata=[MinLen(min_length=1)])]#
- source_id#
- data_type#
- spatial_type#
- general#
- id_mapping#
- path#
- group#
- auth_config#
- storage_options#
- consolidated#