veriflow.datasources.netcdf#
Read and write NetCDF files in a fews compatible format.
Classes
|
A datasource for reading NetCDF files compatible with the internal datamodel. |
|
A NetCDF config element. |
- class veriflow.datasources.netcdf.NetCDF(config)[source]#
A datasource for reading NetCDF files compatible with the internal datamodel.
You can validate that the NetCDF file satisfies the internal datamodel using the following example:
import xarray as xr from veriflow.datasources import validate_input_data dataset = xr.open_dataset("path/to/netcdf/file/example.nc") validated_data = validate_input_data(dataset)
Note
The data variables in the NetCDF file represent the physical variables to be verified. Each data variable should carry a
unitsattribute. The dataset must carry adata_typeattribute that matches one of the supported data types (it will be set from the configuration if missing).- Parameters:
config (NetCDFConfig)
- config_class#
alias of
NetCDFConfig
- supported_data_specs: ClassVar[set[tuple[DataType, SpatialType]]] = {(DataType.observed_historical, SpatialType.point), (DataType.simulated_forecast_ensemble, SpatialType.point), (DataType.simulated_forecast_probabilistic, SpatialType.point), (DataType.simulated_forecast_single, SpatialType.point), (DataType.threshold, SpatialType.point)}#
- config: NetCDFConfig#
- property configured_stations: set[str] | None#
Return the internal station identifiers configured for this datasource.
This is needed for standardization of station identifiers across sources.
- clamp_verification_period_on_frt_to_available_period(available)#
Clamp the configured verification period to the source’s currently available period.
- Parameters:
available (TimePeriod)
- Return type:
None
- property configured_variables: set[str] | None#
Return the internal variable identifiers configured for this datasource.
This is needed for standardization of variable identifiers across sources.
- property configured_variables_internal: set[str] | None#
Return the standardized internal variable identifiers configured for this datasource.
This standardized format is needed for caching across different datasources. If your datasource implementation does not have any configurable variables, return None.
- create_cache_request(cached_dataset)#
Get the cache request based on the cached dataset and the config.
- Parameters:
cached_dataset (Dataset)
- Return type:
- fetch_available_forecast_reference_time_period()#
Return the period actually available from the source right now, if resolvable.
Used to clamp an open-ended/rolling configured verification period (e.g. “up to now”) to what the source can currently serve, so operational runs don’t fail by requesting data that hasn’t been produced (yet) or has fallen outside the source’s retention. Returns None by default, meaning the configured verification period is used as-is.
When implemented, this method is used during caching. In a standard veriflow run, the period for which to fetch data is defined in the configuration. However, the actual available period from the datasource may differ, and this method allows Veriflow to adjust the verification period accordingly. As a result, we never request data outside the source’s actual availability and the cache is only updated with new data if it is available in the datasource.
- Return type:
TimePeriod | None
- fetch_validate_filter_cache(*, clear_cache=False)#
High-level wrapper to fetch, validate, filter and apply id mapping to the dataset.
- filter_dataset(dataset)#
Filter the dataset on lead times and times outside the verification period.
- classmethod from_config(raw_config)#
Initialize class from config dict.
- static get_cached_data(cached_dataset, datasource)#
Get the dataset from the cache based on the datasource configuration.
- Parameters:
cached_dataset (Dataset)
datasource (BaseDatasource)
- Return type:
- property spatial_type: SpatialType#
Spatial structure (point/gridded) of the datasource’s data.
- validate_fetched_data()#
Validate that the dataset is consistent with the config.
- Return type:
None
- class veriflow.datasources.netcdf.NetCDFConfig(*, directory, filename_glob, import_adapter, source_id, data_type, spatial_type=SpatialType.point, general, id_mapping=None, stations=None, variables=None, **extra_data)#
A NetCDF config element.
- Parameters:
directory (str)
filename_glob (str)
import_adapter (Literal[DataSourceKind.NETCDF])
source_id (Annotated[str, StringConstraints(strip_whitespace=None, to_upper=None, to_lower=None, strict=None, min_length=1, max_length=None, pattern=^[A-Za-z][A-Za-z0-9_]*$, ascii_only=None), MinLen(min_length=1)])
data_type (DataType)
spatial_type (SpatialType)
general (Annotated[GeneralInfoConfig, SkipJsonSchema()])
id_mapping (Annotated[IdMappingConfig, SkipJsonSchema()] | None)
stations (Annotated[list[str] | None, MinLen(min_length=1)])
variables (Annotated[list[str] | None, MinLen(min_length=1)])
extra_data (Any)
- property paths: Iterator[Path]#
Return all filepaths as Path objects in a deterministic sorted order.
- property verification_period: TimePeriod#
- property verification_period_on_frt: TimePeriod#
- property verification_period_on_time: TimePeriod#
- stations: Annotated[list[str] | None, FieldInfo(annotation=NoneType, required=True, metadata=[MinLen(min_length=1)])]#
- variables: Annotated[list[str] | None, FieldInfo(annotation=NoneType, required=True, metadata=[MinLen(min_length=1)])]#
- source_id#
- data_type#
- spatial_type#
- general#
- id_mapping#
- directory#
- filename_glob#