veriflow.datasources.base#

Module with the base class that all datasources should inherit from.

Classes

`BaseDatasource`(config)	Class to inherit from, defines the required methods and attributes.
`BaseDatasourceConfig`(*, import_adapter, ...)	Base config for a datasource config.

class veriflow.datasources.base.BaseDatasource(config)[source]#

Class to inherit from, defines the required methods and attributes.

Parameters:: config (BaseDatasourceConfig)

kind: str = ''#

config_class#: alias of BaseDatasourceConfig

supported_data_types: ClassVar[set[DataType]] = {}#

property data_type: DataType#: Whether the instance represents sim or obs data.

property cache: ZarrCache | None#: Return the cache instance if caching is enabled, otherwise None.

abstract property configured_stations: set[str] | None#

Return the standardized internal station identifiers configured for this datasource.

This standardized format is needed for caching across different datasources. If your datasource implementation does not have any configurable stations, return None.

abstract property configured_variables: set[str] | None#

Return the standardized internal variable identifiers configured for this datasource.

This standardized format is needed for caching across different datasources. If your datasource implementation does not have any configurable variables, return None.

property configured_variables_internal: set[str] | None#

Return the standardized internal variable identifiers configured for this datasource.

This standardized format is needed for caching across different datasources. If your datasource implementation does not have any configurable variables, return None.

abstractmethod fetch_data()[source]#

Fetch data from datasource.

Return type:: Self

validate_fetched_data()[source]#

Validate that the dataset is consistent with the config.

Return type:: None

filter_dataset(dataset)[source]#

Filter the dataset on lead times and times outside the verification period.

Parameters:: dataset (Dataset)
Return type:: Dataset

fetch_validate_filter_cache(*, clear_cache=False)[source]#

High-level wrapper to fetch, validate, filter and apply id mapping to the dataset.

Parameters:: clear_cache (bool)
Return type:: Self

create_cache_request(cached_dataset)[source]#

Get the cache request based on the cached dataset and the config.

Parameters:: cached_dataset (Dataset)
Return type:: CacheRequest

classmethod from_config(raw_config)#

Initialize class from config dict.

Parameters:: raw_config (dict[str, Any])
Return type:: Self

static get_cached_data(cached_dataset, datasource)[source]#

Get the dataset from the cache based on the datasource configuration.

Parameters:

cached_dataset (Dataset)
datasource (BaseDatasource)

Return type:

Dataset

get_data()[source]#

Get data and make use of cache if configured.

Return type:: Self

class veriflow.datasources.base.BaseDatasourceConfig(*, import_adapter, source, data_type, general, id_mapping=None, **extra_data)[source]#

Base config for a datasource config.

Specific config definitions should inherit from this base class.

Parameters:

import_adapter (str)
source (Annotated[str, StringConstraints(strip_whitespace=None, to_upper=None, to_lower=None, strict=None, min_length=1, max_length=None, pattern=^[A-Za-z][A-Za-z0-9_]*$, ascii_only=None), MinLen(min_length=1)])
data_type (DataType)
general (Annotated[GeneralInfoConfig, SkipJsonSchema()])
id_mapping (Annotated[IdMappingConfig, SkipJsonSchema()] | None)
extra_data (Any)

import_adapter: str#

source: ", metadata=[MinLen(min_length=1)])]#

data_type: DataType#

general: Annotated[GeneralInfoConfig, SkipJsonSchema()]#

id_mapping: Annotated[IdMappingConfig, SkipJsonSchema()] | None#

property lead_times: LeadTimes | None#

property verification_period: TimePeriod#

property verification_period_on_frt: TimePeriod#

property verification_period_on_time: TimePeriod#