veriflow.cache.cache#
Zarr cache implementation.
Classes
|
The Zarr cache. |
|
Configuration for the veriflow cache. |
|
Determine what requested data is missing from the cache. |
|
Helper class to track requested and cached data and what data is missing from the cache. |
- class veriflow.cache.cache.ZarrCache(config)[source]#
The Zarr cache.
In veriflow, you can cache datasets fetched from any datasource using a Zarr store. The cache enables incremental appends of new data along a single dimension (e.g. time, lead time, or station), and can be materialized on both a local filesystem and remote object storage (e.g. S3). An example use-case for using the cache is the operational application of veriflow, where new forecast data is generated and added to the cache on a daily basis, or when working with multiple people on a centralized verification project, where high-speed access to a shared cache is desired.
For now, the cache is only used to store datasets fetched from other datasources, and is not used to cache computation results.
The cache is configured in the
veriflow.configuration.base.GeneralInfoConfig.cachefield. All caching logic is handled in theveriflow.datasources.base.BaseDatasource.get_data()method, which consults the cache configuration and uses this ZarrCache class to read/write cached datasets as needed.The current cache can handle the following scenarios: 1. Historical data with missing time steps, variables, or stations. 2. Forecast data with missing forecast reference times, lead times, variables, or stations
The current implementation has the following limitations: 1. When data is requested that is missing in the cache along more than one dimension, the cache will not retrieve data from the cache, but will instead fetch all data from the datasource. 2. The current cache does not (yet) support caching of computation results. 3. Large Zarr files on local filesystems may be slow to open due to the overhead of opening many small files. This is a known limitation of Zarr and is not specific to this implementation. However, using consolidated metadata (
consolidated=True) can help mitigate this issue by reducing the number of files that need to be opened. However, datasets with a size of below 1TB should be fine. See: zarr-developers/zarr-python#86. Also note that caching is optional and can always be turned off.The layout of the Zarr cache is as follows, where each source has its own subgroup under the
datasetsgroup. The source name is identical to thesourcefield in the datasource configuration.example_cache.zarr/ ├── datasets/ │ ├── observations/ │ │ ├── temperature/ │ │ ├── precipitation/ │ │ └── discharge/ │ └── forecasts_model_a │ │ ├── temperature/ │ │ ├── precipitation/ │ │ └── discharge/ │ └── forecasts_model_b │ │ ├── temperature/ │ │ ├── precipitation/ │ │ └── discharge/
A flowchart of the caching logic is shown below, which is implemented in the
veriflow.datasources.base.BaseDatasource.get_data()method.flowchart TD A([get_data]) --> B{Cache configured and<br/>data type cacheable?} B -- no --> F[Fetch all data<br/>from datasource] B -- yes --> C[Open cached dataset<br/>from the Zarr store] C --> D{Any data cached<br/>for this source?} D -- no --> F D -- yes --> E[Build CacheRequest and<br/>determine missing dimensions] E --> G{How many dimensions<br/>are missing?} G -- none --> H["Load requested data from cache<br/>(full hit)"] G -- one --> I["Fetch missing, append to cache,<br/>read back dataset"] G -- many --> F F --> W[(Write to cache<br/>if writable)] I --> W H --> R([Return dataset]) W --> R- Parameters:
config (ZarrCacheConfig)
- append(new_dataset, source, append_dim=None)[source]#
Incrementally add
new_datasetto the cache along a single dimension.Only the coordinate index of the existing store is read to skip values that are already cached; the cached data itself is never loaded into memory, keeping the cache lazy-friendly. New coordinate values are appended along
dim(append_dim), new variables are added in place (mode="a"), and an empty store is created from scratch. Duplicate coordinate values withinnew_datasetitself alongappend_dimare dropped (keeping the first occurrence) before writing, since the store is never deduplicated after the fact.
- class veriflow.cache.cache.ZarrCacheConfig(*, path, group=None, auth_config=None, storage_options=None, consolidated=None, read_write_mode=ReadWriteMode.read)[source]#
Configuration for the veriflow cache.
The cache will be materialized as a Zarr store on the local filesystem or remote object storage (e.g. S3). When configured, the cache will be used to store and retrieve datasets from any datasource. For example: when requesting forecast data from a datasource, and part of the data is already cached, the cache will be used to retrieve the cached data and only the missing data will be fetched from the datasource.
- Parameters:
- read_write_mode: ReadWriteMode#
- is_remote_path()#
Return True if
pathlooks like a remote/fsspec URL (e.g.s3://).- Return type:
- property resolved_storage_options: dict[str, object] | None#
Build the final storage_options for
xr.open_zarr/to_zarr.Merges ‘auth_config’ (auto-populated from
S3_-prefixed environment variables for remote paths) with the explicit ‘storage_options’ override. ReturnsNonefor local paths so that xarray opens the store directly from the local filesystem. This is the single place that should ever build storage_options from aBaseZarrConfig- duplicating this logic elsewhere risks silently dropping ‘auth_config’ (and thus falling back to boto3’s default credential discovery instead of the configuredS3_-prefixed credentials).
- validate_zarr_path_accessible()#
Check that a local cache dir exists, or initialize S3 auth for remote paths.
- Return type:
- path#
- group#
- auth_config#
- storage_options#
- consolidated#
- class veriflow.cache.cache.CacheRequest(*, variables, stations, time_period=None, forecast_reference_time_period=None, lead_times=None)[source]#
Determine what requested data is missing from the cache.
Handles both historical requests (
time_period) and forecast requests (forecast_reference_time_period+lead_times); fields that do not apply stayNone.- Parameters:
variables (DataRequest)
stations (DataRequest)
time_period (DataRequest | None)
forecast_reference_time_period (DataRequest | None)
lead_times (DataRequest | None)
- variables: DataRequest#
- stations: DataRequest#
- time_period: DataRequest | None#
- forecast_reference_time_period: DataRequest | None#
- lead_times: DataRequest | None#
- property missing_count: int#
Return the number of dimensions along which data is missing from the cache.
- split_config(datasource)[source]#
Split the datasource into a datasource-fetch and a cache-fetch datasource.
Only splits when exactly one dimension is missing from the cache; otherwise returns
Noneso that all data is fetched from the datasource.- Parameters:
datasource (BaseDatasource)
- Return type:
tuple[BaseDatasource, BaseDatasource] | None
- class veriflow.cache.cache.DataRequest(*, requested, cached)[source]#
Helper class to track requested and cached data and what data is missing from the cache.
- Parameters:
requested (set[str] | TimePeriod | LeadTimes | None)
cached (set[str] | TimePeriod | LeadTimes | None)
- validate_requested_and_cached()[source]#
Validate that the requested and cached values are of the same type.
- Return type:
- static find_missing_and_available_time_period(requested, cached)[source]#
Find the missing time period between the requested and cached time periods.
This helper function is used to determine what time period needs to be fetched from the datasource given what is already available in the cache. The returned tuple contains the missing time period and the available time period, respectively.
The requested period
Ris compared against the cached periodCon the time axis. The five possible cases and their outcomes are shown below (#marks the extent of each period):time ---> (1) R inside C -> full hit R #### C ########## missing: (none) available: R (2) R and C disjoint R #### C #### missing: R available: (none) (3) C inside R R ########## C #### missing: R available: (none) (4) left overlap (R starts before C, C covers R's end) R ###### C ######## missing: R.start .. C.start available: C.start .. R.end (5) right overlap (C covers R's start, R ends after C) R ######## C ###### missing: C.end .. R.end available: R.start .. C.endAn edge shared exactly between R and C (e.g.
R.end == C.end) counts as covered on that side, not as a gap - this matters because R.end routinely resolves to the same rolling-archive boundary that was already cached on a previous run, and treating that as “still missing” would discard an already-cached range and re-fetch everything.- Parameters:
requested (TimePeriod)
cached (TimePeriod)
- Return type:
tuple[TimePeriod | None, TimePeriod | None]