Tip

For an interactive online version click here:

Update forcing data#

Once you have a DELWAQ model, you may want to update your model in order to add new emission data, add sample locations, use different hydrological forcing data, create and run different scenarios etc.

With HydroMT, you can easily read your model and update one or several components of your model using the update function of the command line interface (CLI). Here are the steps and some examples on how to update the model forcing.

All lines in this notebook which starts with ! are executed from the command line. Within the notebook environment the logging messages are shown after completion. You can also copy these lines and paste them in your shell to get more feedback.

This notebook supposes that you already run the delwaq build EM example notebook and that you have a EM_test_full D-Emissions model available in the examples folder.

Import packages#

In this notebook, we will use some functions of HydroMT to visualize available data. Here are the libaries to import.

[1]:

import xarray as xr

[2]:

# setup logging
from  hydromt.log import setuplog
logger = setuplog("update_model_forcing", log_level=10)

2024-06-21 02:54:35,368 - update_model_forcing - log - INFO - HydroMT version: 0.10.0

Hydrological forcing data from Wflow#

D-Emissions or D-Water Quality models can use meteo/hydrological input data from Wflow. The steps to get these data available for your model are:

List the required meteo/hydro fluxes in the Wflow TOML configuration file in order to save them.
Run the Wflow model.
Add the outputs of the Wflow model as a new data source for HydroMT.
Run HydroMT (specifically the setup_hydrology_forcing component) to convert the wflow output to DELWAQ dynamic input.

More information on the meteo/hydrological forcing link between Wflow and DELWAQ is available in docs (Wflow_outputs).

Selection of the Wfow outputs#

For our D-Emissions model, we will use hydrological data from the wflow_piave hydrologic model. For a D-Emissions model, the required meteo/hydrological inputs are:

precipitation
the amount of the precipitation that infiltrates into the soil from unpaved areas
the amount of the precipitation that goes directly to surface runoff from paved areas
the amount of the precipitation that goes directly to surface runoff from unpaved areas
exfiltration from the saturated and unsaturated store of the soil
volumetric water content of the soil pores in the root zone ie root zone soil moisture

If you wish to include transport to the river (overland flow and soil) in D-Emission, then also:

overland flow
subsurface flow

In order to get these data from Wflow, you need to save these fluxes after the model run. These output fluxes are specified in the Wflow TOML configuration file.

To set these options, you can either edit the TOML file manually or use HydroMT to help you do the trick using the hydromt update CLI API!

Here is the HydroMT configuration file needed to update the Wflow model (note in that file both fluxes for demission and for delwaq models are listed):

[3]:

fn_yml = 'wflow_update.yml'
with open(fn_yml, 'r') as f:
    txt = f.read()
print(txt)

setup_config:
  starttime: 2010-02-02T00:00:00
  endtime: 2010-02-10T00:00:00
  timestepsecs: 86400
  # Variables to save for a coupling between wflow and demission + delwaq
  output.vertical.precipitation: "precip"
  output.vertical.actinfilt: "infilt"
  output.vertical.excesswatersoil: "runUnp"
  output.vertical.excesswaterpath: "runPav"
  output.vertical.exfiltustore: "exfiltustore"
  output.vertical.exfiltsatwater: "exfiltsatstore"
  output.vertical.vwc_percroot: "vwcproot"
  output.lateral.river.q_av: "q_river"
  output.lateral.river.volume: "vol_river"
  output.lateral.land.q_av: "q_land"
  output.lateral.land.to_river: "land_to_river"
  output.lateral.subsurface.ssf: "q_ssf"
  output.lateral.subsurface.to_river: "ssf_to_river"
  # Only add if reservoirs in the model
  output.lateral.river.reservoir.volume: "vol_reservoir"
  output.lateral.river.reservoir.totaloutflow: "q_reservoir"
  # Only add if lakes in the model
  output.lateral.river.lake.storage: "vol_lake"
  output.lateral.river.lake.totaloutflow: "q_lake"
  # Only add if 1d floodplain in the model
  # output.lateral.river.floodplain: "vol_floodplain"

write_config:

And the hydromt update CLI to use:

[4]:

! hydromt update wflow wflow_piave -i wflow_update.yml --fo -vvv

2024-06-21 02:54:37,200 - update - log - DEBUG - Appending log messages to file /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/_examples/wflow_piave/hydromt.log.
2024-06-21 02:54:37,200 - update - log - INFO - HydroMT version: 0.10.0
2024-06-21 02:54:37,200 - update - main - INFO - Updating wflow model at /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/_examples/wflow_piave (r+).
2024-06-21 02:54:37,200 - update - main - INFO - Output dir: /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/_examples/wflow_piave
2024-06-21 02:54:37,200 - update - main - INFO - User settings:
2024-06-21 02:54:37,222 - update - model_api - WARNING - Model dir already exists and files might be overwritten: /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/_examples/wflow_piave/staticgeoms.
2024-06-21 02:54:37,222 - update - model_api - WARNING - Model dir already exists and files might be overwritten: /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/_examples/wflow_piave/run_default.
2024-06-21 02:54:37,222 - update - model_api - INFO - Initializing wflow model from hydromt_wflow (v0.6.0).
2024-06-21 02:54:37,222 - update - data_catalog - INFO - Parsing data catalog from /usr/share/miniconda/envs/hydromt-delwaq/lib/python3.11/site-packages/hydromt_wflow/data/parameters_data.yml
2024-06-21 02:54:37,234 - update - model_api - DEBUG - Model config read from /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/_examples/wflow_piave/wflow_sbm.toml
2024-06-21 02:54:37,234 - update - wflow - INFO - Reading model staticgeom files.
2024-06-21 02:54:37,260 - update - wflow - INFO - Read grid from /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/_examples/wflow_piave/staticmaps.nc
2024-06-21 02:54:37,348 - update - model_api - DEBUG - Setting model config options.
2024-06-21 02:54:37,349 - update - model_api - INFO - write_config.config_name: None
2024-06-21 02:54:37,349 - update - model_api - INFO - write_config.config_root: None
2024-06-21 02:54:37,349 - update - model_api - INFO - Writing model config to /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/_examples/wflow_piave/wflow_sbm.toml

The example above means the following: run hydromt with:

update wflow: i.e. update a wflow model
wflow_piave_subbasin: model folder to update
-i wflow_update_forcing.yml: setup configuration file containing the components to update and their different options
--fo: allow to overwrite an existing model.
-v: give some extra verbosity (2 * v) to display feedback on screen. Now debug messages are provided.

The next step would then be to run the wflow_piave model using our updated TOML configuration file.

Available hydrological data from Wflow#

For our D-Emissions model, we will use hydrological data from the wflow_piave hydrologic model.

The model was already run beforehand and the wflow_piave outputs are stored in the wflow_piave/run_default/output.nc in NetCDF format. Let’s have a look at the model outputs and see what data is available and for which periods.

Note: You can also inspect the file using Panoply or QGIS.

[5]:

# Open the file
wflow_output_fn = 'wflow_piave/run_default/output.nc'
outputs = xr.open_dataset(wflow_output_fn, chunks={"time": 10})
#Print available variables and start and end time
print(f"Available outputs from wflow: {outputs.data_vars}")
times = outputs.time.values
print(f"Outputs available from {times[0]} to {times[-1]}")

Available outputs from wflow: Data variables:
    infilt          (time, lat, lon) float32 98kB dask.array<chunksize=(8, 53, 58), meta=np.ndarray>
    vwcproot        (time, lat, lon) float32 98kB dask.array<chunksize=(8, 53, 58), meta=np.ndarray>
    vol_reservoir   (time, lat, lon) float32 98kB dask.array<chunksize=(8, 53, 58), meta=np.ndarray>
    runPav          (time, lat, lon) float32 98kB dask.array<chunksize=(8, 53, 58), meta=np.ndarray>
    vol_river       (time, lat, lon) float32 98kB dask.array<chunksize=(8, 53, 58), meta=np.ndarray>
    precip          (time, lat, lon) float32 98kB dask.array<chunksize=(8, 53, 58), meta=np.ndarray>
    exfiltustore    (time, lat, lon) float32 98kB dask.array<chunksize=(8, 53, 58), meta=np.ndarray>
    runUnp          (time, lat, lon) float32 98kB dask.array<chunksize=(8, 53, 58), meta=np.ndarray>
    exfiltsatstore  (time, lat, lon) float32 98kB dask.array<chunksize=(8, 53, 58), meta=np.ndarray>
    q_land          (time, lat, lon) float32 98kB dask.array<chunksize=(8, 53, 58), meta=np.ndarray>
    vol_lake        (time, lat, lon) float32 98kB dask.array<chunksize=(8, 53, 58), meta=np.ndarray>
    land_to_river   (time, lat, lon) float32 98kB dask.array<chunksize=(8, 53, 58), meta=np.ndarray>
    vol_land        (time, lat, lon) float32 98kB dask.array<chunksize=(8, 53, 58), meta=np.ndarray>
    q_river         (time, lat, lon) float32 98kB dask.array<chunksize=(8, 53, 58), meta=np.ndarray>
    ssf_to_river    (time, lat, lon) float32 98kB dask.array<chunksize=(8, 53, 58), meta=np.ndarray>
    q_reservoir     (time, lat, lon) float32 98kB dask.array<chunksize=(8, 53, 58), meta=np.ndarray>
    q_ssf           (time, lat, lon) float32 98kB dask.array<chunksize=(8, 53, 58), meta=np.ndarray>
    q_lake          (time, lat, lon) float32 98kB dask.array<chunksize=(8, 53, 58), meta=np.ndarray>
Outputs available from 2010-02-03T00:00:00.000000000 to 2010-02-10T00:00:00.000000000

We recognize from above all the variables and names we set previously in the wflow_update.yml file. All the required variables are present and available for 9 days in February 2010.

Registering the wflow outputs as a HydroMT data source#

As we are using hydrological forcing from a user defined run of the wflow model, the wflow output format and attributes can be different depending on the user settings. Wflow outputs are then considered as local or user data source and therefore need to be added to the HydroMT list of data sources using a local yaml library file.

Below you can see the local_sources.yml file corresponding to our wflow outputs:

[6]:

fn_yml = 'local_sources.yml'
with open(fn_yml, 'r') as f:
    txt = f.read()
print(txt)

wflow_output_em:
  path: ./wflow_piave/run_default/output.nc
  crs: 4326
  data_type: RasterDataset
  driver: netcdf
  driver_kwargs:
    chunks:
      lat: -1
      lon: -1
      time: 1
    preprocess: harmonise_dims
  meta:
    category: hydro
  rename:
    q_ssf: q_ss
    ssf_to_river: exfiltssf
  unit_mult:
    q_ss: 0.0000115741
    exfiltssf: 0.0000115741
  attrs:
    precip:
      unit: mm
    infilt:
      unit: mm
    runPav:
      unit: mm
    runUnp:
      unit: mm
    exfiltustore:
      unit: mm
    exfiltsatstore:
      unit: mm
    exfiltssf:
      unit: m3/s
    vwcproot:
      unit: "%"
    q_land:
      unit: m3/s
    q_ss:
      unit: m3/s

wflow_output_wq:
  path: ./wflow_piave/run_default/output.nc
  crs: 4326
  data_type: RasterDataset
  driver: netcdf
  driver_kwargs:
    chunks:
      lat: -1
      lon: -1
      time: 1
    preprocess: harmonise_dims
  meta:
    category: hydro
  rename:
    q_river: sfw>sfw_1
    q_reservoir: sfw>sfw_2
    q_lake: sfw>sfw_3
    vol_river: sfw_1
    vol_reservoir: sfw_2
    vol_lake: sfw_3
    # vol_floodplain: sfw_4
    land_to_river: land>sfw
    ssf_to_river: soil>sfw
  unit_mult:
    soil>sfw: 0.0000115741
    sfw>sfw_2: 0.0000115741
    sfw>sfw_3: 0.0000115741
  attrs:
    sfw_1:
      unit: m3
    sfw_2:
      unit: m3
    sfw_3:
      unit: m3
    # sfw_4:
    #   unit: m3
    sfw>sfw_1:
      unit: m3/s
    sfw>sfw_2:
      unit: m3/s
    sfw>sfw_3:
      unit: m3/s
    land>sfw:
      unit: m3/s
    soil>sfw:
      unit: m3/s

Here are some explanations about the file. You can have a look at the HydroMT yaml data libary documentation for more information.

The first thing to define in the yaml library is the name of the data source you want to add in HydroMT. For example, here we use the name wflow_output_em for the EM variables definition and wflow_output_wq for the WQ variables. This name is important and is used in the HydroMT config .yml file to tell HydroMT which data source you wish to use. Once the name of the data source is set, the data attributes are listed:

path: path to where the data is stored.
crs: coordinate system of the data.
data_type: HydroMT DataCatalog type either RasterDataset (gridded data), GeoDataFrame (vector data), DataFrame (tables), Dataset (non-geospatial N-dim data) or GeoDataset (point timeseries).
driver: driver used to open the data. For example raster (GDAL compliant raster file), netcdf (NetCDF file), zarr (zarr file) or vector (GDAL compliant vector file). See the hydromt docs for all options.
driver_kwargs: optional arguments to read the data. Depends on the driver.
meta: optional additional information on the data.
rename: list used to rename the variables inside of the data to HydroMT compliant names. The format is “name_in_dataset: name_in_HydroMT”. Note that the names present in the NetCDF file are the ones set up in the wflow TOML file. The list of standard HydroMT names for demission forcing variables are: time, precip, infilt, runPav, runUnp, exfilt*, q_ss and q_land.
attrs unit: unit attribute of the variables in the data. Used by Delwaq only in order to convert from mm to m3/s and from m to m3 (requires complex unit conversion than just a multiplication or addition with a constant value).

Model setup configuration#

As with building, you can prepare a HydroMT configuration file that includes every components and settings that you want to run during your update.

The configuratio file (yml) contains the model setup configuration and determines which components are updated and in which sequence and sets optional arguments for each component. This configuration is passed to hydromt using -i <path_to_configuration_file>.

Each header as shown between [...] (e.g. [setup_hydrology_forcing]) corresponds with a model component which are explained in the docs(model_components).

Let’s open the example configuration file delwaq_update_EM_forcing.yml from the model repository [examples folder] and have a look at the settings.

[7]:

fn_ini = 'delwaq_update_EM_forcing.yml'
with open(fn_ini, 'r') as f:
    txt = f.read()
print(txt)

setup_hydrology_forcing:
  hydro_forcing_fn: wflow_output_em
  starttime: "2010-02-04 00:00:00"
  endtime: "2010-02-10 00:00:00"
  timestepsecs: 86400
  include_transport: True

write_config:

write_forcing:

write_waqgeom:

Here we can see that we will run setup_hydrology_forcing component to prepare daily hydrological forcing for 8 days in February 2010 using our wflow_output.

Note that the add_volume_offset function is set to True. This is because Delwaq needs water volumes at the beginning of the timestep. In some models, like wflow, volumes are written at the end of the timestep and therefore an offset of one timestep needs to be added for consistency.

You can find more information on the different components and their options in the docs (model_components).

HydroMT CLI update interface#

Using the hydromt update API, we can update one or several components of an already existing Delwaq model. Let’s get an overview of the available options:

[8]:

# Print the options available from the update command
! hydromt update --help

Usage: hydromt update [OPTIONS] MODEL MODEL_ROOT

  Update a specific component of a model.

  Set an output directory to copy the edited model to a new folder, otherwise
  maps are overwritten.

  Example usage: --------------

  Update (overwrite!) landuse-landcover based maps in a Wflow model: hydromt
  update wflow /path/to/model_root -c setup_lulcmaps --opt lulc_fn=vito -d
  /path/to/data_catalog.yml -v

  Update Wflow model components outlined in an .yml configuration file and
  write the model to a directory: hydromt update wflow /path/to/model_root  -o
  /path/to/model_out  -i /path/to/wflow_config.yml  -d
  /path/to/data_catalog.yml -v

Options:
  -o, --model-out DIRECTORY  Output model folder. Maps in MODEL_ROOT are
                             overwritten if left empty.
  -i, --config PATH          Path to hydroMT configuration file, for the model
                             specific implementation.
  -c, --components TEXT      Model methods from configuration file to run
  --opt TEXT                 Method specific keyword arguments, see the method
                             documentation of the specific model for more
                             information about the arguments.
  -d, --data TEXT            Path to local yaml data catalog file OR name of
                             predefined data catalog.
  --dd, --deltares-data      Flag: Shortcut to add the "deltares_data" catalog
  --fo, --force-overwrite    Flag: If provided overwrite existing model files
  --cache                    Flag: If provided cache tiled rasterdatasets
  -q, --quiet                Decrease verbosity.
  -v, --verbose              Increase verbosity.
  --help                     Show this message and exit.

Update demission forcing layers#

[9]:

# NOTE: copy this line (without !) to your shell for more direct feedback
! hydromt update demission ./EM_test_full -i delwaq_update_EM_forcing.yml -d local_sources.yml --fo -vv

2024-06-21 02:54:41,916 - update - log - DEBUG - Appending log messages to file /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/_examples/EM_test_full/hydromt.log.
2024-06-21 02:54:41,916 - update - log - INFO - HydroMT version: 0.10.0
2024-06-21 02:54:41,916 - update - main - INFO - Updating demission model at /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/_examples/EM_test_full (r+).
2024-06-21 02:54:41,916 - update - main - INFO - Output dir: /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/_examples/EM_test_full
2024-06-21 02:54:41,917 - update - main - INFO - User settings:
2024-06-21 02:54:41,933 - update - data_catalog - INFO - Parsing data catalog from local_sources.yml
2024-06-21 02:54:41,938 - update - model_api - WARNING - Model dir already exists and files might be overwritten: /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/_examples/EM_test_full/hydromodel.
2024-06-21 02:54:41,938 - update - model_api - WARNING - Model dir already exists and files might be overwritten: /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/_examples/EM_test_full/staticdata.
2024-06-21 02:54:41,938 - update - model_api - WARNING - Model dir already exists and files might be overwritten: /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/_examples/EM_test_full/geoms.
2024-06-21 02:54:41,938 - update - model_api - WARNING - Model dir already exists and files might be overwritten: /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/_examples/EM_test_full/config.
2024-06-21 02:54:41,939 - update - model_api - INFO - Initializing demission model from hydromt_delwaq (v0.2.2.dev0).
2024-06-21 02:54:41,939 - update - model_api - DEBUG - Reading model file monareas.
2024-06-21 02:54:41,967 - update - model_api - DEBUG - Reading model file staticdata.
2024-06-21 02:54:42,125 - update - model_api - DEBUG - Reading model file basins.
2024-06-21 02:54:42,136 - update - model_api - INFO - setup_hydrology_forcing.hydro_forcing_fn: wflow_output_em
2024-06-21 02:54:42,136 - update - model_api - INFO - setup_hydrology_forcing.starttime: 2010-02-04 00:00:00
2024-06-21 02:54:42,136 - update - model_api - INFO - setup_hydrology_forcing.endtime: 2010-02-10 00:00:00
2024-06-21 02:54:42,136 - update - model_api - INFO - setup_hydrology_forcing.timestepsecs: 86400
2024-06-21 02:54:42,136 - update - model_api - INFO - setup_hydrology_forcing.include_transport: True
2024-06-21 02:54:42,136 - update - demission - INFO - Setting dynamic data from hydrology source wflow_output_em.
2024-06-21 02:54:42,139 - update - rasterdataset - INFO - Reading wflow_output_em netcdf data from /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/_examples/wflow_piave/run_default/output.nc
2024-06-21 02:54:42,168 - update - rasterdataset - DEBUG - Slicing time dim ('2010-02-04 00:00:00', '2010-02-10 00:00:00')
2024-06-21 02:54:42,178 - update - rasterdataset - DEBUG - Clip to [11.775, 45.808, 12.742, 46.692] (epsg:4326))
2024-06-21 02:54:42,180 - update - rasterdataset - DEBUG - Convert units for 2 variables.
No numerical nodata value found, skipping set_nodata
2024-06-21 02:54:42,362 - update - model_api - ERROR - Model config file not found at /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/_examples/EM_test_full/demission.inp
2024-06-21 02:54:42,363 - update - delwaq - INFO - Writing model config to file.
2024-06-21 02:54:42,364 - update - model_api - INFO - write_forcing.write_nc: False
2024-06-21 02:54:42,386 - update - demission - INFO - Writing dynamicmap files.
Writing dynamic data: 100%|███████████████████████| 7/7 [00:00<00:00, 33.08it/s]
2024-06-21 02:54:42,600 - update - delwaq - INFO - Writting waqgeom.nc file

The example above means the following: run hydromt with:

update demission: i.e. update a delwaq model
./EM_test_full: model folder
-i delwaq_update_EM_forcing.yml: setup configuration file containing the components to update and their different options
-d local_sources.yml: local data library, here containing the hydrological outputs from wflow.
--fo: allow to update and overwrite an existing model.
-v: give some extra verbosity (2 * v) to display feedback on screen. Now debug messages are provided.

Visualization of the outputs#

From the information above, you can see that the different forcing variables where updated. If you have a look at the output files, you can see that the hydrological data were added to the dynamicdata folder but also several information were added in the config folder.

[10]:

import os
root = 'EM_test_full'
for path, _, files in os.walk(root):
    print(path)
    for name in files:
        if name.endswith('.xml'):
            continue
        print(f' - {name}')

EM_test_full
 - hydromt.log
EM_test_full/dynamicdata
 - hydrology.bin
EM_test_full/staticdata
 - ptiddown.dat
 - gdp_world.dat
 - river.dat
 - porosity.dat
 - ghs_pop_2015.dat
 - SoilThickness.dat
 - streamorder.dat
 - slope.dat
 - staticdata.nc
EM_test_full/hydromodel
 - basins.tif
 - river.tif
 - ptid.tif
 - modelmap.tif
 - resarea.tif
 - rivwth.tif
 - rivlen.tif
 - elevtn.tif
 - ldd.tif
 - ptiddown.tif
 - lakearea.tif
 - rivarea.tif
EM_test_full/geoms
 - monareas.geojson
 - basins.geojson
EM_test_full/config
 - B7_geometry.bin
 - B2_timers_only.inc
 - B3_nrofseg.inc
 - B7_geometry.inc
 - B7_surf.inc
 - B2_timers.inc
 - B3_waqgeom.nc
 - B2_monareas.inc
 - B5_boundlist.inc
 - B2_outputtimes.inc
 - B1_timestamp.inc
 - B7_hydrology.inc
 - B4_nrofexch.inc
 - B2_sysclock.inc
 - B7_geometry-parameters.inc
 - B2_stations.inc
 - B2_nrofmon.inc
 - B3_attributes.inc
EM_test_full/fews

You can have a look at some of these files:

[11]:

import os
model_path = './EM_test_full'
fn_config = 'config/B1_timestamp.inc'
with open(os.path.join(model_path,fn_config), 'r') as f:
    txt = f.read()
print(txt)

'T0: 2010.02.04 00:00:00  (scu=       1s)'

For the hydrological data directly, the main files are:

dynamicdata/hydrology.bin (Binary data)
config/B7_hydrology.inc (Headers for the variables inside of hydrology.bin)

You can also see that a NetCDF file was created: B3_waqgeom.nc. This file can be used to produce NetCDF outputs directly when running D-Emissions or D-Water Quality but also to visualize the model in Delft-FEWS.

[12]:

import os
model_path = './EM_test_full'
fn_config = 'config/B7_hydrology.inc'
with open(os.path.join(model_path,fn_config), 'r') as f:
    txt = f.read()
print(txt)

SEG_FUNCTIONS
Rainfall RunoffPav RunoffUnp Infiltr Exfiltr vwcproot Overland Subsurface