Build a DELWAQ D-Water Quality model from scratch

This notebook demonstrates how to prepare DELWAQ D-Water Quality model from scratch using the command line interace (CLI).

All lines in this notebook which start with ! are executed from the command line. Within the notebook environment the logging messages are shown after completion. You can also copy these lines and paste these in your shell to get more direct feedback.

hydroMT CLI build interface

In hydroMT, you can interact with DELWAQ models either to be used by D-Emissions (EM) or D-Water Quality (WQ).

Lets first check if the delwaq model is recognized by hydromt

[1]:

# this should return "delwaq, wflow, wflow_sediment"
!hydromt --models

hydroMT model plugins: wflow, wflow_sediment, delwaq

Using the hydromt build API we can setup a complete model from scratch. Let’s get an overview of all the available options:

[2]:

!hydromt build --help

Usage: hydromt build [OPTIONS] MODEL MODEL_ROOT REGION

  Build models from source data.

  Example usage:
  --------------

  To build a wflow model for a subbasin using and point coordinates snapped to cells with stream order >= 4
  hydromt build wflow /path/to/model_root "{'subbasin': [-7.24, 62.09], 'strord': 4}" -i /path/to/wflow_config.ini

  To build a wflow model based on basin ID
  hydromt build wflow /path/to/model_root "{'basin': 230001006}"

  To build a sfincs model based on a bbox (for Texel)
  hydromt build sfincs /path/to/model_root "{'bbox': [4.6891,52.9750,4.9576,53.1994]}"

Options:
  -r, --res FLOAT             Model resolution in model src.
  --build-base / --build-all  Deprecated!
  --opt TEXT                  Component specific keyword arguments, see the
                              setup_<component> method of the specific model
                              for more information about the arguments.

  -i, --config PATH           Path to hydroMT configuration file, for the
                              model specific implementation.

  -d, --data PATH             File path to yml data sources file. See
                              documentation for required yml file format.

  --dd, --deltares-data       Parse default deltares data yml from
                              https://github.com/DirkEilander/hydromt-
                              artifacts/releases

  -q, --quiet                 Decrease verbosity.
  -v, --verbose               Increase verbosity.
  --help                      Show this message and exit.

Setup delwaq EM model base layers

DELWAQ models are quite specific in the sense that they are usually added on top of an already existing hydrologic / hydraulic or hydrodynamic model. In hydroMT, for now, DELWAQ models (EM for D-Emissions and WQ for D-Water Quality) can only be built on top of Wflow hydrologic models. You can find more information on this coupling in docs(coupling_wflow).

Here is how you can build a DELWAQ model:

[3]:

# NOTE: copy this line (without !) to your shell for more direct feedback
!hydromt build delwaq "./WQ_test_base" "{'wflow': 'wflow_piave'}" --opt global.mtype=WQ -vvv

2021-05-06 13:38:36,769 - build - log - DEBUG - Writing log messages to new file /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/examples/examples/WQ_test_base/hydromt.log.
2021-05-06 13:38:36,769 - build - log - INFO - HydroMT version: 0.4.0
2021-05-06 13:38:36,769 - build - main - INFO - Building instance of delwaq model at /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/examples/examples/WQ_test_base.
2021-05-06 13:38:36,769 - build - main - INFO - User settings:
2021-05-06 13:38:36,769 - build - cli_utils - INFO - global.mtype: WQ
2021-05-06 13:38:36,770 - build - model_api - DEBUG - setup_basemaps.region: {'wflow': 'wflow_piave'}
2021-05-06 13:38:36,770 - build - model_api - DEBUG - setup_basemaps.include_soil: False
2021-05-06 13:38:36,770 - build - basin_mask - DEBUG - Parsed region (kind=model): {'mod': <hydromt_wflow.wflow.WflowModel object at 0x7f8bb91e1850>}
2021-05-06 13:38:36,770 - build - delwaq - INFO - Preparing WQ basemaps from hydromodel.
2021-05-06 13:38:36,773 - build - model_api - DEBUG - Model config read from /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/examples/examples/wflow_piave/wflow_sbm.toml
2021-05-06 13:38:36,774 - build - wflow - INFO - Read staticmaps from /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/examples/examples/wflow_piave/staticmaps.nc
2021-05-06 13:38:37,696 - build - delwaq - INFO - Write model data to /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/examples/examples/WQ_test_base
2021-05-06 13:38:37,704 - build - delwaq - INFO - Writing staticmap files.
2021-05-06 13:38:37,714 - build - delwaq - INFO - Writing model config to file.
2021-05-06 13:38:37,714 - build - delwaq - INFO - Writing hydromap files.
2021-05-06 13:38:37,755 - build - delwaq - INFO - Writting pointer file in root/config
2021-05-06 13:38:37,772 - build - delwaq - WARNING - Warning: no dynamic map, skipping write_dynamicmaps.

The example above means the following: run hydromt build with:

  • delwaq : i.e. build a delwaq model

  • ./WQ_test_base : output model folder

  • "{'wflow': 'wflow_piave'}" : derive the model from the existing wflow model located in wflow_piave folder. All REGION options are described in the docs. For delwaq, the only possible options is from an existing wflow model.

  • -opt global.mtype=WQ : specifies which type of delwaq model to build. Here WQ for D-Water Quality.

  • -vv : give some extra verbosity (2 * v) to display feedback on screen. Now debug messages are provided.

NOTE: As we did not specify a model configuration, only the base maps (grid, segment IDs) have been setup using default parameters. To build a complete model we need the use a configuraton ini-file.

Model setup configuration

The ini-file contains the model setup configuration and determines which components are build and in which order and optionally sets non-default arguments for each component. This configuration is passed to hydromt using -i <path_to_ini_file>. We have prepared several example ini-files which are available in the model repository examples folder and from the docs(build_configuration).

Each header as shown between [...] (e.g. [setup_basemaps]) corresponds to a model component. All model components are explained in the docs(model_components).

Almost each DELWAQ model is unique depending on which substances and sources are included but also what kind of emission data is available. For these reasons, there is no default build ini file for a delwaq model and still some manual steps required to build and run a D-Water Quality model with hydroMT. You can learn more about these steps in the docs(hydromt_D-Water Quality).

We will load an example delwaq build WQ ini file for inspection:

[4]:

fn_ini = 'delwaq_build_WQ.ini'
with open(fn_ini, 'r') as f:
    txt = f.read()
print(txt)

[global]
mtype             = WQ                   # type of Delwaq model ['EM', 'WQ']
data_libs         = ['local_sources.yml']                   # add optional paths to data yml files

[setup_basemaps]
include_soil      = False                # add a soil compartment in addition to the surface waters

[setup_monitoring]
mon_points        = wflow_piave/staticgeoms/gauges.geojson  # monitoring points ['None', 'segments', 'path to station location', 'source in DataCatalog']
mon_areas         = compartments         # monitoring areas ['None', 'compartments', 'subcatch']

[setup_hydrology_forcing]
hydro_forcing_fn  = wflow_output                 # source name of the hydrological forcing in the yaml file
starttime         = 2010-02-03 00:00:00  # start time of the Delwaq run
endtime           = 2010-02-10 00:00:00  # end time of the Delwaq run
timestepsecs      = 86400                # model timestep in seconds
add_volume_offset = true                 # add a one-timestep offset to the volume data in the hydrological forcing file compared to the flows

Some explanations for the sections in the ini file above:

  • global: options true for every model components of delwaq. This is where you can specify if you want to build an EM or WQ model using the mtype option. You can also add local data libaries in data_libs instead f in the command line.

  • setup_basemaps: prepares the first delwaq model layers (grid, segment ID) and for WQ also the pointer.

  • setup_monitoring: prepares monitoring points using the gauges locations of the wflow model and the monitoring areas by compartments (surface waters only here)

  • setup_hydrology_forcing: prepares hydrological data from wflow outputs. Information on the wflow_output source is available in the local_sources.yml.

For more information on all the options used (resampling method, NaN values handling…), please check the docs(model_components)

Data for model setup

The <...>_fn arguments correspond to a data source from the DataCatalog which is based on a yml-file with references to the data paths/urls and how the data should be read. This file can be provided to hydromt using -d <path_to_yml_file>.

By default some example data for the Piave basin will be downloaded to ~/.hydromt_data/ which is also used for this example. An overview of the available example data is provided here This example data is a based on the data which are available from the Deltares p-drive. If you have acces to this drive, a pre-configured catalog file can be loaded using the --dd flag.

More background how to write a data catalog yml file can be found in the hydromt core docs

Setup complete delwaq WQ model

[5]:

# NOTE: copy this line (without !) to your shell for more direct feedback
!hydromt build delwaq "./WQ_test_full" "{'wflow': 'wflow_piave'}" -i delwaq_build_WQ.ini -vv

2021-05-06 13:38:39,978 - build - log - DEBUG - Writing log messages to new file /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/examples/examples/WQ_test_full/hydromt.log.
2021-05-06 13:38:39,978 - build - log - INFO - HydroMT version: 0.4.0
2021-05-06 13:38:39,978 - build - main - INFO - Building instance of delwaq model at /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/examples/examples/WQ_test_full.
2021-05-06 13:38:39,979 - build - main - INFO - User settings:
2021-05-06 13:38:39,979 - build - cli_utils - INFO - global.mtype: WQ
2021-05-06 13:38:39,980 - build - cli_utils - INFO - global.data_libs: [PosixPath('/home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/examples/examples/local_sources.yml')]
2021-05-06 13:38:39,980 - build - cli_utils - INFO - setup_basemaps.include_soil: False
2021-05-06 13:38:39,980 - build - cli_utils - INFO - setup_monitoring.mon_points: /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/examples/examples/wflow_piave/staticgeoms/gauges.geojson
2021-05-06 13:38:39,980 - build - cli_utils - INFO - setup_monitoring.mon_areas: compartments
2021-05-06 13:38:39,980 - build - cli_utils - INFO - setup_hydrology_forcing.hydro_forcing_fn: wflow_output
2021-05-06 13:38:39,980 - build - cli_utils - INFO - setup_hydrology_forcing.starttime: 2010-02-03 00:00:00
2021-05-06 13:38:39,980 - build - cli_utils - INFO - setup_hydrology_forcing.endtime: 2010-02-10 00:00:00
2021-05-06 13:38:39,980 - build - cli_utils - INFO - setup_hydrology_forcing.timestepsecs: 86400
2021-05-06 13:38:39,980 - build - cli_utils - INFO - setup_hydrology_forcing.add_volume_offset: true
2021-05-06 13:38:39,985 - build - model_api - DEBUG - setup_basemaps.region: {'wflow': 'wflow_piave'}
2021-05-06 13:38:39,985 - build - model_api - DEBUG - setup_basemaps.include_soil: False
2021-05-06 13:38:39,985 - build - basin_mask - DEBUG - Parsed region (kind=model): {'mod': <hydromt_wflow.wflow.WflowModel object at 0x7ff168155e80>}
2021-05-06 13:38:39,985 - build - delwaq - INFO - Preparing WQ basemaps from hydromodel.
2021-05-06 13:38:39,988 - build - model_api - DEBUG - Model config read from /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/examples/examples/wflow_piave/wflow_sbm.toml
2021-05-06 13:38:39,988 - build - wflow - INFO - Read staticmaps from /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/examples/examples/wflow_piave/staticmaps.nc
2021-05-06 13:38:40,930 - build - model_api - DEBUG - setup_monitoring.mon_points: /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/examples/examples/wflow_piave/staticgeoms/gauges.geojson
2021-05-06 13:38:40,930 - build - model_api - DEBUG - setup_monitoring.mon_areas: compartments
2021-05-06 13:38:40,930 - build - delwaq - INFO - Setting monitoring points and areas
2021-05-06 13:38:40,959 - build - data_adapter - INFO - DataCatalog: Getting gauges GeoDataFrame vector data from /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/examples/examples/wflow_piave/staticgeoms/gauges.geojson
2021-05-06 13:38:41,012 - build - delwaq - INFO - Gauges locations read from /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/examples/examples/wflow_piave/staticgeoms/gauges.geojson
2021-05-06 13:38:41,035 - build - model_api - DEBUG - setup_hydrology_forcing.hydro_forcing_fn: wflow_output
2021-05-06 13:38:41,035 - build - model_api - DEBUG - setup_hydrology_forcing.starttime: 2010-02-03 00:00:00
2021-05-06 13:38:41,035 - build - model_api - DEBUG - setup_hydrology_forcing.endtime: 2010-02-10 00:00:00
2021-05-06 13:38:41,035 - build - model_api - DEBUG - setup_hydrology_forcing.timestepsecs: 86400
2021-05-06 13:38:41,036 - build - model_api - DEBUG - setup_hydrology_forcing.add_volume_offset: true
2021-05-06 13:38:41,036 - build - delwaq - INFO - Setting dynamic data from hydrology source wflow_output.
2021-05-06 13:38:41,046 - build - data_adapter - INFO - DataCatalog: Getting wflow_output RasterDataset netcdf data from /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/examples/examples/wflow_piave/run_default/output.nc
2021-05-06 13:38:41,078 - build - data_adapter - DEBUG - RasterDataset: Clip with geom - [11.775, 45.808, 12.742, 46.692]
2021-05-06 13:38:41,243 - build - delwaq - INFO - Write model data to /home/runner/work/hydromt_delwaq/hydromt_delwaq/docs/examples/examples/WQ_test_full
2021-05-06 13:38:41,256 - build - delwaq - INFO - Writing staticmap files.
2021-05-06 13:38:41,269 - build - delwaq - INFO - Writing model staticgeom to file.
2021-05-06 13:38:41,303 - build - delwaq - INFO - Writing model config to file.
2021-05-06 13:38:41,303 - build - delwaq - INFO - Writing hydromap files.
2021-05-06 13:38:41,343 - build - delwaq - INFO - Writting pointer file in root/config
2021-05-06 13:38:41,377 - build - delwaq - INFO - Writing dynamicmap files.
2021-05-06 13:38:41,377 - build - delwaq - INFO - Writting dynamic data for timestep 1/8
2021-05-06 13:38:41,400 - build - delwaq - INFO - Writting dynamic data for timestep 2/8
2021-05-06 13:38:41,419 - build - delwaq - INFO - Writting dynamic data for timestep 3/8
2021-05-06 13:38:41,439 - build - delwaq - INFO - Writting dynamic data for timestep 4/8
2021-05-06 13:38:41,460 - build - delwaq - INFO - Writting dynamic data for timestep 5/8
2021-05-06 13:38:41,479 - build - delwaq - INFO - Writting dynamic data for timestep 6/8
2021-05-06 13:38:41,500 - build - delwaq - INFO - Writting dynamic data for timestep 7/8
2021-05-06 13:38:41,520 - build - delwaq - INFO - Writting dynamic data for timestep 8/8
2021-05-06 13:38:41,541 - build - delwaq - INFO - Writting waqgeom.nc file

With this example we build a complete delwaq WQ model including hydrological data. Compared to the previous hydromt build we have added: * -i delwaq_build_WQ.ini : setup configuration file including all components to build and their arguments

Note that the --opt global.mtype=WQ is not present anymore, as this option is directly set in the delwaq_build_WQ.ini file.

Next we check which files and folders have been created:

  • config: basic configuration setups for the Delwaq input file .inp. Names of the ASCII files start with the corresponding block in the .inp file.

  • dynamicdata: Delwaq time-dependant data (hydrological fluxes) in binary format.

  • fews: additional files for connection with Delft-FEWS (not implemented yet)

  • hydromodel: data extracted to connect the hydrological model (wflow) and Delwaq (tif format). These files are not mandatory for Delwaq but produced extra from HydroMT.

  • staticdata: static (spatial but non time-dependant) data for Delwaq (typically emission data) in binary format. A copy of the data in NetCDF format is also available for easier visualization.

  • staticgeoms: related geometry files (geojson format). These files are not mandatory for Delwaq but produced extra from HydroMT.

[6]:

import os
root = 'WQ_test_full'
for path, _, files in os.walk(root):
    print(path)
    for name in files:
        if name.endswith('.xml'):
            continue
        print(f' - {name}')

WQ_test_full
 - hydromt.log
WQ_test_full/fews
WQ_test_full/dynamicdata
 - flow.dat
 - volume.dat
WQ_test_full/staticgeoms
 - monpoints.geojson
 - monareas.geojson
 - basins.geojson
WQ_test_full/staticdata
 - staticmaps.nc
 - surface.dat
 - slope.dat
 - manning.dat
 - streamorder.dat
WQ_test_full/hydromodel
 - ptiddown.tif
 - rivwth.tif
 - ldd.tif
 - ptid.tif
 - basins.tif
 - modelmap.tif
 - rivlen.tif
WQ_test_full/config
 - B4_nrofexch.inc
 - B2_nrofmon.inc
 - B2_stations.inc
 - B5_boundlist.inc
 - B1_timestamp.inc
 - B2_stations-balance.inc
 - B2_monareas.inc
 - B2_timers.inc
 - B2_outputtimes.inc
 - B3_attributes.inc
 - B4_pointer.poi
 - B3_nrofseg.inc
 - B3_waqgeom.nc
 - B2_sysclock.inc
 - B2_timers_only.inc
 - B4_pointer.inc

You can see that we didn’t create any emission data here. In some cases, you may require some additional GIS data in order to run a WQ model, for example, when you do not need to run the D-Emissions model beforehand, but include point sources that emit pollutants into the surface water directly. When this is the case, you can still use the various [setup_emission_*] components of hydroMT_delwaq.

Visualize and/or inspect model schematization

  • The delwaq plot example notebook contains scripts to visualize your model (staticdata, dynamicdata, hydromodel and staticgeoms).

In the meantime, feel free to have a look at some configuration files that were prepared by HydroMT. You can change the file name from the example code below.

[7]:

import os
model_path = './WQ_test_full'
fn_config = 'config/B3_nrofseg.inc'
with open(os.path.join(model_path,fn_config), 'r') as f:
    txt = f.read()
print(txt)

1653 ; nr of segments

In the config folder, you can also see two pointer files:

  • B4_pointer.poi (Binary version)

  • B4_pointer.inc (ASCII version)

You can also see that a NetCDF file was created: B3_waqgeom.nc. This file can be used to produce NetCDF outputs directly when running D-Emissions or D-Water Quality but also to visualize the model in FEWS.