Tip
Example: Working with models from CLI#
HydroMT has the following high-level functionality from the Command Line Interface (CLI) for setting up models from raw data or adjusting models:
building a model: building a model from scratch.
updating a model: adding or changing model components of an existing model.
clipping a model: changing the spatial domain of an existing model (e.g. select subbasins from a larger model).
Here we show how to build and update a hypothetical distributed model from the command line interface (CLI) based on the generic HydroMT grid_model.
Lets first check which models are available in our environment:
[1]:
!hydromt --models
model plugins:
generic models (hydromt 0.7.0):
- grid_model
- lumped_model
- mesh_model
- network_model
Build a model from CLI#
To build a model you always follow the next four steps.
Prepare or use a pre-defined data catalog with all the required data sources
Define your model region, see the overview of model region options.
Prepare a model configuration which describes the complete pipeline to build your model, see preparing a model configuration.
Build you model using the CLI or Python interface
Here we focus steps 2-4 and use data from the predefined aritifact_data data catalog
Using the hydromt build method we can setup a complete model from scratch. Let’s get an overview of the method and its arguments.
Note the required MODEL (i.e. name of the model), MODEL_ROOT (i.e. folder where to save the model) arguments. As of version v0.7.0 the REGION (i.e. area of interest) argument is optional. This argument can be used by adding -r or –region flag.
[2]:
!hydromt build --help
Usage: hydromt build [OPTIONS] MODEL MODEL_ROOT
Build models from scratch.
Example usage:
--------------
To build a wflow model for a subbasin using and point coordinates snapped to cells with stream order >= 4
hydromt build wflow /path/to/model_root -i /path/to/wflow_config.ini -r "{'subbasin': [-7.24, 62.09], 'strord': 4}" -d deltares_data -d /path/to/data_catalog.yml -v
To build a sfincs model based on a bbox
hydromt build sfincs /path/to/model_root -i /path/to/sfincs_config.ini -r "{'bbox': [4.6891,52.9750,4.9576,53.1994]}" -d /path/to/data_catalog.yml -v
Options:
--opt TEXT Method specific keyword arguments, see the method
documentation of the specific model for more
information about the arguments.
-i, --config PATH Path to hydroMT configuration file, for the model
specific implementation.
-r, --region TEXT Set the region for which to build the model, e.g.
{'subbasin': [-7.24, 62.09]}
-d, --data TEXT Path to local yaml data catalog file OR name of
predefined data catalog.
--dd, --deltares-data Flag: Shortcut to add the "deltares_data" catalog
--fo, --force-overwrite Flag: If provided overwrite existing model files
--cache Flag: If provided cache tiled rasterdatasets
-v, --verbose Increase verbosity.
-q, --quiet Decrease verbosity.
--help Show this message and exit.
[3]:
!hydromt build grid_model ./tmp_grid_model --region "{'bbox': [11.70, 45.35, 12.95, 46.70]}" -vv
2023-02-22 04:42:35,300 - build - log - DEBUG - Writing log messages to new file /home/runner/work/hydromt/hydromt/docs/_examples/tmp_grid_model/hydromt.log.
2023-02-22 04:42:35,300 - build - log - INFO - HydroMT version: 0.7.0
2023-02-22 04:42:35,300 - build - main - INFO - Building instance of grid_model model at /home/runner/work/hydromt/hydromt/docs/_examples/tmp_grid_model.
2023-02-22 04:42:35,300 - build - main - INFO - User settings:
2023-02-22 04:42:35,313 - build - model_api - INFO - Initializing grid_model model from hydromt (v0.7.0).
2023-02-22 04:42:35,313 - build - model_api - INFO - setup_region.region: {'bbox': [11.7, 45.35, 12.95, 46.7]}
2023-02-22 04:42:35,313 - build - model_api - INFO - setup_region.hydrography_fn: merit_hydro
2023-02-22 04:42:35,313 - build - model_api - INFO - setup_region.basin_index_fn: merit_hydro_index
2023-02-22 04:42:35,313 - build - basin_mask - DEBUG - Parsed region (kind=bbox): {'bbox': [11.7, 45.35, 12.95, 46.7]}
2023-02-22 04:42:35,317 - build - model_api - INFO - Writing model data to /home/runner/work/hydromt/hydromt/docs/_examples/tmp_grid_model
2023-02-22 04:42:35,317 - build - model_api - INFO - Writing model config to /home/runner/work/hydromt/hydromt/docs/_examples/tmp_grid_model/model.ini
2023-02-22 04:42:35,317 - build - model_grid - DEBUG - No grid data found, skip writing.
2023-02-22 04:42:35,317 - build - model_api - DEBUG - Writing file geoms/region.geojson
2023-02-22 04:42:35,348 - build - model_api - DEBUG - No forcing data found, skip writing.
2023-02-22 04:42:35,348 - build - model_api - DEBUG - No states data found, skip writing.
The example above means the following: run hydromt build with:
grid_model: i.e. build a generic GridModel instance./tmp_grid_model: output model folder--region "{'bbox': [11.70, 45.35, 12.95, 46.70]}": set the region of interest using a bounding box defined by its [xmin, ymin, xmax, ymax] coordinates (in WGS84)-vv: give some extra verbosity (2 * v) to display feedback on screen. Now debug messages are provided.
As we did not specify a hydromt configuration, besides the hydromt.log file, an empty model simulation configuration file, only the model region (area of interest) has been defined and is saved in the geoms folder. To build a complete model we need the use a hydromt configuration ini-file.
[4]:
# print MODEL_ROOT folder
import os
def print_dir(root):
for path, _, files in os.walk(root):
print(path)
for name in files:
if name.endswith(".xml"):
continue
print(f" - {name}")
print_dir("tmp_grid_model")
tmp_grid_model
- hydromt.log
- model.ini
tmp_grid_model/geoms
- region.geojson
The configuration file exists of available setup_methods for your model, listed in the order of execution. Most methods start with reading input data using the DataAdapter, transforming the data using workflows (e.g. reprojection, reclassification, aggregation, etc…) and adding the new model data to the right model component. An overview of the available methods can be found in the online API reference for the GridModel, LumpedModel, and MeshModel
Note that these methods for the generic model classes are still quite limited. To get an idea of potential setup_ methods, checkout the model plugins
The configuration ini file can be passed to the hydromt build method using the -i flag.
[5]:
# check the content of the ini file (for sake of the example only)
fn_ini = "grid_model_build.ini"
with open(fn_ini, "r") as f:
txt = f.read()
print(txt)
[setup_config]
header.settings = value
timers.start = "2010-02-05"
timers.end = "2010-02-15"
[setup_region]
# Note: the "region" argument can be passed directly to the build method
hydrography_fn = merit_hydro
basin_index_fn = merit_hydro_index
[setup_maps_from_raster]
raster_fn = merit_hydro_1k
variables = elevtn
fill_method = None
reproject_method=None
[setup_maps_from_raster_reclass]
raster_fn = vito
reclass_table_fn = vito_reclass # Note: from data/vito_reclass.yml data catalog
reclass_variables= ['manning']
[write]
components = ['config', 'geoms', 'maps']
[6]:
!hydromt build grid_model ./tmp_grid_model1 -r "{'bbox': [11.70, 45.35, 12.95, 46.70]}" -i grid_model_build.ini -d artifact_data -d data/vito_reclass.yml -vv
2023-02-22 04:42:37,347 - build - log - DEBUG - Writing log messages to new file /home/runner/work/hydromt/hydromt/docs/_examples/tmp_grid_model1/hydromt.log.
2023-02-22 04:42:37,347 - build - log - INFO - HydroMT version: 0.7.0
2023-02-22 04:42:37,347 - build - main - INFO - Building instance of grid_model model at /home/runner/work/hydromt/hydromt/docs/_examples/tmp_grid_model1.
2023-02-22 04:42:37,347 - build - main - INFO - User settings:
2023-02-22 04:42:37,414 - build - data_catalog - INFO - Reading data catalog artifact_data v0.0.8 from archive
2023-02-22 04:42:37,415 - build - data_catalog - INFO - Parsing data catalog from /home/runner/.hydromt_data/artifact_data/v0.0.8/data_catalog.yml
2023-02-22 04:42:37,486 - build - data_catalog - INFO - Parsing data catalog from data/vito_reclass.yml
2023-02-22 04:42:37,487 - build - model_api - INFO - Initializing grid_model model from hydromt (v0.7.0).
2023-02-22 04:42:37,487 - build - model_api - DEBUG - Setting model config options.
2023-02-22 04:42:37,487 - build - model_api - INFO - setup_region.region: {'bbox': [11.7, 45.35, 12.95, 46.7]}
2023-02-22 04:42:37,487 - build - model_api - INFO - setup_region.hydrography_fn: merit_hydro
2023-02-22 04:42:37,488 - build - model_api - INFO - setup_region.basin_index_fn: merit_hydro_index
2023-02-22 04:42:37,488 - build - basin_mask - DEBUG - Parsed region (kind=bbox): {'bbox': [11.7, 45.35, 12.95, 46.7]}
2023-02-22 04:42:37,491 - build - model_api - INFO - setup_maps_from_raster.raster_fn: merit_hydro_1k
2023-02-22 04:42:37,491 - build - model_api - INFO - setup_maps_from_raster.variables: elevtn
2023-02-22 04:42:37,491 - build - model_api - INFO - setup_maps_from_raster.fill_method: None
2023-02-22 04:42:37,491 - build - model_api - INFO - setup_maps_from_raster.name: None
2023-02-22 04:42:37,491 - build - model_api - INFO - setup_maps_from_raster.reproject_method: None
2023-02-22 04:42:37,491 - build - model_api - INFO - setup_maps_from_raster.split_dataset: True
2023-02-22 04:42:37,491 - build - model_api - INFO - Preparing maps data from raster source merit_hydro_1k
2023-02-22 04:42:37,492 - build - data_catalog - INFO - DataCatalog: Getting merit_hydro_1k RasterDataset raster data from /home/runner/.hydromt_data/artifact_data/v0.0.8/merit_hydro_1k/{variable}.tif
2023-02-22 04:42:37,522 - build - rasterdataset - DEBUG - RasterDataset: Clip with geom - [11.700, 45.350, 12.950, 46.700]
2023-02-22 04:42:37,534 - build - model_api - INFO - setup_maps_from_raster_reclass.raster_fn: vito
2023-02-22 04:42:37,534 - build - model_api - INFO - setup_maps_from_raster_reclass.reclass_table_fn: vito_reclass
2023-02-22 04:42:37,534 - build - model_api - INFO - setup_maps_from_raster_reclass.reclass_variables: ['manning']
2023-02-22 04:42:37,534 - build - model_api - INFO - setup_maps_from_raster_reclass.variable: None
2023-02-22 04:42:37,534 - build - model_api - INFO - setup_maps_from_raster_reclass.fill_method: None
2023-02-22 04:42:37,534 - build - model_api - INFO - setup_maps_from_raster_reclass.reproject_method: None
2023-02-22 04:42:37,534 - build - model_api - INFO - setup_maps_from_raster_reclass.name: None
2023-02-22 04:42:37,534 - build - model_api - INFO - setup_maps_from_raster_reclass.split_dataset: True
2023-02-22 04:42:37,534 - build - model_api - INFO - Preparing map data by reclassifying the data in vito based on vito_reclass
2023-02-22 04:42:37,534 - build - data_catalog - INFO - DataCatalog: Getting vito RasterDataset raster data from /home/runner/.hydromt_data/artifact_data/v0.0.8/vito.tif
2023-02-22 04:42:37,551 - build - rasterdataset - DEBUG - RasterDataset: Clip with geom - [11.700, 45.350, 12.950, 46.700]
2023-02-22 04:42:37,566 - build - data_catalog - INFO - DataCatalog: Getting vito_reclass DataFrame csv data from /home/runner/work/hydromt/hydromt/docs/_examples/data/vito_reclass.csv
2023-02-22 04:42:37,566 - build - dataframe - INFO - DataFrame: Read csv data.
2023-02-22 04:42:37,577 - build - model_api - INFO - write.components: ['config', 'geoms', 'maps']
2023-02-22 04:42:37,577 - build - model_api - INFO - Writing model data to /home/runner/work/hydromt/hydromt/docs/_examples/tmp_grid_model1
2023-02-22 04:42:37,577 - build - model_api - INFO - Writing model config to /home/runner/work/hydromt/hydromt/docs/_examples/tmp_grid_model1/model.ini
2023-02-22 04:42:37,577 - build - model_api - DEBUG - Writing file geoms/region.geojson
2023-02-22 04:42:37,608 - build - model_api - DEBUG - Writing file maps/elevtn.nc
2023-02-22 04:42:37,808 - build - model_api - DEBUG - Writing file maps/manning.nc
2023-02-22 04:42:38,000 - build - model_api - INFO - Writing model data to /home/runner/work/hydromt/hydromt/docs/_examples/tmp_grid_model1
2023-02-22 04:42:38,000 - build - model_api - INFO - Writing model config to /home/runner/work/hydromt/hydromt/docs/_examples/tmp_grid_model1/model.ini
2023-02-22 04:42:38,000 - build - model_grid - DEBUG - No grid data found, skip writing.
2023-02-22 04:42:38,000 - build - model_api - DEBUG - Writing file geoms/region.geojson
2023-02-22 04:42:38,006 - build - model_api - DEBUG - No forcing data found, skip writing.
2023-02-22 04:42:38,006 - build - model_api - DEBUG - No states data found, skip writing.
The example above means the following: run hydromt build with:
grid_model: i.e. build a generic GridModel instance./tmp_grid_model1: output model folder-r "{'bbox': [11.70, 45.35, 12.95, 46.70]}": set the region of interest using a bounding box defined by its [xmin, ymin, xmax, ymax] coordinates (in WGS84)-i grid_model_build.ini: use this ini file to configure the model build-d artifact_data -d data/vito_reclass.yml: parse the pre-defined artifact_data and the local vito_reclass data catalogs-vv: give some extra verbosity (2 * v) to display feedback on screen. Now debug messages are provided.
[7]:
print_dir("tmp_grid_model1")
tmp_grid_model1
- hydromt.log
- model.ini
tmp_grid_model1/maps
- manning.nc
- elevtn.nc
tmp_grid_model1/geoms
- region.geojson
[8]:
# checkout the content of the hypothetical model simulation configuration
fn_ini = "tmp_grid_model1/model.ini"
with open(fn_ini, "r") as f:
txt = f.read()
print(txt)
[header]
settings = value
[timers]
start = 2010-02-05
end = 2010-02-15
Update a model from CLI#
Using the hydromt update method we can update an existing model with new components or modify existing components. Let’s get an overview of the method and its arguments.
Note that the MODEL (i.e. name of the model), and MODEL_ROOT (i.e. folder of existing model) are still required. There is an optional -o --model-out option to save the updated model in a different directory.
[9]:
!hydromt update --help
Usage: hydromt update [OPTIONS] MODEL MODEL_ROOT
Update a specific component of a model. Set an output directory to copy the
edited model to a new folder, otherwise maps are overwritten.
Example usage:
--------------
Update (overwrite!) landuse-landcover based maps in a Wflow model
hydromt update wflow /path/to/model_root -c setup_lulcmaps --opt lulc_fn=vito -d /path/to/data_catalog.yml -v
Update Wflow model components outlined in an .ini configuration file and write the model to a directory
hydromt update wflow /path/to/model_root -o /path/to/model_out -i /path/to/wflow_config.ini -d /path/to/data_catalog.yml -v
Options:
-o, --model-out DIRECTORY Output model folder. Maps in MODEL_ROOT are
overwritten if left empty.
-i, --config PATH Path to hydroMT configuration file, for the model
specific implementation.
-c, --components TEXT Model methods from ini file to run
--opt TEXT Method specific keyword arguments, see the method
documentation of the specific model for more
information about the arguments.
-d, --data TEXT Path to local yaml data catalog file OR name of
predefined data catalog.
--dd, --deltares-data Flag: Shortcut to add the "deltares_data" catalog
--cache Flag: If provided cache tiled rasterdatasets
-q, --quiet Decrease verbosity.
-v, --verbose Increase verbosity.
--help Show this message and exit.
In this basic example we use the hydromt update method to update the GridModel instance with an upstream area raster map. Then we write only the updated model map component to file.
The %%writefile magic saves the content below to a file. The content of the ini file start from the second line.
[10]:
%%writefile ./tmp_grid_model1/grid_model_update.ini
[setup_maps_from_raster]
raster_fn = merit_hydro_1k
variables = uparea
[write_maps]
Writing ./tmp_grid_model1/grid_model_update.ini
[11]:
!hydromt update grid_model ./tmp_grid_model1 -i ./tmp_grid_model1/grid_model_update.ini -vv
2023-02-22 04:42:42,073 - update - log - DEBUG - Appending log messages to file /home/runner/work/hydromt/hydromt/docs/_examples/tmp_grid_model1/hydromt.log.
2023-02-22 04:42:42,073 - update - log - INFO - HydroMT version: 0.7.0
2023-02-22 04:42:42,073 - update - main - INFO - Updating grid_model model at /home/runner/work/hydromt/hydromt/docs/_examples/tmp_grid_model1 (r+).
2023-02-22 04:42:42,073 - update - main - INFO - Output dir: /home/runner/work/hydromt/hydromt/docs/_examples/tmp_grid_model1
2023-02-22 04:42:42,073 - update - main - INFO - User settings:
2023-02-22 04:42:42,086 - update - model_api - WARNING - Model dir already exists and files might be overwritten: /home/runner/work/hydromt/hydromt/docs/_examples/tmp_grid_model1/.
2023-02-22 04:42:42,086 - update - model_api - INFO - Initializing grid_model model from hydromt (v0.7.0).
2023-02-22 04:42:42,088 - update - model_api - DEBUG - Reading model file region.
2023-02-22 04:42:42,110 - update - model_api - INFO - setup_maps_from_raster.raster_fn: merit_hydro_1k
2023-02-22 04:42:42,110 - update - model_api - INFO - setup_maps_from_raster.variables: uparea
2023-02-22 04:42:42,110 - update - model_api - INFO - setup_maps_from_raster.fill_method: None
2023-02-22 04:42:42,110 - update - model_api - INFO - setup_maps_from_raster.name: None
2023-02-22 04:42:42,110 - update - model_api - INFO - setup_maps_from_raster.reproject_method: None
2023-02-22 04:42:42,111 - update - model_api - INFO - setup_maps_from_raster.split_dataset: True
2023-02-22 04:42:42,111 - update - model_api - INFO - Preparing maps data from raster source merit_hydro_1k
2023-02-22 04:42:42,167 - update - data_catalog - INFO - Reading data catalog artifact_data v0.0.8 from archive
2023-02-22 04:42:42,167 - update - data_catalog - INFO - Parsing data catalog from /home/runner/.hydromt_data/artifact_data/v0.0.8/data_catalog.yml
2023-02-22 04:42:42,237 - update - data_catalog - INFO - DataCatalog: Getting merit_hydro_1k RasterDataset raster data from /home/runner/.hydromt_data/artifact_data/v0.0.8/merit_hydro_1k/{variable}.tif
2023-02-22 04:42:42,267 - update - rasterdataset - DEBUG - RasterDataset: Clip with geom - [11.700, 45.350, 12.950, 46.700]
2023-02-22 04:42:42,279 - update - model_api - INFO - write_maps.fn: maps/{name}.nc
2023-02-22 04:42:42,279 - update - model_api - DEBUG - Writing file maps/uparea.nc
The example above means the following: run hydromt update with:
grid_model: i.e. update a generic GridModel instance./tmp_model1: the folder of the to-be updated model-i ./tmp_grid_model1/grid_model_update.ini: the hydromt configuration listing the methods to be executed-vv: give some extra verbosity (2 * v) to display feedback on screen. Now debug messages are provided.
[12]:
# note the difference with the original model
print_dir("tmp_grid_model1")
tmp_grid_model1
- hydromt.log
- grid_model_update.ini
- model.ini
tmp_grid_model1/maps
- manning.nc
- uparea.nc
- elevtn.nc
tmp_grid_model1/geoms
- region.geojson