{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9a73590b",
   "metadata": {},
   "source": [
    "## Model building using HydroMT Delft3D FM"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee510c6e",
   "metadata": {},
   "source": [
    "This notebook demonstrates how to prepare a **Delft3D FM** model from scratch using the **Command Line Interface (CLI)**. We will outline the required steps for model building and show the contents of a **configuration file** and the **data catalog**, which are essential for setting up and managing your model. After following this guide, you should be able to use the example notebooks for building 1D, 2D, and coupled 1D2D models.\n",
    "\n",
    "To run this notebook, you need a Python environment with **HydroMT Core** and the **HydroMT Delft3D FM plugin** installed. Additionally, you will need packages such as **jupyterlab** and **notebook** for notebook functionality, and **cartopy** and **matplotlib** for plotting purposes. The installation guide for HydroMT Delft3D FM can be found in the [User Guide](https://deltares.github.io/hydromt_delft3dfm/latest/getting_started/installation.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b96aa15",
   "metadata": {},
   "source": [
    "> **NOTE**: \n",
    "All lines in this notebook which start with `!` are executed from the command line. Within the notebook environment, the logging messages are shown after completion. You can also copy these lines and paste them in your shell to get more direct feedback."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "451c8168",
   "metadata": {},
   "source": [
    "### Accessing HydroMT through CLI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c70853e",
   "metadata": {},
   "source": [
    "Let's first check if the Delft3D FM model is recognized by HydroMT using the command `hydromt --models`. This should return **dflowfm**, as well as the generic HydroMT models: **grid_model**, **lumped_model**, **mesh_model**, **network_model**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c045f489",
   "metadata": {},
   "source": [
    "> **NOTE**: \n",
    "If you have other HydroMT plugins installed in the same environment (Wflow, SFINCS, FIAT, and more), then these models will also appear after this command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40470302",
   "metadata": {},
   "outputs": [],
   "source": [
    "!hydromt --models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "920c45b2",
   "metadata": {},
   "source": [
    "We can use the command `hydromt --help` to get an overview of all the commands that can be accessed through the command line interface. HydroMT is typically used to `hydromt build` a model using a configuration file. This command sets up the model based on the parameters and data specified in the configuration file.\n",
    "\n",
    "It is also possible to `hydromt update` or `hydromt clip` existing models:\n",
    "- **`hydromt update`**: This command updates an existing model with new data or parameters.\n",
    "- **`hydromt clip`**: This command clips the model to a specified region, which can be useful for focusing on a smaller area within a larger model.\n",
    "\n",
    "The command `hydromt export` is not directly related to model building but is used to export data from a data catalog. This can be useful for extracting specific datasets for analysis or use in other applications.\n",
    "\n",
    "The command `hydromt check` is used to validate the configuration file and the linked data catalogs and region settings. This can be especially useful to catch mistakes in the model building process without having to pass all the model building steps, which can take a lot of time for large models.\n",
    "\n",
    "A more detailed description of these commands can be found in the [HydroMT API](https://deltares.github.io/hydromt/latest/api/cli.html). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed06a240",
   "metadata": {},
   "outputs": [],
   "source": [
    "!hydromt --help"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0ef43af",
   "metadata": {},
   "source": [
    "> **NOTE**: \n",
    "When inspecting the documentation of HydroMT Core, make sure that you open the documentation for the corresponding HydroMT version. The HydroMT version of your Python environment is listed as a part of the `hydromt --models` command, or can be found by simply using `hydromt --version`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "add0c270",
   "metadata": {},
   "outputs": [],
   "source": [
    "!hydromt --version"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3f8ece2",
   "metadata": {},
   "source": [
    "### Requirements for building a model in HydroMT Delft3D FM"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d9b5a20",
   "metadata": {},
   "source": [
    "Using the **HydroMT build** API, we can set up a complete model from scratch. This process involves several steps and options that allow you to customize your model according to your specific needs. Let's get an overview of all the available options to understand how we can effectively use this API for model building."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d39e4a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "!hydromt build --help"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "261a2edc",
   "metadata": {},
   "source": [
    "We can see that there are two required arguments for building a model: `MODEL`, which describes the type of model to build, and `MODEL_ROOT`, which is the root path of the model. Additionally, a HydroMT configuration file can be specified using either `-i` or `--config`, a data catalog using either `-d` or `--data`, and region settings using `-r` or `--region`. Both the configuration file and the data catalog will be explained in greater detail in this notebook.\n",
    "\n",
    "- **`MODEL`**: This argument specifies the type of model you want to build (e.g., `dflowfm`).\n",
    "- **`MODEL_ROOT`**: This argument defines the root directory where the model will be stored.\n",
    "- **`-i` or `--config`**: Use this option to specify the path to the HydroMT configuration file.\n",
    "- **`-d` or `--data`**: Use this option to specify the data catalog that contains the datasets required for the model.\n",
    "\n",
    "These options provide flexibility and control over the model-building process, allowing you to customize your model according to your specific needs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e84ea23c",
   "metadata": {},
   "source": [
    "> **NOTE**: \n",
    "In the current version of HydroMT Delft3D FM, no region settings are required when building a model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebd31242",
   "metadata": {},
   "source": [
    "### Model setup configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb8282c9",
   "metadata": {},
   "source": [
    "The HydroMT configuration file contains the model setup configuration and determines which methods are used to prepare the different components of a Delft3D FM model, in which order, and optionally sets non-default arguments for each method. This configuration is passed to HydroMT using `-i <path_to_configuration_file>`. We have prepared several example YAML files which are available in the model repository [examples folder](https://github.com/Deltares/hydromt_delft3dfm/tree/main/examples) and from the [docs (building a model)](https://deltares.github.io/hydromt_delft3dfm/latest/user_guide/dflowfm_build). These example YAML files are also used in this notebook to build the 1D, 2D, and coupled 1D2D models.\n",
    "\n",
    "Each section, in the ``steps`` list (e.g., `setup_rivers_from_dem`), corresponds to a model method. A model method is a step inside the model building process. Each method adds a new component to the model, thus building the model step-by-step. All model methods are explained in the [docs (model components)](https://deltares.github.io/hydromt_delft3dfm/latest/user_guide/dflowfm_model_setup.html). The `global` section contains direct model initialization properties which are required when the empty model is initialized. It typically contains global settings such as the coordinate system of the model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc3fd44d",
   "metadata": {},
   "source": [
    "We will load the default dflowfm build 1D2D yaml file for inspection:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7393a54d",
   "metadata": {},
   "outputs": [],
   "source": [
    "fn_yml = \"dflowfm_build_piave.yml\"\n",
    "with open(fn_yml, \"r\") as f:\n",
    "    txt = f.read()\n",
    "print(txt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "708139e9",
   "metadata": {},
   "source": [
    "It can be seen that many arguments are required for the methods. Some arguments require data sources, typically indicated using `<...>_fn`. For example, the `setup_pipes` method requires a data source for the geometry of the pipes: `pipes_fn`, and additionally a data source with the default geometry of these pipes: `pipes_default_fn`. These data sources are obtained either directly by providing a path to the corresponding file, or by using an entry from a given **data catalog**. The next section explains how to create and use a **data catalog**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0242689",
   "metadata": {},
   "source": [
    "> **NOTE**: \n",
    "In this example, we apply the region argument in `setup_rivers_from_dem`, `setup_pipes`, and `setup_mesh2d`. They are identical in this example, but you can alter them to have different regions for rivers, pipes, and the 2D mesh. This allows you to customize your model as much as possible, as the extent of the 1D river can differ from the extent of the 1D urban network or the 2D inundation grid. The region bounding boxes (bbox) are in WGS 84 as required, but you should specify the dflowfm model destination Coordinate Reference System (CRS) in the `global` section. This ensures that you can use grid definition data (rivers, pipes) from different data sources or CRS."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7584dcae",
   "metadata": {},
   "source": [
    "### Data for model setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dad35651",
   "metadata": {},
   "source": [
    "Most of the time, the `<...>_fn` arguments correspond to a data source from the `DataCatalog`, which is based on a YAML file with references to the data paths/URLs and how the data should be read. This file can be provided to HydroMT using `-d <path_to_yml_file>`. The functionality of the data catalog is prescribed by HydroMT Core, and therefore, the detailed description on how to create and use a data catalog can be found in the [HydroMT core docs](https://deltares.github.io/hydromt/latest/user_guide/data_prepare_cat.html). It is possible to use multiple data catalogs by repeating the `-d` flag.\n",
    "\n",
    "A data catalog in HydroMT is a structured way to organize and reference various datasets, making it easier to manage and access data for modeling. Its entries describe where data can be found, which adapter to use to read data with which settings, how to modify the data when reading (scaling, offsetting), and metadata.\n",
    "\n",
    "> **NOTE**: In HydroMT-Delft3D FM, an additional data catalog with all default parameter values is always used: [parameters_data.yml](https://github.com/Deltares/hydromt_delft3dfm/tree/main/hydromt_delft3dfm/data/parameters_data.yml). It contains default values for 1D network elements and structures, as well as default 2D land use parameter mapping tables."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68dc497c",
   "metadata": {},
   "source": [
    "We use the pre-defined `artifact_data` catalog in this example notebook, which contains some example data for the Piave basin. This data will be downloaded to `~/.hydromt_data/`. To give an example of a data catalog, we will use `hydromt export` to download and export some entries of the `artifact_data` catalog. This will be done for the same region as the example build models, and the target directory will be the **data** folder, which should already exist in the same folder as this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13770b1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "!hydromt export ./data -s grip_roads -s merit_hydro -s vito_2015 -r \"{'bbox': [12.4331, 46.4661, 12.5212, 46.5369]}\" -d artifact_data -v "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53c535ca",
   "metadata": {},
   "source": [
    "After downloading the data, we can inspect the corresponding data catalog:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da1cb1f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "dc_yml = \"./data/data_catalog.yml\"\n",
    "with open(dc_yml, \"r\") as f:\n",
    "    txt = f.read()\n",
    "print(txt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85c7918e",
   "metadata": {},
   "source": [
    "### Next steps: Model building notebooks"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8e24ef7",
   "metadata": {},
   "source": [
    "In this notebook, you have explored the basic concepts of accessing HydroMT Delft3D FM using the Command Line Interface (CLI). We have explained the contents of a **configuration file** and a **data catalog**, and how they are used to build and manage Delft3D FM models. \n",
    "\n",
    "You can now proceed to the example notebooks where Delft3D FM models will be built using an example configuration file and the artifact data. It is recommended to start with the 1D and 2D models (`build_1dmodel.ipynb` and `build_2dmodel.ipynb`) before moving on to the coupled 1D2D model (`build_1d2dmodel.ipynb`)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0dde1d6",
   "metadata": {},
   "source": [
    "Additionally, there are example notebooks that highlight the functionality of grid refinement (`update_refine_2dgrid.ipynb`) and how to plot the mesh of a model using Python (`plot_dflowfm_mesh.ipynb`)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dsd_env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.20"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}