\caption{\label{fig:flow-diagram}\textbf{Diagram for the data flow during automatized validation procedure.} The main data flow is from left to right. The arrows depict the particular flow of the data from one validation task to the next. The raw model/reference data (left column) is first pre-processed into a generic format (right column) and subsequently analyzed by \cdo\ operations (upper row). The figures are then plotted by Python tools (lower row) and compiled into a validation report (bottom).}
\end{figure}
As a first indicator for the model's accuracy, temporal means of two-dimensional surface variables $\langle\phi\rangle_S(x,y)$ are considered for different seasons $S$, i.e.
First, the raw data of one model within the \esm\ as well as the chosen reference data are pre-processed such that all considered model variables are stored in files with the name of the model variable.
%
where $t$ are all time steps that are contained in season $S$. The number of these time steps is given by $N_S$, where the considered time period is \ValidationTime.
The corresponding two-dimensional seasonal anomaly $\langle\Delta\phi\rangle_S(x,y)$ with respect to a reference field $\phi_{\mathrm{ref}}(x,y,t)$ is then consequently
For instance the MOM variable \texttt{SST} (sea-surface temperature) is stored in file \texttt{SST.nc} that is the broadly used NetCDF~\cite{rew1990netcdf} format.
The same step is applied to the reference data, where usually the variable is renamed such that it coincides with model's variable name.
%
The smaller the magnitude of these anomalies are the better is the performance.
\subsection{Time series}
In addition to the two-dimensional data also time series of model and reference data are compared.
Additionally other transformations can be applied such as unit conversion, multiplication by a factor, etc.
%
The time series are considered at various coordinates where measurement stations are located as well as spatial means over certain regions.
The stations and regions may be naturally different for the atmospheric and the ocean model, as it is depicted in \Fig{fig:stations-and-regions}.
Practically these steps are implemented as calls of the \cdo~\cite{schulzweida_uwe_2019_2558193} tool and can be configured via a configuration file that is described below.
\caption{\label{fig:stations-and-regions-CCLM} Stations and regions that are used for validation of the atmospheric component (\cclm) of the \esm.}
\end{subfigure}
\captionsetup{width=\linewidth}
\caption{\label{fig:stations-and-regions}
\textbf{Particularly analyzed domains of the ocean and atmospheric models.}
}
\end{figure}
\subsection{Taylor Diagrams}
Since comparing time series by eye will only allow qualitative judgment of the model results, Taylor diagrams~\citesqr{taylor2001} are created for each of the above mentioned time series.
In order to calculate differences between model and reference data, the latter is automatically remapped onto the model's grid.
%
Note that not only ERA5 but also any other data that can be transformed in the described way would be a suitable reference.
%
Taylor diagrams graphically indicate which of several model data represents best a given reference data.
Note further that the pre-processing might be very specific to the model/reference since usually different models/references feature different data formats.
%
In order to quantify the degree of correspondence between the modeled and observed behavior, three statistical measures determine the diagram, i.e. the Pearson correlation coefficient, the \rmse, and the standard deviation.
However, after these very first pre-processing steps the model/reference data format is intended to be model-independent since only the variable's name is needed to perform the following analysis steps.
%
Here both data, model and reference, consist of the same number of samples that correspond to a time series starting from \ValidationTime\ post-processed with different temporal means.
This generic data format avoids code duplication since all of the analysis and the plotting scripts can then be equally applied to different models within the \esm\ having different reference data.
The analysis and the plotting steps of validation procedure are configured in an own module implemented in the programming language Python~\cite{python3} via a so-called Python \textit{dictionary}.
%
Each model variable of interest is assigned to a key with the variable's name in that dictionary.
%
The value belonging to this key is a dictionary itself, that can contain the
\begin{itemize}
\item seasons for temporal means, i.e. a list months to be averaged over
\item stations defined by coordinates
\item regions defined by coordinates of a rectangle or a mask file that cuts the particular region from the data
\item temporal mean operations that can be applied to time series data and vertical profiles that are extracted for the stations and regions
\item file pattern to the reference data files that contain the corresponding reference variable
\item configuration for the plotting
\item long name and description of the variable
\end{itemize}
\subsection{Cost functions}
The cost function $c$ as it is defined here, further summarizes the information given in a Taylor diagram.
The analysis is performed according to the configuration by Python scripts that mainly call \cdo\ routines.
%
It measures the \rmse
In addition Taylor diagrams and cost functions are calculated for the time series data.
Importantly, all interim data that is produced during the analysis is stored in NetCDF files to ensure reproducibility of the figures.
%
of the model data $\phi(t)$in units of the standard deviation $\sigma_{\mathrm{ref}}$ of reference data $\phi_{\mathrm{ref}}(t)$, i.e.
\begin{align}
c = \epsilon / \sigma_{\mathrm{ref}}.
\end{align}
Both data consist of $N$ samples corresponding to a time series starting from $t_1$ and ending at $t_N$, i.e. spanning a time of \ValidationTime.
\subsection{Vertical profiles}
In order to go beyond the analysis of surface fields, vertical profiles of important ocean state variables are compared against observation data at particular stations.
Subsequently to the analysis, the results are plotted with the help of various Python libraries.
%
The vertical profiles are generated from a four-dimensional field $\phi(x, y, z, t)$ at the chosen stations $\zeta$ (i.e. fixing $x = x_\zeta$ and $y = y_\zeta$ and using remapping to nearest neighbors) accompanied by performing the configured seasonal means for the aforementioned seasons $S$.
The actual plotting script for each figure is provided to the user in order to allow later customizations.
%
In other words, the vertical profile for a station $\zeta$ and seasons $S$ is given by
The plots are then compiled into a single report that is either provided as a Markdown~\cite{markdown-guide} file, an interactive Jupyter Notebook~\cite{kluyver2016jupyter} or as ready-to-read PDF file.
The described analysis is automatically done within developed framework that is driving the coupled \esm, as it will be published elsewhere with technical details.
The Jupyter Notebook format enables the user to adapt the individual plotting scripts to customize figures to be used in publications or other documents.
%
An example of two validation reports in the PDF format can be found in \Sec{sec:appendix}.
\section{Impact of the chosen exchange grid}
@ -591,6 +563,28 @@ One can suppose from \Fig{fig:remappings-atmos}, that largest local inconsistenc
The impact of these inconsistencies is quantitatively discussed in \open{\Sec{???}}.
\subsection{Instability with atmospheric exchange}
