vignettes/accuracy_indices.Rmd
accuracy_indices.RmdLet \[ \hat{e}_{i,j,t}^{[k],h} = y_{i,t+h}^{[k]} - \hat{y}_{i,j,t}^{[k],h}, \quad \begin{array}{l} i=1,\ldots,n, \\ j=0,\ldots,J, \end{array} \; t=1,\ldots, q , \; \begin{array}{l} k \in {\cal K}, \\ h=1,\ldots,h_k , \end{array} \] be the forecast error, where \(y\) and \(\hat{y}\) are the actual and the forecasted values, respectively, suffix \(i\) denotes the variable of interest, \(j\) is the forecasting technique, where \(j=0\) is the benchmark forecasting procedure, \(t\) is the forecast origin, \({\cal K}\) is the set of the time frequencies at which the series is observed, and \(h\) is the forecast horizon, whose lead time depends on the time frequency \(k\).
Denote by A\(_{i,j}^{[k],h}\) the
forecasting accuracy of the technique \(j\), computed across \(q\) forecast origins, for the \(h\)-step-ahead forecasts of the variable
\(i\) at the temporal aggregation level
\(k\). For example, A\(_{i,j}^{[k],h} = MSE_{i,j}^{[k],h}\)
(type=“mse” in score_index() function,
default), otherwise we might have A\(_{i,j}^{[k],h} = MAE_{i,j}^{[k],h}\)
(type=“mae” in score_index() function) or A\(_{i,j}^{[k],h} = RMSE_{i,j}^{[k],h}\)
(type=“rmse” in score_index() function), where
\[
\begin{array}{rcl}
MSE_{i,j}^{[k],h} & = &
\displaystyle\frac{1}{q}\displaystyle\sum_{t=1}^{q}
\left(\hat{e}_{i,j}^{[k],h}\right)^2 \\
MAE_{i,j}^{[k],h} & = &
\displaystyle\frac{1}{q}\displaystyle\sum_{t=1}^{q}
\left|\hat{e}_{i,j}^{[k],h}\right| \\
RMSE_{i,j}^{[k],h} & = &
\sqrt{\displaystyle\frac{1}{q}\displaystyle\sum_{t=1}^{q}
\left(\hat{e}_{i,j}^{[k],h}\right)^2}
\end{array}
\]
In any case, we consider the relative version of the accuracy index \(A_{i,j}^{[k],h}\), given by: \[ r_{i,j}^{[k],h} = \displaystyle\frac{A_{i,j}^{[k],h}}{A_{i,0}^{[k],h}} , \quad i=1,\ldots,n, \quad j=0,\ldots, J, \quad k \in {\cal K}, \quad h=1, \ldots, h_k, \] and use it to compute the Average relative accuracy index of the forecasting procedure \(j\), for given \(k\) and \(h\), through the geometric mean: \[ \text{AvgRelA}_{j}^{[k],h} = \left(\displaystyle\prod_{i=1}^{n} r_{i,j}^{[k],h} \right)^{\frac{1}{n}} , \quad j=0,\ldots,J . \]
We may consider the following average relative accuracy indices for selected groups of variables/time frequencies and forecast horizons:
Average relative accuracy indices for a single variable at a given time frequency, for multiple forecast horizons \[ \label{AvgRelAikh1h2} \text{AvgRelA}_{i,j}^{[k],q_1:q_2} = \left(\prod_{h=q_1}^{q_2} r_{i,j}^{[k],h}\right)^{\frac{1}{q_2 - q_1 + 1}}, \; \begin{array}{l} i=1,\ldots,n, \\ j=0,\ldots,J, \end{array} \; \begin{array}{l} k \in {\cal K}, \\ 1 \le q_1 \le q_2 \le h_k \end{array} . %i=1,\ldots,n , \; j=0,\ldots, J, \; k \in {\cal K}, \; 1 \le q_1 \le q_2 \le h_k. \] Average relative accuracy indices for a group of variables (either all, or selected groups, e.g. a: uts, b: bts) at a given time frequency, either for a single forecast horizon or across them \[ \begin{array}{rcll} \text{AvgRelA}^{[k],h}_j & = & \left(\displaystyle\prod_{i=1}^{n} r_{i,j}^{[k],h}\right)^{\frac{1}{n}}, & j=0,\ldots,J , \; k \in {\cal K}, \; h=1,\ldots,h_k \\[.5cm] \text{AvgRelA}^{[k],h}_{a,j} & = & \left(\displaystyle\prod_{i=1}^{n_a} r_{i,j}^{[k],h}\right)^{\frac{1}{n_a}}, & j=0,\ldots, J, \; k \in {\cal K} \\[.15cm] \text{AvgRelA}^{[k],h}_{b,j} & = & \left(\displaystyle\prod_{i=n_a+1}^{n} r_{i,j}^{[k],h}\right)^{\frac{1}{n_b}}, & j=0,\ldots, J, \; k \in {\cal K} \\[.15cm] \text{AvgRelA}^{[k]}_j & = & \left(\displaystyle\prod_{i=1}^{n} \prod_{h=1}^{h_k} r_{i,j}^{[k],h}\right)^{\frac{1}{n h_k}}, & \; j=0,\ldots,J, \; k \in {\cal K} \\[.15cm] \text{AvgRelA}^{[k]}_{a,j} & = &\left(\displaystyle\prod_{i=1}^{n_a} \prod_{h=1}^{h_k} r_{i,j}^{[k],h}\right)^{\frac{1}{n_a h_k}}, & j=0,\ldots, J, \; k \in {\cal K} \\[.15cm] \text{AvgRelA}^{[k]}_{b,j} & = &\left(\displaystyle\prod_{i=n_a+1}^{n} \prod_{h=1}^{h_k} r_{i,j}^{[k],h}\right)^{\frac{1}{n_b h_k}}, & j=0,\ldots, J, \; k \in {\cal K} \end{array} \]
Average relative accuracy indices for a single variable or for a group of variables (all, a: uts, b: bts), across all time frequencies and forecast horizons \[ \begin{array}{rcll} \text{AvgRelA}_{i,j} & = & \left(\displaystyle\prod_{k \in {\cal K}} \prod_{h=1}^{h_k} r_{i,j}^{[k],h}\right)^{\frac{1}{k^*+m}}, & \begin{array}{l} i=1,\ldots,n \\ j=0,\ldots,J \end{array} \\[.15cm] \text{AvgRelA}_j & = & \left(\displaystyle\prod_{i=1}^{n} \prod_{k \in {\cal K}} \prod_{h=1}^{h_k} r_{i,j}^{[k],h}\right)^{\frac{1}{n(k^*+m)}} , & j=0,\ldots,J \\[.15cm] \text{AvgRelA}_{a,j} & = & \left(\displaystyle\prod_{i=1}^{n_a} \prod_{k \in {\cal K}} \prod_{h=1}^{h_k} r_{i,j}^{[k],h}\right)^{\frac{1}{n_a(k^*+m)}} , & j=0,\ldots,J \\[.15cm] \text{AvgRelA}_{b,j} & = & \left(\displaystyle\prod_{i=n_a+1}^{n} \prod_{k \in {\cal K}} \prod_{h=1}^{h_k} r_{i,j}^{[k],h}\right)^{\frac{1}{n_b(k^*+m)}} , & j=0,\ldots,J \end{array} \]
score_index() function outputs
The score_index() function returns a summary table
called Avg\(\_\)mat
(if compact option is TRUE, default),
otherwise it returns a list of four tables. Fixed method \(j\), we have:
| \(\textbf{all}\) | \(\textbf{uts}\) | \(\textbf{bts}\) | |
| \(\textbf{m}\) | \(\text{AvgRelA}^{[m]}_j\) | \(\text{AvgRelA}^{[m]}_{a,j}\) | \(\text{AvgRelA}^{[m]}_{b,j}\) |
| \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) |
| \(\textbf{k}\) | \(\text{AvgRelA}^{[k]}_j\) | \(\text{AvgRelA}^{[k]}_{a,j}\) | \(\text{AvgRelA}^{[k]}_{b,j}\) |
| \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) |
| \(\textbf{1}\) | \(\text{AvgRelA}^{[1]}_j\) | \(\text{AvgRelA}^{[1]}_{a,j}\) | \(\text{AvgRelA}^{[1]}_{b,j}\) |
| \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) |
| \(\textbf{all}\) | \(\text{AvgRelA}_j\) | \(\text{AvgRelA}_{a,j}\) | \(\text{AvgRelA}_{b,j}\) |
| \({\cal K}\) | \(\textbf{m}\) | \(\dots\) | \(\textbf{k}\) | \(\dots\) | \(\textbf{1}\) | ||||
| \(\textbf{h}\) | \(\textbf{1}\) | \(\dots\) | \(\textbf{1}\) | \(\dots\) | \(\mathbf{h_k}\) | \(\dots\) | \(\textbf{1}\) | \(\dots\) | \(\mathbf{m}\) |
| \(\textbf{1}\) | \(\text{RelA}^{[m],1}_{1,j}\) | \(\dots\) | \(\text{RelA}^{[k],1}_{1,j}\) | \(\dots\) | \(\text{RelA}^{[k],h_k}_{1,j}\) | \(\dots\) | \(\text{RelA}^{[1],1}_{1,j}\) | \(\dots\) | \(\text{RelA}^{[1],m}_{1,j}\) |
| \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) | ||||
| \(\textbf{i}\) | \(\text{RelA}^{[m],1}_{i,j}\) | \(\dots\) | \(\text{RelA}^{[k],1}_{i,j}\) | \(\dots\) | \(\text{RelA}^{[k],h_k}_{i,j}\) | \(\dots\) | \(\text{RelA}^{[1],1}_{i,j}\) | \(\dots\) | \(\text{RelA}^{[1],m}_{i,j}\) |
| \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) | ||||
| \(\textbf{n}\) | \(\text{RelA}^{[m],1}_{n,j}\) | \(\dots\) | \(\text{RelA}^{[k],1}_{n,j}\) | \(\dots\) | \(\text{RelA}^{[k],h_k}_{n,j}\) | \(\dots\) | \(\text{RelA}^{[1],1}_{n,j}\) | \(\dots\) | \(\text{RelA}^{[1],m}_{n,j}\) |
| \({\cal K}\) | \(\textbf{m}\) | \(\dots\) | \(\textbf{k}\) | \(\dots\) | \(\textbf{1}\) | ||||
| \(\textbf{h}\) | \(\textbf{1:1}\) | \(\dots\) | \(\textbf{1:1}\) | \(\dots\) | \(\mathbf{1:h_k}\) | \(\dots\) | \(\textbf{1:1}\) | \(\dots\) | \(\mathbf{1:m}\) |
| \(\textbf{1}\) | \(\text{RelA}^{[m],1:1}_{1,j}\) | \(\dots\) | \(\text{RelA}^{[k],1:1}_{1,j}\) | \(\dots\) | \(\text{RelA}^{[k],1:h_k}_{1,j}\) | \(\dots\) | \(\text{RelA}^{[1],1:1}_{1,j}\) | \(\dots\) | \(\text{RelA}^{[1],1:m}_{1,j}\) |
| \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) | ||||
| \(\textbf{i}\) | \(\text{RelA}^{[m],1:1}_{i,j}\) | \(\dots\) | \(\text{RelA}^{[k],1:1}_{i,j}\) | \(\dots\) | \(\text{RelA}^{[k],1:h_k}_{i,j}\) | \(\dots\) | \(\text{RelA}^{[1],1:1}_{i,j}\) | \(\dots\) | \(\text{RelA}^{[1],1:m}_{i,j}\) |
| \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) | ||||
| \(\textbf{n}\) | \(\text{RelA}^{[m],1:1}_{n,j}\) | \(\dots\) | \(\text{RelA}^{[k],1:1}_{n,j}\) | \(\dots\) | \(\text{RelA}^{[k],1:h_k}_{n,j}\) | \(\dots\) | \(\text{RelA}^{[1],1:1}_{n,j}\) | \(\dots\) | \(\text{RelA}^{[1],1:m}_{n,j}\) |
| \(\textbf{m}\) | \(\dots\) | \(\textbf{k}\) | \(\dots\) | \(\textbf{1}\) | \(\dots\) | \(\mathbf{all}\) | |
| \(\textbf{1}\) | \(\text{AvgRelA}^{[m]}_{1,j}\) | \(\dots\) | \(\text{AvgRelA}^{[k]}_{1,j}\) | \(\dots\) | \(\text{AvgRelA}^{[1]}_{1,j}\) | \(\dots\) | \(\text{AvgRelA}_{1,j}\) |
| \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) | |||
| \(\textbf{i}\) | \(\text{AvgRelA}^{[m]}_{i,j}\) | \(\dots\) | \(\text{AvgRelA}^{[k]}_{i,j}\) | \(\dots\) | \(\text{AvgRelA}^{[1]}_{i,j}\) | \(\dots\) | \(\text{AvgRelA}_{i,j}\) |
| \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) | |||
| \(\textbf{n}\) | \(\text{AvgRelA}^{[m]}_{n,j}\) | \(\dots\) | \(\text{AvgRelA}^{[k]}_{n,j}\) | \(\dots\) | \(\text{AvgRelA}^{[1]}_{n,j}\) | \(\dots\) | \(\text{AvgRelA}_{n,j}\) |
| \({\cal K}\) | \(\textbf{m}\) | \(\dots\) | \(\textbf{k}\) | \(\dots\) | \(\textbf{1}\) | ||||
| \(\textbf{h}\) | \(\textbf{1}\) | \(\dots\) | \(\textbf{1}\) | \(\dots\) | \(\mathbf{h_k}\) | \(\dots\) | \(\textbf{1}\) | \(\dots\) | \(\mathbf{m}\) |
| \(\textbf{all}\) | \(\text{AvgRelA}^{[m],1}_{j}\) | \(\dots\) | \(\text{AvgRelA}^{[k],1}_{j}\) | \(\dots\) | \(\text{AvgRelA}^{[k],h_k}_{j}\) | \(\dots\) | \(\text{AvgRelA}^{[1],1}_{j}\) | \(\dots\) | \(\text{AvgRelA}^{[1],m}_{j}\) |
| \(\textbf{a}\) | \(\text{AvgRelA}^{[m],1}_{a,j}\) | \(\dots\) | \(\text{AvgRelA}^{[k],1}_{a,j}\) | \(\dots\) | \(\text{AvgRelA}^{[k],h_k}_{a,j}\) | \(\dots\) | \(\text{AvgRelA}^{[1],1}_{a,j}\) | \(\dots\) | \(\text{AvgRelA}^{[1],m}_{a,j}\) |
| \(\textbf{b}\) | \(\text{AvgRelA}^{[m],1}_{b,j}\) | \(\dots\) | \(\text{AvgRelA}^{[k],1}_{b,j}\) | \(\dots\) | \(\text{AvgRelA}^{[k],h_k}_{b,j}\) | \(\dots\) | \(\text{AvgRelA}^{[1],1}_{b,j}\) | \(\dots\) | \(\text{AvgRelA}^{[1],m}_{b,j}\) |
| \({\cal K}\) | \(\textbf{m}\) | \(\dots\) | \(\textbf{k}\) | \(\dots\) | \(\textbf{1}\) | ||||
| \(\textbf{h}\) | \(\textbf{1:1}\) | \(\dots\) | \(\textbf{1:1}\) | \(\dots\) | \(\mathbf{1:h_k}\) | \(\dots\) | \(\textbf{1:1}\) | \(\dots\) | \(\mathbf{1:m}\) |
| \(\textbf{all}\) | \(\text{AvgRelA}^{[m],1:1}_{j}\) | \(\dots\) | \(\text{AvgRelA}^{[k],1:1}_{j}\) | \(\dots\) | \(\text{AvgRelA}^{[k],1:h_k}_{j}\) | \(\dots\) | \(\text{AvgRelA}^{[1],1:1}_{j}\) | \(\dots\) | \(\text{AvgRelA}^{[1],1:m}_{j}\) |
| \(\textbf{a}\) | \(\text{AvgRelA}^{[m],1:1}_{a,j}\) | \(\dots\) | \(\text{AvgRelA}^{[k],1:1}_{a,j}\) | \(\dots\) | \(\text{AvgRelA}^{[k],1:h_k}_{a,j}\) | \(\dots\) | \(\text{AvgRelA}^{[1],1:1}_{a,j}\) | \(\dots\) | \(\text{AvgRelA}^{[1],1:m}_{a,j}\) |
| \(\textbf{b}\) | \(\text{AvgRelA}^{[m],1:1}_{b,j}\) | \(\dots\) | \(\text{AvgRelA}^{[k],1:1}_{b,j}\) | \(\dots\) | \(\text{AvgRelA}^{[k],1:h_k}_{b,j}\) | \(\dots\) | \(\text{AvgRelA}^{[1],1:1}_{b,j}\) | \(\dots\) | \(\text{AvgRelA}^{[1],1:m}_{b,j}\) |
library(FoReco)
data(FoReco_data)
# Cross-temporal framework
oct_recf <- octrec(FoReco_data$base, m = 12, C = FoReco_data$C,
comb = "bdshr", res = FoReco_data$res)$recf
oct_score <- score_index(recf = oct_recf,
base = FoReco_data$base,
test = FoReco_data$test, m = 12, nb = 5)
# Cross-sectional framework#'
# monthly base forecasts
mbase <- FoReco2matrix(FoReco_data$base, m = 12)$k1
# monthly test set
mtest <- FoReco2matrix(FoReco_data$test, m = 12)$k1
# monthly residuals
mres <- FoReco2matrix(FoReco_data$res, m = 12)$k1
# monthly reconciled forecasts
mrecf <- htsrec(mbase, C = FoReco_data$C, comb = "shr", res = mres)$recf
# score
hts_score <- score_index(recf = mrecf, base = mbase, test = mtest, nb = 5)
# Temporal framework
data(FoReco_data)
# top ts base forecasts ([lowest_freq' ... highest_freq']')
topbase <- FoReco_data$base[1, ]
# top ts residuals ([lowest_freq' ... highest_freq']')
topres <- FoReco_data$res[1, ]
# top ts test ([lowest_freq' ... highest_freq']')
toptest <- FoReco_data$test[1, ]
# top ts recf ([lowest_freq' ... highest_freq']')
toprecf <- thfrec(topbase, m = 12, comb = "acov", res = topres)$recf
# score
thf_score <- score_index(recf = toprecf, base = topbase, test = toptest, m = 12)