Forecast error metrics: sMAPE & MASE & NMAE & NMSE for cross data set comparison

• เศษส่วนเรียกว่า Fraction; เศษ เรียกว่า Numerator; ส่วน เรียกว่า Denominator

Mean Absolute Percentage Error (MAPE)

• The formula is:

  $$\text{MAPE} = \frac{1}{n} \sum_{t=1}^{n} \left( \frac{|y_t - \hat{y_t}|}{|y_t|} \right) \times 100$$

  Where:

  • $y_t$ is the actual value at time $t$.
  • $\hat{y_t}$ is the forecasted (predicted) value at time $t$.
  • $n$ is the total number of observations.
  • MAPE expresses the error as a percentage of the actual values and may exceed 100 percent.
  • Lower MAPE means better forecasting accuracy.
  • Higher MAPE indicates larger errors.

Symmetric Mean Absolute Percentage Error (sMAPE) 

• Another popular metric for evaluating the accuracy of forecasting models. It is a modified version of the Mean Absolute Percentage Error (MAPE) that addresses some of MAPE's limitations, particularly when the actual values are close to zero, which can lead to inflated percentage errors in MAPE.

• The formula for sMAPE is:

  $$\text{sMAPE} = \frac{1}{n} \sum_{t=1}^{n} \frac{|y_t - \hat{y_t}|}{\frac{|y_t| + |\hat{y_t}|}{2}} \times 100$$

  Where:

  • $y_t$ is the actual value at time $t$.
  • $\hat{y_t}$ is the forecasted value at time $t$.
  • $n$ is the number of data points.
• It is considered symmetric because it treats overestimation and underestimation equally:
• Absolute Difference: It calculates the absolute difference between the forecast and the actual value for each data point. This ensures that both overestimations (positive differences) and underestimations (negative differences) are treated as positive values.
• Sum of Actual and Forecast: In the denominator, it sums the absolute values of the actual and forecast values. This ensures that the denominator is always positive and symmetric, regardless of whether the forecast is over or underestimating the actual value.
• IMHO, it's difficult to interpret sMAPE because its denominator includes errorneous forecast values as well.

Mean Absolute Scaled Error (MASE)

• MASE is calculated as the ratio of the mean absolute error of the forecasting model to the mean absolute error of a naive forecasting model. A "naive" model, in this context, refers to a model that simply uses the previous value as the forecast value for the next time period (e.g., for time series data) i.e. เอาค่าจริงที่ t-1 มาเป็นค่า forecast ที่ t.

  The formula for MASE is:

  $$\text{MASE}=\frac{\frac{1}{n}\sum _{t=1}^n\mathrm{\mid }{y}_t-\widehat{y_t}\mathrm{\mid }}{\frac{1}{n-1}\sum _{t=2}^n\mathrm{\mid }{y}_t-y_{t-1}\mathrm{\mid }}$$

  Where:

  • $y_t$ is the actual observed value at time $t$
  • $\hat{y_t}$ is the forecasted value at time $t$
  • $n$ is the number of observations
  • The denominator is the mean absolute error of a naive forecasting algorithm (often the "random walk" method, which uses the previous period’s value as the forecast).

  Interpretation:

  • MASE = 1: The model performs as well as the naive forecast.
  • MASE < 1: The model outperforms the naive forecast (i.e., it produces lower errors).
  • MASE > 1: The model underperforms compared to the naive forecast.
• Scale independence: MASE can be used for comparison purpose across different datasets with varying scales since MASE itself has no unit.
• IMHO, MASE aims at comparing the proposed forecasting algorithm with the (in-sample one-step) naive algorithm.
• NMAE and NMSE are also scale independent but aim at quantifying the forecast errors in relation to the actual values. NMAE and NMSE are just MAE and MSE divided by one of the following normalizers:
• Range: The difference between the maximum and minimum values of the actual data ($\text{max}(y_t) - \text{min}(y_t)$).
• Mean: The mean of the actual values ($\bar{y}$).
• Median: Sometimes the median of the actual values is used.

Cf. Gemini, ChatGPT, Wikipedia