Benchmark & Baseline

Benchmarking involves measuring and comparing the performance of systems, models, or processes against a standard or across different systems. This process helps to evaluate how well a model or system performs in relation to known standards or to other models. In machine learning, the benchmark is usually set by the performance of the baseline or other leading models.

Benchmarking uses baselines: The baseline serves as the initial point of comparison in benchmarking. When evaluating a model or system, the baseline provides the first performance standard. If a model performs better than the baseline, it’s an indicator that the model has some value, and further benchmarking against other models can help assess its true effectiveness.

Baseline establishes expectations: Without a baseline, benchmarking would lack a clear starting point. By defining what "acceptable" or "expected" performance looks like, the baseline enables meaningful benchmarking comparisons.

Normalized RMSE & Normalized MAE

To compare a model's performance across different datasets by scaling the error metrics, you can normalize both RMSE and MAE:

Normalized RMSE:

• using the mean of data set: $NRMSE=\frac{RMSE}{\overline{y}}$ (i.e., Coefficient of Variation of RMSE)

• using the difference between maximum and minimum in data set: $NRMSE=\frac{RMSE}{y_{max}-y_{min}}$,

• using the standard deviation of data set: $NRMSE=\frac{RMSE}{\sigma }$, or

• using the interquartile range of data set; $NRMSE=\frac{RMSE}{Q1-Q3}$, i.e. the difference between 25th and 75th percentile, of observations.

https://www.marinedatascience.co/blog/2019/01/07/normalizing-the-rmse/

Normalized MAE: 

• aka. Coefficient of Variation of MAE; Coefficient of Variation (CV) =  the ratio of the standard deviation ${\displaystyle \sigma }$ to the mean ${\displaystyle \mu }$, ${\displaystyle CV=\frac{\sigma }{\mu }.}$

• using the mean of data set: $N\mathrm{MA}E=\frac{\mathrm{MA}E}{\overline{y}}$ 

https://d2uars7xkdmztq.cloudfront.net/app_resources/38997/documentation/135539_en.pdf

===

What is the Coefficient of Variation (CV)?

The Coefficient of Variation (CV) is a standardized measure of the dispersion of a probability distribution or frequency distribution.

In plain English: It tells you how much variation there is relative to the mean.

The Formula

$$CV = \frac{\sigma}{\mu}$$

Where:

• $\sigma$ = Standard deviation (the measure of absolute spread)
  
  
• $\mu$ = Mean (the average)
  
  

(Note: It is often expressed as a percentage by multiplying the result by 100: $CV = (\frac{\sigma}{\mu}) \times 100\%$)

Why Do We Need It? (The Problem CV Solves)

Standard deviation ($\sigma$) is measured in the same units as your data. This makes it hard to compare the variability of two different datasets if they have different units or drastically different means.

Example: Elephants vs. Mice

Imagine you are a biologist measuring weights:

• Dataset A (Elephants): Mean weight $\mu = 4,000 \text{ kg}$, Standard deviation $\sigma = 400 \text{ kg}$.
  
  
• Dataset B (Mice): Mean weight $\mu = 20 \text{ g}$, Standard deviation $\sigma = 2 \text{ g}$.
  
  

If you just look at standard deviation, elephants look 200 times more variable ($400$ vs $2$). But that is misleading because elephants are massive.

Let's look at the Coefficient of Variation:

• CV for Elephants: $\frac{400}{4000} = 0.10$ (or 10%)
  
  
• CV for Mice: $\frac{2}{20} = 0.10$ (or 10%)
  
  

Once normalized by the mean, you realize that both populations have the exact same relative amount of variability.

Key Characteristics & Use Cases

1. Unitless: Because you divide a unit by the same unit (e.g., kilograms divided by kilograms), the CV has no units. This allows you to compare completely different types of data (e.g., comparing the volatility of stock prices against daily temperature fluctuations).
   
   
2. Finance & Risk Management: In portfolio management, CV is often used as the Sharpe ratio (or related risk-to-return metrics) to see how much risk (standard deviation) you are taking per unit of expected return (mean).
   
   
3. Machine Learning / Feature Engineering: CV is often used to check feature stability. If a numerical feature has a very low CV across your dataset, it is essentially a constant and might provide little to no predictive value for a machine learning model.
   
   

Limitations

• Unstable near zero: If your mean ($\mu$) is close to zero, the CV explodes to infinity or becomes wildly uninterpretable, making it useless for data centered around zero (like temperature in Celsius or profit/loss statements that swing between positive and negative).