Information Gain

Information Gain measures how much information entropy is reduced after splitting the dataset on an attribute. It helps identify the attribute that provides the most information about the target class.

 is H is information entropy. D is data set.

Key Idea: The larger the reduction in entropy after the split, the greater the Information Gain. Attributes with higher Information Gain are preferred for splitting.

Applications in Decision Trees: At each node, the algorithm selects the attribute with the highest Information Gain to split the dataset.

Types of regression models

There are many types of regression models, each suited to different types of data and relationships. Some common types include:

1. Linear Regression: Models the relationship between two variables by fitting a straight line.

2. Logistic Regression: Used for binary classification, predicting probabilities for categories (e.g., yes/no, 0/1).

3. Polynomial Regression: Extends linear regression to model nonlinear relationships by using polynomial functions.

4. Ridge Regression: A type of linear regression that includes a regularization term to prevent overfitting.

5. Lasso Regression: Similar to ridge regression, but it can reduce some coefficients to zero, effectively selecting features.

6. Elastic Net Regression: Combines ridge and lasso regression for a balance between feature selection and regularization.

7. Quantile Regression: Estimates the median or other quantiles of the response variable, not just the mean.

8. Poisson Regression: Used for count data, modeling how often an event happens.

9. Ordinal Regression: Models ordinal (ranked) outcomes, where categories have an order but no specific distance between them.

10. Multinomial Logistic Regression: Extends logistic regression for multiclass classification problems.

11. Bayesian Regression: Applies Bayesian principles to linear regression for probabilistic prediction.

12. Support Vector Regression (SVR): A type of regression that uses support vector machine concepts for both linear and nonlinear relationships.

ชุดข้อมูลท่องเที่ยวไทย

https://www.travellink.go.th/dashboard?category=spending-insights&displayMode=tiles

Metrics for machine translation

BLEU (bilingual evaluation understudy) is an algorithm for evaluating the quality of text which has been machine-translated from one natural language to another. Quality is considered to be the correspondence between a machine's output and that of a human: "the closer a machine translation is to a professional human translation, the better it is"

METEOR (Metric for Evaluation of Translation with Explicit ORdering) is a metric for the evaluation of machine translation output. The metric is based on the harmonic mean of unigram precision and recall, with recall weighted higher than precision. It also has several features that are not found in other metrics, such as stemming and synonymy matching, along with the standard exact word matching. The metric was designed to fix some of the problems found in the more popular BLEU metric, and also produce good correlation with human judgement at the sentence or segment level. This differs from the BLEU metric in that BLEU seeks correlation at the corpus level.

ROUGE, or Recall-Oriented Understudy for Gisting Evaluation, is a set of metrics and a software package used for evaluating automatic summarization and machine translation software in natural language processing. The metrics compare an automatically produced summary or translation against a reference or a set of references (human-produced) summary or translation. ROUGE metrics range between 0 and 1, with higher scores indicating higher similarity between the automatically produced summary and the reference.

Image classification process

The general steps in conducting image classification research typically follow a systematic workflow, from data collection to evaluation. Here’s an outline of the key steps involved:

1. Define the Problem and Objectives

• Clearly state the research question and the specific goals of the image classification task.
• Identify the types of objects or patterns the model should classify (e.g., cats vs. dogs, handwritten digits, or specific medical conditions in images).

2. Data Collection and Preparation

• Collect or Source Images: Gather images relevant to the classification task. The dataset may be obtained from public repositories (e.g., ImageNet, CIFAR-10, MNIST) or created through manual data collection (e.g., using cameras or web scraping).
• Labeling Data: Annotate the images with the appropriate class labels. This is essential for supervised learning.
• Data Augmentation: Apply transformations to the images (e.g., rotations, flips, zooms) to artificially increase the size of the dataset and improve generalization.
• Data Splitting: Split the dataset into training, validation, and test sets, typically with ratios like 70% for training, 15% for validation, and 15% for testing.

3. Feature Extraction or Data Preprocessing

• Preprocessing: Normalize image pixel values, resize images to a fixed size, and perform any other required adjustments (e.g., grayscale conversion or color channel manipulations).
• Feature Extraction (Traditional Methods): If using traditional machine learning methods, extract features such as edges, textures, or color histograms using techniques like HOG (Histogram of Oriented Gradients) or SIFT (Scale-Invariant Feature Transform).
• Data Representation: If using deep learning, raw pixel data can be directly input into a convolutional neural network (CNN), bypassing traditional feature extraction.

4. Model Selection

• Traditional Machine Learning Models: Choose algorithms such as Support Vector Machines (SVM), K-Nearest Neighbors (KNN), or Random Forests for cases where handcrafted features are used.
• Deep Learning Models: Use CNNs (Convolutional Neural Networks), which are highly effective for image classification. You can start with a simple CNN architecture or use pre-trained models (like ResNet, VGG, or EfficientNet) through transfer learning.

5. Training the Model

• Model Initialization: Set the model parameters, define the loss function (e.g., categorical cross-entropy for multi-class classification), and choose an optimizer (e.g., Adam, SGD).
• Training: Train the model using the training dataset. This involves feeding the images into the model, adjusting weights via backpropagation, and minimizing the loss function.
• Hyperparameter Tuning: Optimize hyperparameters such as learning rate, batch size, and number of epochs for better performance.

6. Validation and Fine-Tuning

• Validation Set: Evaluate the model’s performance on the validation dataset after each training epoch to monitor overfitting.
• Model Fine-Tuning: Adjust the model architecture, add regularization techniques (e.g., dropout, batch normalization), or fine-tune hyperparameters based on the validation results.

7. Testing and Evaluation

• Test Set Evaluation: Once the model is trained, evaluate its performance on the test dataset, which was unseen during training.
• Performance Metrics: Use metrics like accuracy, precision, recall, F1-score, confusion matrix, and AUC-ROC (Area Under Curve - Receiver Operating Characteristic) to measure the model’s effectiveness.

8. Model Deployment 

Image segmentation can be used as a preprocessing step before classification. For example: 

• Object Detection: First, segment objects within an image and then classify those segmented objects separately. If your goal is to understand which parts of the image correspond to different objects or regions, then image segmentation becomes relevant.
• Saliency Detection: Use segmentation to highlight key regions of interest, which may improve classification by focusing only on relevant parts of the image.

Difference between two concepts:

• Image Classification: The task is to assign a single label to an entire image, identifying what the main object or scene is (e.g., "cat" or "dog"). The goal is not to understand the precise location of objects but to categorize the whole image.

• Image Segmentation: This involves dividing the image into multiple segments or regions, each corresponding to different objects or parts of objects, essentially assigning a label to every pixel in the image (e.g., separating the sky from the trees in a landscape).

--ChatGPT

Regularization

Regularization is a technique used in statistical modeling and machine learning to prevent overfitting, which occurs when a model learns to perform very well on training data but fails to generalize to unseen data. Regularization techniques add a penalty to the loss function to constrain the model's complexity. Here are some common regularization techniques:

1. L1 Regularization (Lasso)

• Description: Adds the absolute value of the coefficients as a penalty term to the loss function.
• Effect: Encourages sparsity in the model by driving some coefficients to zero, effectively selecting a simpler model that uses fewer features.
• Loss Function: $L = L_0 + \lambda \sum |w_i|$
  • $L_0$: original loss (e.g., mean squared error)
  • $w_i$: coefficients
  • $\lambda$: regularization parameter controlling the strength of the penalty.

2. L2 Regularization (Ridge)

• Description: Adds the square of the coefficients as a penalty term to the loss function.
• Effect: Tends to reduce the size of coefficients but does not set any to zero. It shrinks the weights more evenly across all features, making the model more stable.
• Loss Function: $L = L_0 + \lambda \sum w_i^2$

3. Elastic Net Regularization

• Description: Combines both L1 and L2 regularization. It can select features (like L1) while also encouraging smaller weights (like L2).
• Effect: Useful when there are multiple features correlated with each other.
• Loss Function: $L = L_0 + \lambda_1 \sum |w_i| + \lambda_2 \sum w_i^2$

4. Dropout

• Description: A regularization technique specifically used in neural networks where randomly selected neurons are ignored (dropped out) during training.
• Effect: Prevents co-adaptation of neurons, helping the network to generalize better by forcing it to learn robust features that are useful independently of others.

5. Early Stopping

• Description: Involves monitoring the model's performance on a validation set during training and stopping the training process when performance starts to degrade (indicating overfitting).
• Effect: Prevents the model from learning noise in the training data.

6. Data Augmentation

• Description: Increasing the amount of training data by applying transformations (e.g., rotation, scaling, flipping) to existing data.
• Effect: Helps the model generalize better by exposing it to various forms of data.

7. Weight Regularization

• Description: Adding constraints on the weights (e.g., constraining the weights to lie within a certain range).
• Effect: Helps in controlling model complexity and prevents overfitting.

8. Batch Normalization

• Description: Normalizes the output of a layer to stabilize learning, effectively acting as a form of regularization.
• Effect: Reduces internal covariate shift and can lead to faster training.

--ChatGPT

Photogrammetry

Photogrammetry is the science and technology of obtaining accurate measurements and detailed information about physical objects and environments from photographs. It involves the use of photographs, often taken from multiple angles, to reconstruct 3D models or create maps, by analyzing the geometry of the images.

งานวิจัยที่ Contribution น้อย

 รวมทั้งงานวิจัยที่ใช้ระเบียบวิธีวิจัยซ้ำกับงานวิจัยอื่น เปลี่ยนแค่ data sets เรียกว่า conceptual replication research พบได้มากในสาขา data mining เช่น classify เม็ดเลือด, classify มะเร็ง, classify X-ray images, classify สัตว์

แนวทางการเป็นกรรมการสอบความก้าวหน้า Thesis/IS

ถามให้สอดคล้องกับ Stage ของการทำวิจัยพัฒนา

ถ้าความก้าวหน้าปีแรก:

• แผน ก: ถามเกี่ยวกับ feasibility study และช่วยดูความเป็นไปได้ในการตีพิมพ์
• แผน ข: ถามเกี่ยวกับ feasibility study และช่วยดูความเป็นนวัตกรรม

ความก้าวหน้าปีท้าย:

• ให้ถามเจาะผลการทดลองเบื้องต้นที่นำเสนอ มีความยุติธรรมในการเปรียบเทียบไหม ให้อภิปรายทำไมค่าเราสูงกว่าหรือต่ำกว่า ดูการวิเคราะห์การทบทวนงานวิจัยที่เกี่ยวข้อง/ระบบงานที่เกี่ยวข้องนอกจากแง่มุมที่นิสิตได้เรียนรู้แล้วต้องมีแง่มุมที่เป็นสาเหตุของการทำวิจัยพัฒนาครั้งนี้ด้วย

สิ่งที่ควรเลี่ยง

• อย่าแนะนำ general mistakes เช่น เขียนวัตถุประสงค์ผิด งานวิจัยที่เกี่ยวข้องน้อยไปเก่าไป เพราะเป็นหน้าที่ของที่ปรึกษา ให้ถามเจาะลึกเทคนิคที่ใช้

สิ่งควรทำ

• เวลาแนะนำใช้คำว่า น่าจะทำแบบนั้นแบบนี้ ถ้าทำได้ก็ดี ไม่ควรสั่งถ้าไม่ใช่ Advisee อย่าพยายามเอาชนะคะคาน
• ถามเป็นความรู้

หัวข้อที่ไม่เชี่ยวชาญโดยตรง

• กล่าวถึงระเบียบวิธีวิจัยภาพรวมของสาขานั้น เช่น NLP (https://drjiw.blogspot.com/2024/08/world-of-nlp.html) Image processing (https://drjiw.blogspot.com/2024/10/general-steps-in-conducting-image.html) เพื่อสร้างความน่าเชื่อถือให้ผู้ฟังก่อนเริ่มทำการ comment
• ควรถาม objective/loss function ที่จะ optimize คืออะไรช่วยอธิบายสูตร และมีวิธีการ optimize ให้ได้ค่าดีที่สุดอย่างไร
• ถามรูปแบบของผลลัพธ์การวิจัยพัฒนา

Examples of GAI (Generative AI)

While Large Language Models (LLMs) are a prominent example of Generative AI, they're not the only ones. Here are some other notable types:

1. Generative Adversarial Networks (GANs)

 * How they work: GANs consist of two neural networks: a generator that creates new data, and a discriminator that evaluates its authenticity. They compete, improving each other over time.

 * Applications: Image generation, style transfer, and creating realistic synthetic data.

2. Variational Autoencoders (VAEs)

 * How they work: VAEs are a type of neural network that learns a latent representation of data. They can generate new data points that are similar to the training data.

 * Applications: Image generation, data imputation, and anomaly detection.

3. Diffusion Models

 * How they work: Diffusion models gradually add noise to data and then learn to reverse the process. This can be used to generate new data points.

 * Applications: Image generation, text-to-image generation, and audio synthesis.

4. Flow-based Models

 * How they work: Flow-based models learn a sequence of invertible transformations that can map data to and from a simple distribution. This can be used to generate new data points.

 * Applications: Image generation, density estimation, and anomaly detection.

5. Neural Style Transfer

 * How it works: This technique combines the content of one image with the style of another using neural networks.

 * Applications: Artistic creation, image editing, and video effects.

Nondifferentiable functions

 1. Absolute Value Function

$$f(x) = |x|$$

This function has a sharp corner (or cusp) at $x = 0$. The slope changes abruptly from negative on the left side of the origin to positive on the right side, making it non-differentiable at $x = 0$.

2. Piecewise Function

A piecewise function can be non-differentiable at the points where its pieces join if there's a sudden change in slope.

$$f(x)=\{\begin{array}{ll}x+1& \text{for }x<0,\\ -x+1& \text{for }x\ge 0.\end{array}$$

This function is non-differentiable at $x = 0$ because the slopes from the left and right do not match.

3. Functions with Vertical Tangents

The function

$$f(x) = \sqrt[3]{x}$$

is differentiable everywhere except at $x = 0$, where the slope becomes infinitely steep (a vertical tangent).

4. Discontinuous Functions

A function that has a jump or gap is not differentiable at the point of discontinuity. For example:

$$f(x) = \begin{cases} 1 & \text{for } x < 0, \\ 2 & \text{for } x \geq 0. \end{cases}$$

This function is non-differentiable at $x = 0$ due to the discontinuity.

In general, a function is not differentiable at points where it is not continuous or where it has abrupt changes in behavior, such as sharp corners or vertical slopes.

--ChatGPT

PointCloud

In 3D modeling, a point cloud is a set of data points in a 3D coordinate system—commonly known as the XYZ axes.

https://en.wikipedia.org/wiki/Point_cloud

Geospatial DBMS

https://postgis.net/

PostGIS extends the capabilities of the PostgreSQL relational database by adding support for storing, indexing, and querying geospatial data.

Retrieval Augmented Generation (RAG) for GAI

 https://www.gai.in.th/prompting-techniques/retrieval-augmented-generation-rag

ChatGPT vs Gemini

 From my experience, ChatGPT is good at linguistic tasks but math while Gemini is opposite.

Forecast error metrics: sMAPE & MASE & NMAE & NMSE for cross data set comparison

• เศษส่วนเรียกว่า Fraction; เศษ เรียกว่า Numerator; ส่วน เรียกว่า Denominator

Mean Absolute Percentage Error (MAPE)

• The formula is:

  $$\text{MAPE} = \frac{1}{n} \sum_{t=1}^{n} \left( \frac{|y_t - \hat{y_t}|}{|y_t|} \right) \times 100$$

  Where:

  • $y_t$ is the actual value at time $t$.
  • $\hat{y_t}$ is the forecasted (predicted) value at time $t$.
  • $n$ is the total number of observations.
  • MAPE expresses the error as a percentage of the actual values and may exceed 100 percent.
  • Lower MAPE means better forecasting accuracy.
  • Higher MAPE indicates larger errors.

Symmetric Mean Absolute Percentage Error (sMAPE) 

• Another popular metric for evaluating the accuracy of forecasting models. It is a modified version of the Mean Absolute Percentage Error (MAPE) that addresses some of MAPE's limitations, particularly when the actual values are close to zero, which can lead to inflated percentage errors in MAPE.

• The formula for sMAPE is:

  $$\text{sMAPE} = \frac{1}{n} \sum_{t=1}^{n} \frac{|y_t - \hat{y_t}|}{\frac{|y_t| + |\hat{y_t}|}{2}} \times 100$$

  Where:

  • $y_t$ is the actual value at time $t$.
  • $\hat{y_t}$ is the forecasted value at time $t$.
  • $n$ is the number of data points.
• It is considered symmetric because it treats overestimation and underestimation equally:
• Absolute Difference: It calculates the absolute difference between the forecast and the actual value for each data point. This ensures that both overestimations (positive differences) and underestimations (negative differences) are treated as positive values.
• Sum of Actual and Forecast: In the denominator, it sums the absolute values of the actual and forecast values. This ensures that the denominator is always positive and symmetric, regardless of whether the forecast is over or underestimating the actual value.
• IMHO, it's difficult to interpret sMAPE because its denominator includes errorneous forecast values as well.

Mean Absolute Scaled Error (MASE)

• MASE is calculated as the ratio of the mean absolute error of the forecasting model to the mean absolute error of a naive forecasting model. A "naive" model, in this context, refers to a model that simply uses the previous value as the forecast value for the next time period (e.g., for time series data) i.e. เอาค่าจริงที่ t-1 มาเป็นค่า forecast ที่ t.

  The formula for MASE is:

  $$\text{MASE}=\frac{\frac{1}{n}\sum _{t=1}^n\mathrm{\mid }{y}_t-\widehat{y_t}\mathrm{\mid }}{\frac{1}{n-1}\sum _{t=2}^n\mathrm{\mid }{y}_t-y_{t-1}\mathrm{\mid }}$$

  Where:

  • $y_t$ is the actual observed value at time $t$
  • $\hat{y_t}$ is the forecasted value at time $t$
  • $n$ is the number of observations
  • The denominator is the mean absolute error of a naive forecasting algorithm (often the "random walk" method, which uses the previous period’s value as the forecast).

  Interpretation:

  • MASE = 1: The model performs as well as the naive forecast.
  • MASE < 1: The model outperforms the naive forecast (i.e., it produces lower errors).
  • MASE > 1: The model underperforms compared to the naive forecast.
• Scale independence: MASE can be used for comparison purpose across different datasets with varying scales since MASE itself has no unit.
• IMHO, MASE aims at comparing the proposed forecasting algorithm with the (in-sample one-step) naive algorithm.
• NMAE and NMSE are also scale independent but aim at quantifying the forecast errors in relation to the actual values. NMAE and NMSE are just MAE and MSE divided by one of the following normalizers:
• Range: The difference between the maximum and minimum values of the actual data ($\text{max}(y_t) - \text{min}(y_t)$).
• Mean: The mean of the actual values ($\bar{y}$).
• Median: Sometimes the median of the actual values is used.

Cf. Gemini, ChatGPT, Wikipedia

Java's key advantages over Python

 1. Type Safety & Reliability

- Static typing catches errors at compile-time rather than run time error as in Python 

- Stronger type checking

- Less runtime surprises

- Better for large codebases

- Clearer code contracts

2. Performance

- Faster execution speed

- True multithreading (no GIL)

- Better memory management

- More efficient for CPU-intensive tasks

- Better scalability

3. Enterprise Features

- Robust frameworks (Spring)

- Better security features

- Strong enterprise tooling

- Better transaction management

- Industry-standard enterprise solutions

4. Team Development

- Easier to maintain in large teams

- Better code organization

- Clearer package structure

- Enhanced refactoring capabilities

- Better for modular development

5. Tools & IDE Support

- Superior debugging tools

- Better code analysis

- More sophisticated IDEs

- Better profiling tools

- Comprehensive development environments

6. Stability & Maintenance

- Better backward compatibility

- Longer support cycles

- More stable ecosystem

- Clearer deprecation paths

- Better for long-term projects

7. Security

- Built-in security manager

- Better access control

- Enterprise-grade security

- More secure by design

- Better authentication/authorization

8. Documentation & Standards

- More standardized practices

- Better documentation

- Clearer coding standards

- Industry-wide best practices

- More consistent coding patterns

This makes Java particularly suitable for:

- Enterprise applications

- Large-scale systems

- Mission-critical software

- Long-term projects

- Team-based development

While Python excels in:

- Rapid prototyping

- Data science

- Script automation

- Small to medium projects

- Quick development cycles

Python is not for enterprise applications.

Because it lacks extensibility and maintainability unlike Java language.

Basic Python (high school levels)

 https://colab.research.google.com/drive/1L2bi2Fjaxo2Ciz3z0AXCsUd4AXNmM4vL?usp=sharing

https://colab.research.google.com/drive/1oWbaWd7EcZHtlDIRIedBCZeen6SwSikB?usp=sharing

Speaking to the public

 

Customgpt VS Chatgpt

 CustomGPT utilizes ChatGPT for transfer learning in two primary ways:

1. Foundation Model: ChatGPT serves as the foundation for CustomGPT models. This means that the underlying architecture and knowledge base of ChatGPT are used as a starting point for training CustomGPT models.
2. Fine-Tuning: CustomGPT models are further trained on specific datasets relevant to their intended use cases. This process, known as fine-tuning, allows the models to learn and adapt to the unique characteristics of the target domain.

By combining ChatGPT's general-purpose capabilities with fine-tuning on specific datasets, CustomGPT can create highly specialized AI models that are tailored to perform well in various applications.

Neural Net's Learning rate adjustment

 One cycle policy (OCP)  is increasing and decreasing learning rate (i.e. adaptive learning rate) between upper & lower bounds to avoid slow training as much as possible.

https://medium.com/@varunsivamani/one-cycle-policy-a-deep-understanding-6d4d352ec7b1

https://medium.com/dsnet/the-1-cycle-policy-an-experiment-that-vanished-the-struggle-in-training-neural-nets-184417de23b9

SLA Tiers

Level 1 – 7/24/4 

• 7 days a week, 24 hours a day, reaction within 4 hours 
• 7×24 Help Desk 
• 7×24 Monitoring 
• Initial Phone-Based Diagnosis 
• Field Engineer on-site (if required) within 4 hours after customer contact 
• On-Site Spares Kit 
• Parts Replacement 
• Certified Experts 

Level 2 – 5/9/4 

• 8 hours a day, 5 working days per week, reaction within 4 hours 
• 8am-5pm Help Desk 
• 8am-5pm Monitoring 
• Initial Phone-Based Diagnosis 
• Field Engineer on-site (if required) within 4 hours after customer contact 
• On-Site Spares Kit 
• Parts Replacement 
• Certified Experts 

Level 3 – Next Day 

• Next business day response/parts 
• Initial Phone-Based Diagnosis  
• Field Engineer on-site (if required) the following business day after customer contact 
• On-Site Spares Kit 
• Parts Replacement 
• Certified Experts 

https://www.parkplacetechnologies.com/blog/what-does-24x7x4-support-mean-for-your-business/

Predicting Continuous Values with Transformers

Gemini:

Transformers can be effectively used to predict continuous values. While they were initially designed for natural language processing (NLP) tasks, their ability to capture long-range dependencies and complex patterns has made them versatile for various machine learning applications, including regression problems.

How Transformers Work for Regression

 * Input Encoding: The continuous values are typically encoded into a sequence of numerical representations, often using techniques like:

   * Embedding: For categorical variables

   * Normalization: For numerical variables

 * Positional Encoding: To provide the model with information about the relative position of each element in the sequence, positional encoding is added.

 * Transformer Layers: The input sequence is processed through multiple transformer layers, each consisting of:

   * Self-Attention: Captures relationships between different elements in the sequence.

   * Feed-forward Network: Applies a non-linear transformation to each element.

 * Output Layer: The final layer is a linear layer that maps the output of the transformer layers to a continuous value.

Advantages of Using Transformers for Regression

 * Long-range dependencies: Transformers can capture long-range dependencies, making them suitable for time series forecasting or tasks involving complex relationships between distant elements.

 * Flexibility: They can handle both sequential and non-sequential data, providing flexibility in various applications.

 * State-of-the-art performance: Transformers have achieved state-of-the-art results in many regression tasks, including time series forecasting and medical image analysis.

Key Considerations

 * Data preprocessing: Proper data preprocessing, including normalization and handling missing values, is crucial for optimal performance.

 * Hyperparameter tuning: Experimenting with different hyperparameters, such as the number of layers, attention heads, and hidden dimensions, can help improve results.

 * Computational resources: Training large transformer models can be computationally expensive, requiring significant GPU resources.

In conclusion, Transformers are a powerful tool for predicting continuous values, especially in tasks involving complex relationships and long-range dependencies. By understanding their architecture and key considerations, you can effectively leverage them for your regression problems.

Claude:

Transformers can be used to predict continuous values. This is often referred to as "regression" in machine learning terminology. Here's a brief overview:

1. Output layer modification: Instead of using a softmax layer for classification, you'd use a linear layer to output a single continuous value.

2. Loss function: You'd typically use mean squared error (MSE) or mean absolute error (MAE) as the loss function, rather than cross-entropy used in classification tasks.

3. Architecture: The core Transformer architecture (self-attention, feed-forward layers, etc.) remains largely the same.

4. Applications: This approach has been used successfully in various domains, including:

   - Time series forecasting

   - Financial predictions

   - Scientific modeling (e.g., predicting physical properties)

5. Variants: Some models like Temporal Fusion Transformers (TFT) are specifically designed for time series regression tasks.

Example:

https://medium.com/@Matthew_Frank/stock-price-prediction-using-transformers-2d84341ff213#:~:text=A%20Global%20Average%20Pooling%20layer,values%2C%20such%20as%20stock%20prices.

Benchmark & Baseline

Benchmarking involves measuring and comparing the performance of systems, models, or processes against a standard or across different systems. This process helps to evaluate how well a model or system performs in relation to known standards or to other models. In machine learning, the benchmark is usually set by the performance of the baseline or other leading models.

Benchmarking uses baselines: The baseline serves as the initial point of comparison in benchmarking. When evaluating a model or system, the baseline provides the first performance standard. If a model performs better than the baseline, it’s an indicator that the model has some value, and further benchmarking against other models can help assess its true effectiveness.

Baseline establishes expectations: Without a baseline, benchmarking would lack a clear starting point. By defining what "acceptable" or "expected" performance looks like, the baseline enables meaningful benchmarking comparisons.

Normalized RMSE & Normalized MAE

To compare model performance across different datasets by scaling the error metrics, you can normalize both RMSE and MAE:

Normalized RMSE:

• using the mean of data set: $NRMSE=\frac{RMSE}{\overline{y}}$ (i.e., Coefficient of Variation of RMSE)

• using the difference between maximum and minimum in data set: $NRMSE=\frac{RMSE}{y_{max}-y_{min}}$,

• using the standard deviation of data set: $NRMSE=\frac{RMSE}{\sigma }$, or

• using the interquartile range of data set; $NRMSE=\frac{RMSE}{Q1-Q3}$, i.e. the difference between 25th and 75th percentile, of observations.

https://www.marinedatascience.co/blog/2019/01/07/normalizing-the-rmse/

Normalized MAE: 

• aka. Coefficient of Variation of MAE; Coefficient of Variation (CV) =  the ratio of the standard deviation ${\displaystyle \sigma }$ to the mean ${\displaystyle \mu }$, ${\displaystyle CV=\frac{\sigma }{\mu }.}$

• using the mean of data set: $N\mathrm{MA}E=\frac{\mathrm{MA}E}{\overline{y}}$ 

https://d2uars7xkdmztq.cloudfront.net/app_resources/38997/documentation/135539_en.pdf