While Large Language Models (LLMs) are a prominent example of Generative AI, they're not the only ones. Here are some other notable types:
1. Generative Adversarial Networks (GANs)
* How they work: GANs consist of two neural networks: a generator that creates new data, and a discriminator that evaluates its authenticity. They compete, improving each other over time.
* How they work: VAEs are a type of neural network that learns a latent representation of data. They can generate new data points that are similar to the training data.
* Applications: Image generation, data imputation, and anomaly detection.
3. Diffusion Models
* How they work: Diffusion models gradually add noise to data and then learn to reverse the process. This can be used to generate new data points.
* Applications: Image generation, text-to-image generation, and audio synthesis.
4. Flow-based Models
* How they work: Flow-based models learn a sequence of invertible transformations that can map data to and from a simple distribution. This can be used to generate new data points.
* Applications: Image generation, density estimation, and anomaly detection.
5. Neural Style Transfer
* How it works: This technique combines the content of one image with the style of another using neural networks.
* Applications: Artistic creation, image editing, and video effects.
This function has a sharp corner (or cusp) at x=0. The slope changes abruptly from negative on the left side of the origin to positive on the right side, making it non-differentiable at x=0.
2. Piecewise Function
A piecewise function can be non-differentiable at the points where its pieces join if there's a sudden change in slope.
This function is non-differentiable at x=0 because the slopes from the left and right do not match.
3. Functions with Vertical Tangents
The function
is differentiable everywhere except at x=0, where the slope becomes infinitely steep (a vertical tangent).
4. Discontinuous Functions
A function that has a jump or gap is not differentiable at the point of discontinuity. For example:
This function is non-differentiable at x=0 due to the discontinuity.
In general, a function is not differentiable at points where it is not continuous or where it has abrupt changes in behavior, such as sharp corners or vertical slopes.
yt^ is the forecasted (predicted) value at time t.
n is the total number of observations.
MAPE expresses the error as a percentage of the actual values and may exceed 100 percent.
Lower MAPE means better forecasting accuracy.
Higher MAPE indicates larger errors.
Symmetric Mean Absolute Percentage Error (sMAPE)
Another popular metric for evaluating the accuracy of forecasting models. It is a modified version of the Mean Absolute Percentage Error (MAPE) that addresses some of MAPE's limitations, particularly when the actual values are close to zero, which can lead to inflated percentage errors in MAPE.
The formula for sMAPE is:
sMAPE=n1t=1∑n2∣yt∣+∣yt^∣∣yt−yt^∣×100
Where:
yt is the actual value at time t.
yt^ is the forecasted value at time t.
n is the number of data points.
It is considered symmetric because it treats overestimation and underestimation equally:
Absolute Difference: It calculates the absolute difference between the forecast and the actual value for each data point. This ensures that both overestimations (positive differences) and underestimations (negative differences) are treated as positive values.
Sum of Actual and Forecast: In the denominator, it sums the absolute values of the actual and forecast values. This ensures that the denominator is always positive and symmetric, regardless of whether the forecast is over or underestimating the actual value.
IMHO, it's difficult to interpret sMAPE because its denominator includes errorneous forecast values as well.
Mean Absolute Scaled Error (MASE)
MASE is calculated as the ratio of the mean absolute error of the forecasting model to the mean absolute error of a naive forecasting model. A "naive" model, in this context, refers to a model that simply uses the previous value as the forecast value for the next time period (e.g., for time series data) i.e. เอาค่าจริงที่ t-1 มาเป็นค่า forecast ที่ t.
The formula for MASE is:
Where:
yt is the actual observed value at time t
yt^ is the forecasted value at time t
n is the number of observations
The denominator is the mean absolute error of a naive forecasting algorithm (often the "random walk" method, which uses the previous period’s value as the forecast).
Interpretation:
MASE = 1: The model performs as well as the naive forecast.
MASE < 1: The model outperforms the naive forecast (i.e., it produces lower errors).
MASE > 1: The model underperforms compared to the naive forecast.
Scale independence: MASE can be used for comparison purpose across different datasets with varying scales since MASE itself has no unit.
IMHO, MASE aims at comparing the proposed forecasting algorithm with the (in-sample one-step) naive algorithm.
NMAE and NMSE are also scale independent but aim at quantifying the forecast errors in relation to the actual values. NMAE and NMSE are just MAE and MSE divided by one of the following normalizers:
Range: The difference between the maximum and minimum values of the actual data (max(yt)−min(yt)).
Mean: The mean of the actual values (yˉ).
Median: Sometimes the median of the actual values is used.
CustomGPT utilizes ChatGPT for transfer learning in two primary ways:
Foundation Model: ChatGPT serves as the foundation for CustomGPT models. This means that the underlying architecture and knowledge base of ChatGPT are used as a starting point for training CustomGPT models.
Fine-Tuning: CustomGPT models are further trained on specific datasets relevant to their intended use cases. This process, known as fine-tuning, allows the models to learn and adapt to the unique characteristics of the target domain.
By combining ChatGPT's general-purpose capabilities with fine-tuning on specific datasets, CustomGPT can create highly specialized AI models that are tailored to perform well in various applications.
One cycle policy (OCP) is increasing and decreasing learning rate (i.e. adaptive learning rate) between upper & lower bounds to avoid slow training as much as possible.
Transformers can be effectively used to predict continuous values. While they were initially designed for natural language processing (NLP) tasks, their ability to capture long-range dependencies and complex patterns has made them versatile for various machine learning applications, including regression problems.
How Transformers Work for Regression
* Input Encoding: The continuous values are typically encoded into a sequence of numerical representations, often using techniques like:
* Embedding: For categorical variables
* Normalization: For numerical variables
* Positional Encoding: To provide the model with information about the relative position of each element in the sequence, positional encoding is added.
* Transformer Layers: The input sequence is processed through multiple transformer layers, each consisting of:
* Self-Attention: Captures relationships between different elements in the sequence:
===
Self-attention is a mechanism in Transformer models that allows each token to determine which other tokens are most relevant for understanding context.
First, each word/token is converted into a numerical vector representation called an embedding. This is conceptually similar to Word2Vec, where words are represented as vectors capturing semantic meaning.
Example:
[ \text{cat} \rightarrow x=[1.2,\ 0.5,\ -0.7] ]
This embedding vector is then transformed into three learned vectors:
Component
Meaning
Query (Q)
what the token is searching for
Key (K)
what the token offers/matches
Value (V)
information carried by the token
They are generated using learned matrices:
Q=xW_Q
K=xW_K
V=xW_V
The matrices , , and in Transformer models are not manually designed. They are learned automatically during training using backpropagation and gradient descent.
The model computes an attention score between tokens by comparing Queries and Keys using a dot product:
score=Q K
Higher scores mean stronger relevance.
After normalization (softmax), the scores become attention weights that determine how much information from each Value vector contributes to the final representation.
Thus, self-attention allows each word to dynamically focus on the most relevant words in the sentence, enabling transformers to capture contextual meaning and long-range relationships efficiently.
======
* Feed-forward Network: Applies a non-linear transformation to each element.
* Output Layer: The final layer is a linear layer that maps the output of the transformer layers to a continuous value.
Advantages of Using Transformers for Regression
* Long-range dependencies: Transformers can capture long-range dependencies, making them suitable for time series forecasting or tasks involving complex relationships between distant elements.
* Flexibility: They can handle both sequential and non-sequential data, providing flexibility in various applications.
* State-of-the-art performance: Transformers have achieved state-of-the-art results in many regression tasks, including time series forecasting and medical image analysis.
Key Considerations
* Data preprocessing: Proper data preprocessing, including normalization and handling missing values, is crucial for optimal performance.
* Hyperparameter tuning: Experimenting with different hyperparameters, such as the number of layers, attention heads, and hidden dimensions, can help improve results.
* Computational resources: Training large transformer models can be computationally expensive, requiring significant GPU resources.
In conclusion, Transformers are a powerful tool for predicting continuous values, especially in tasks involving complex relationships and long-range dependencies. By understanding their architecture and key considerations, you can effectively leverage them for your regression problems.
Claude:
Transformers can be used to predict continuous values. This is often referred to as "regression" in machine learning terminology. Here's a brief overview:
1. Output layer modification: Instead of using a softmax layer for classification, you'd use a linear layer to output a single continuous value.
2. Loss function: You'd typically use mean squared error (MSE) or mean absolute error (MAE) as the loss function, rather than cross-entropy used in classification tasks.
3. Architecture: The core Transformer architecture (self-attention, feed-forward layers, etc.) remains largely the same.
4. Applications: This approach has been used successfully in various domains, including:
5. Variants: Some models like Temporal Fusion Transformers (TFT) are specifically designed for time series regression tasks.
Transformer architecture consisting of encoder & decoder that generate the probability vector (indicated as output probabilities in the daigram on the page below) of a model's vocabulary (the total list of words it knows). The word with the highest probability is chosen as the output and this output is fed into the model at the buttom right in the diagram:
1. BERT (Bidirectional Encoder Representations from Transformers)
A powerful language model that can understand the context of a word based on its surrounding words.
Widely used for various NLP tasks like text classification, question answering, and text generation.
2. GPT-3 (Generative Pre-trained Transformer 3)
A state-of-the-art language model capable of generating human-quality text.
Can be used for tasks like writing different kinds of creative content, translating languages, and answering questions in an informative way.
3. DistilBERT
A smaller, faster version of BERT, trained using knowledge distillation.
Maintains most of BERT's performance while being more efficient.
Ideal for resource-constrained environments or real-time applications.
4. RoBERTa (Robustly Optimized BERT Pretraining Approach)
An improved version of BERT, trained on a larger dataset and with more aggressive hyperparameters.
Often outperforms BERT on various NLP benchmarks.
5. T5 (Text-To-Text Transfer Transformer)
A unified framework for different text-to-text tasks, including translation, summarization, and question answering.
Can be fine-tuned on specific tasks with minimal effort.
6. XLNet
A generalized autoregressive pretraining method that outperforms BERT on many NLP benchmarks.
Captures bidirectional context while avoiding the limitations of masked language modeling.
7. BART (Bidirectional and Auto-Regressive Transformers)
A model designed for both generative and discriminative tasks.
Can be used for tasks like text summarization, question answering, and text generation.
Key Advantages of Transformer Models:
Strong performance:Transformer models consistently achieve state-of-the-art results on a wide range of NLP tasks.
Flexibility: They can be adapted to various tasks with minimal modifications.
Scalability:They can be scaled to handle large datasets and complex tasks.
Interpretability: While still a challenge, techniques are being developed to better understand how transformer models work.
Transformer vs LSTM
While Transformers have largely replaced LSTMs in large-scale applications like natural language processing (NLP), they have not completely replaced them. LSTMs remain highly relevant for specific tasks where their unique architectural traits offer better efficiency or performance. [1, 2, 3, 4]
Where LSTMs Still Win [5]
Small Datasets: LSTMs are often more robust when data is limited. Transformers typically require massive datasets to learn complex relationships and can easily overfit on smaller ones.
Edge & Mobile Devices: LSTMs generally have lower computational and memory requirements than standard Transformers. This makes them better for real-time applications on hardware with limited resources, like smart sensors or mobile apps.
Sequential Prediction: In some specialized time-series tasks, like certain stock price forecasting or localized weather modeling, LSTMs' built-in "sequential bias" can sometimes outperform the global attention of a Transformer.
Very Long Sequences (Memory Efficiency): While Transformers struggle with "quadratic scaling" (memory usage grows rapidly as sequences get longer), LSTMs use constant memory to process sequences step-by-step, making them more memory-efficient for extremely long streams of data. [3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
Where Transformers Have Replaced LSTMs
Large-Scale NLP: For tasks like translation (e.g., Google Translate), summarization, and generative AI (e.g., ChatGPT), Transformers are the undisputed standard due to their ability to capture long-range context.
Parallel Training: Transformers can process entire sequences simultaneously, allowing them to train much faster on modern GPUs than the token-by-token approach of LSTMs. [11, 16, 17, 18]
Comparison Summary
Feature [5, 6, 7, 9, 11, 12, 17, 18, 19]
LSTM
Transformer
Best For
Small data, resource-constrained devices
Large data, complex context
Memory Scaling
Constant (Linear)
Quadratic (Expensive for long sequences)
Processing
Sequential (One word at a time)
Parallel (All words at once)
Long-Term Memory
Limited by vanishing gradients
High (Direct attention to all tokens)
Would you like to see a Python code example comparing how both models handle a simple time-series prediction?
