Data science tools

 

Handwriting notetaking tool for all OSs

 https://xournalpp.github.io/

Classical optimization algorithms in quantum computing

Standard classical code doesn't execute raw on quantum chips, but classical optimization theory forms the indispensable foundation of quantum-era optimization.

1. The Mechanics: Adaptation vs. Foundation

• Why code isn't directly transferable: Digital optimization code relies on classical primitives—if/else logic based on mid-computation values, memory updates, and variable copying. Quantum gates must be unitary (deterministic and reversible) and linear. You cannot simply feed standard C++ or Python optimization code into a quantum processing unit (QPU).
• Why the theory carries over: The underlying mathematics—objective function modeling, convex vs. non-convex geometry, cost landscapes, and loss minimization—remains identical. Quantum algorithms reframe how state space is explored, but what constitutes a optimal point is anchored in classical decision theory.

2. The Evolutionary Spectrum

To visualize how classical optimization bridges into the quantum era:

Level	Role of Classical Optimization	Example Frameworks
Direct Controller	Classical algorithms wrap around the QPU, updating circuit angles using measurements from quantum executions.	SPSA, COBYLA, Adam in VQE/QAOA
Theoretical Blueprint	Classical metaheuristics modified using quantum mechanics (wavefunctions, quantum tunneling).	QPSO (Quantum-Behaved PSO), Quantum Simulated Annealing
Quantum-Native Logic	Strictly quantum linear algebra algorithms designed to solve optimization models natively on QPUs.	HHL algorithm (linear systems), Quantum Interior-Point Methods

3. The Practical Reality of the Quantum Era

Quantum optimization won't make classical optimization obsolete; it elevates it:

• Hybrid is the permanent paradigm: Even in the fault-tolerant quantum era, hybrid classical-quantum loops will remain standard because measuring and evaluating intermediate quantum states is computationally expensive compared to running classical matrix operations.
• Classical algorithms handle the heavy lifting: NP-hard combinatorial optimization problems (e.g., portfolio selection, logistics routing, molecular docking) are formulated classically first before mapped onto quantum Hamiltonians.

In short: classical optimization theory is the blueprint and runtime orchestrator; quantum mechanics provides the enhanced hardware speedups for exploring intractable search spaces.

Out-of-Fold (OOF)

Out-of-Fold (OOF) is a validation and model-building technique used heavily in machine learning (especially in competitions like Kaggle) to prevent data leakage and generate unbiased predictions on your training data.

To understand it, you first need to recall how K-Fold Cross-Validation works:

1. You split your training data into $K$ parts (e.g., $K = 5$ folds).
2. You train your model $5$ times. In each iteration, you use $4$ folds for training and leave $1$ fold out for validation.
   
   

How Out-of-Fold Predictions Work

An Out-of-Fold prediction is a prediction made on a validation fold by a model that was not trained on that fold.

If you do this for all $5$ folds:

• Fold 1 is validated by a model trained on Folds 2, 3, 4, 5.
• Fold 2 is validated by a model trained on Folds 1, 3, 4, 5.
• (and so on...)
  
  

Once all $K$ iterations are finished, every single row in your original training dataset has been predicted exactly once by a model that had never seen it during training.

When you stitch all these predictions back together in the correct order, you get a full column of Out-of-Fold predictions spanning your entire training dataset.

Why is OOF Prediction Important?

1. Unbiased Evaluation of Model Performance

If you evaluate your model on data it was trained on, you get overoptimistic metrics due to overfitting. OOF metrics give you a realistic, honest estimate of how your model will perform on completely unseen test data.

2. Creating Meta-Features for Stacking (Blending)

This is the most powerful use case for OOF predictions.

• If you want to build a stacked ensemble (where Model C takes the predictions of Model A and Model B as its inputs), you cannot just feed Model C the predictions your models made on the training data. If you do, Model C will overfit because Models A and B already "memorized" those training rows.
• Instead, you feed Model C the OOF predictions of Model A and Model B. Because those OOF predictions were generated when each row was "out-of-fold," they mimic how the models behave on truly unseen data.
  
  

3. Threshold Tuning and Post-Processing

You can use your OOF predictions to tune decision thresholds (e.g., finding the optimal probability cutoff for classification) without risking data leakage, because those predictions were never biased by the training labels of those specific rows.

Stratified K-Fold Cross-Validation

คือเทคนิคการแบ่งชุดข้อมูล (Data Splitting) สำหรับทำ Cross-Validation โดยคำนึงถึง สัดส่วนของคลาสเป้าหมาย (Class Distribution) เป็นสำคัญ เพื่อให้ทุก Fold มีสัดส่วนของแต่ละคลาสใกล้เคียงกับชุดข้อมูลจริงทั้งหมด

ใน K-Fold ทั่วไป ข้อมูลจะถูกสุ่มแบ่งเป็น K ส่วนเท่าๆ กันแบบสุ่ม (Random) ซึ่งอาจทำให้บาง Fold มีสัดส่วนของคลาสใดคลาสหนึ่งมากหรือน้อยเกินไป (หรือในกรณีข้อมูลไม่สมดุล - Imbalanced Data บาง Fold อาจไม่มีคลาสที่เป็นกลุ่มน้อยเลย)

Stratified K-Fold จะแก้ไขปัญหานี้โดยบังคับให้แต่ละ Fold มีสัดส่วนของแต่ละคลาส (เช่น คลาส 0 และคลาส 1) เท่ากับหรือใกล้เคียงกับสัดส่วนใน Dataset ทั้งหมด มากที่สุด

HPC vs Parallel computing vs Cluster computing

Parallel Computing (The Method): A strategy where a single, complex task is broken down into smaller sub-tasks executed simultaneously across multiple processors or cores to save time.

Cluster Computing (The Infrastructure): An architecture that connects multiple independent computers (nodes) over a network so they function together as a single, unified system.

High-Performance Computing / HPC (The Capability & Discipline): The umbrella field that leverages advanced parallel computing techniques, massive compute clusters, high-speed networks, and specialized accelerators (like GPUs) to solve extreme, data-intensive problems.

C2PA

C2PA (Coalition for Content Provenance and Authenticity) provides an open, cryptographically secure standard to trace the origin (provenance) and edit history of digital media like images, videos, and audio.
Instead of trying to detect deepfakes or altered pixels after the fact, C2PA embeds verifiable metadata directly into the media file, establishing a clear line of custody.
Key Technical Architecture
C2PA relies on Public Key Infrastructure (PKI) and cryptographic hashing to bind history to media across four main layers:
 Assertions: Standardized data blocks containing specific claims—such as camera model, capture timestamp, edit actions (e.g., cropping, color adjustment), or AI generation tools used (e.g., DALL-E, Adobe Firefly).
 Claims: A summary structure that aggregates all assertions made during a specific edit session, along with a cryptographic hash of the raw media asset payload (the "Content Binding").
 Claim Signature: A digital signature produced by an authorized signer (a camera manufacturer, software application, or platform) using their private key and validated against a Public Key Certificate.
 Manifest & Manifest Store: The complete bundle of assertions, claims, and signatures embedded inside the file container (JPEG, MP4, WAV, etc.) or hosted externally via a sidecar link.
Step-by-Step Provenance Workflow
1. Creation: A C2PA-compliant device (like a compatible Leica camera) captures a photo. It creates the initial ingredient manifest, hashes the image bytes, attaches hardware metadata, and signs it using a certificate baked into the device.
2. Editing: When opened in an editing tool (like Photoshop), the application reads the previous manifest, applies changes, logs the edit operations, hashes the newly edited file, and appends a new signed manifest pointing back to the parent manifest.
3. Verification: A user or platform runs a validator (such as Content Credentials or an inspection API). The validator re-computes the image hash to check if the file was modified without a signature update, checks the signature against trusted Certificate Authorities (CAs), and displays the full audit trail.

This standard will be used to implement watermark for AI generated content as directed by EU law: The legislation is the EU Artificial Intelligence Act (Regulation (EU) 2024/1689).  

Specifically, Article 50 of the Act outlines strict transparency obligations regarding AI-generated synthetic content.  

Key Requirements Under Article 50

 Technical Watermarking (Article 50(2)): Providers of generative AI systems (producing images, audio, video, or text) must ensure that outputs are marked in a machine-readable format and detectable as artificially generated or manipulated. Companies fulfill this using techniques like C2PA metadata provenance or invisible embedded watermarking (such as Google's SynthID).  

 Deepfake Disclosure (Article 50(4)): Deployers who publish manipulated image, audio, or video content resembling real people, places, or events ("deepfakes") must explicitly label it to prevent deception.  

 Public Interest Text Disclosure: AI-generated text published to inform the public on matters of public interest must carry a clear disclosure tag unless it has undergone human editorial review.  

Scope & Technical Feasibility

The law includes a "technological proportionality" clause—acknowledging that watermarking must be effective and robust as far as technically feasible. For example, invisible watermarks on very short text snippets (under ~200 tokens) are largely exempt because current technology cannot reliably encode robust signals into very brief text without degrading quality.

Firewall against DDoS

 What Firewalls Can Block

• Protocol-Based Attacks: Filters out exploited protocol vulnerabilities like ICMP floods (ping floods) or spoofed packet floods using IP/port rules.

• Basic Connection Limits: Standard firewalls can rate-limit incoming TCP connections from a single IP to prevent connection exhaustion.

• Stateful Inspection: Next-Generation Firewalls (NGFW) can drop incomplete or suspicious connection attempts (such as SYN floods) before they hit internal servers.

Where Firewalls Fail Against DoS/DDoS

• Volumetric Attacks: If an attacker sends 100 Gbps of traffic and your network pipe is 10 Gbps, the link saturates upstream before traffic even hits the firewall.

• Resource Exhaustion: Stateful firewalls track active connections in a state table. Flooding millions of request IPs exhausts the firewall's own CPU/RAM, causing the firewall itself to crash or drop legitimate traffic.

• Application-Layer (Layer 7) Attacks: HTTP GET/POST floods mimic legitimate user behavior. Standard firewalls cannot easily distinguish valid web traffic from malicious requests.

Effective Defense Strategy For complete protection, firewalls must be combined with dedicated DDoS mitigation systems:

• Cloud-based Anycast Scrubbing: Services (like Cloudflare, AWS Shield, or Akamai) absorb massive volumetric traffic upstream before it reaches your network interface.

• Web Application Firewalls (WAF): Specifically inspect Layer 7 traffic to filter out malicious HTTP requests, botnets, and rate-limit specific endpoints.

• ISP-Level Rate Limiting: Routers at the ISP level drop BGP blackhole or rate-limited traffic during high-volume spikes.

• CDN: Absorbs massive volumetric traffic spikes using Anycast networks and integrates Edge Web Application Firewalls (WAF).