---
title: "Manual 03: Embeddings & Vector Space Geometry — RAG Canon"
description: "Technical implementation manual for Embeddings & Vector Space Geometry in the RAG Canon."
image: "https://tinycto.tv/assets/rag-canon/rag_manuals_og.jpg"
canonicalUrl: "https://tinycto.tv/rag-canon/manuals/03-embeddings-vector-spaces"
locale: "en"
---

# Engineering Manual 03: Embeddings & Vector Space Geometry

## 1. Vector Space Representations
Dense vector embeddings map unstructured text passages into continuous high-dimensional vector spaces (e.g. 768 to 3072 dimensions) where semantic proximity corresponds to cosine similarity or inner product distance.

### Modern Embedding Paradigms
1. **Dense Bi-Encoders:**
   - Single pooled vector per passage (e.g. BGE-large, text-embedding-3-large).
   - High search speed via approximate nearest neighbor (ANN) indexing.
2. **Sparse Lexical Encoders (SPLADE):**
   - Projects passages into vocabulary-dimensional space, predicting expanded keyword weights.
   - Combines lexical precision with semantic expansion.
3. **Late-Interaction Token Encoders (ColBERT):**
   - Preserves individual token vectors and computes late similarity via the MaxSim operator.
   - Retains fine-grained token match sensitivity without cross-encoder computational overhead.

---

## 2. Quantization & Dimensionality Optimization
- **Matryoshka Representation Learning (MRL):** Models trained to allow vector truncation from 3072 to 512 dimensions with minimal degradation in recall (<1.5%).
- **Scalar & Product Quantization (SQ/PQ):** Compresses 32-bit floating-point coordinates into 8-bit integers or codebook centroid indices, reducing RAM footprint by 4x to 8x.