Next-Generation RAG Powered by Contextual Embeddings
Perplexity Research, in collaboration with turbopuffer, has announced the release of pplx-embed-v2-context-9b-preview, a new contextual embedding model. This development is set to enhance Retrieval Augmented Generation (RAG) workflows by enabling more sophisticated document retrieval. Unlike previous methods, this model embeds document chunks while considering the entirety of the source material, ensuring a richer contextual understanding. Its core innovation lies in a training methodology that focuses on retrieving both the precise answer and the corroborating information needed to validate it, rather than isolating a single 'best' passage.
The model is currently available as a self-hosted preview, with its weights accessible on Hugging Face under an MIT license. Deployment necessitates transformers>=5.4.0 with trust_remote_code=True. It is important to note that this preview is not yet integrated into the Perplexity API, and its weights and interface are subject to modifications that may not maintain backward compatibility.
Overcoming Limitations of Traditional RAG
Conventional RAG systems typically segment lengthy documents into smaller chunks. A common drawback is that these isolated chunks frequently lack essential contextual information, such as definitions or related entities, which may be located elsewhere in the document. While some contextual models attempt to mitigate this by processing documents in a single pass before pooling per chunk, a significant challenge persists in the training phase.
Often, training assigns a single 'gold chunk' as the correct answer for a given query, classifying all other chunks as negative examples. This includes passages that might contain valuable information for verifying the answer. Perplexity highlights additional issues with this binary labeling approach, including a coarse training signal, the escalating cost of LLM annotation with larger datasets, and the rigid dependency on a specific chunking strategy.
Innovative Training Methodology
The pplx-embed-v2-context-9b-preview model employs a sophisticated distillation process, utilizing Perplexity's query-aware context compression model as its 'teacher.' This teacher model analyzes the query alongside the full document, assigning a relevance score to every token. These token scores are then used to calculate:
- Chunk Relevance: Determined by averaging the top 'n' token scores within each chunk.
- Soft Target: A temperature-scaled softmax applied to chunks within the relevant document, with chunks from unrelated documents receiving zero scores.
- Distillation Loss: Measured using forward KL divergence between the teacher and student model distributions.
- Document Loss: Employing InfoNCE, where a document's score is derived from its highest-scoring chunk, inspired by ColBERT's MaxSim.
During training, the model samples diverse chunking strategies for each batch. Chunks are delineated by a learned <|chunk_sep|> token and subsequently mean-pooled. Crucially, the teacher model operates exclusively during the training phase, ensuring no additional latency or storage overhead during inference.
Technical Specifications and Performance
The model is initialized from an in-house 9 billion-parameter ColBERT retrieval model. It features a linear projection layer that outputs 2048-dimensional embeddings. Matryoshka training further supports 1024-dimensional outputs, and quantization-aware training enables native int8 embeddings for efficiency. This release is a 'model soup,' indicating it's an ensemble of several optimized checkpoints. The training regimen involved approximately 430 datasets, encompassing over 50 languages, without leveraging ConTEB data.
This article is a rewritten summary based on publicly available reporting. For the original story, visit the source.
Source: MarkTechPost
