rhrddlmZddlZddlmZddlZddlmZeje Z GddeZ y)) annotationsN)Literal)ModuleceZdZUdZdZddgZgdZded< d dfd Z dd Z d d dd Z dd Z ddZ xZ S) SpladePoolingaL SPLADE Pooling module for creating the sparse embeddings. This module implements the SPLADE pooling mechanism that: 1. Takes token logits from a masked language model (MLM). 2. Applies a sparse transformation using an activation function followed by log1p (i.e., log(1 + activation(MLM_logits))). 3. Applies a pooling strategy `max` or `sum` to produce sparse embeddings. The resulting embeddings are highly sparse and capture lexical information, making them suitable for efficient information retrieval. Args: pooling_strategy (str): Pooling method across token dimensions. Choices: - `sum`: Sum pooling (used in original SPLADE see https://arxiv.org/pdf/2107.05720). - `max`: Max pooling (used in SPLADEv2 and later models see https://arxiv.org/pdf/2109.10086 or https://arxiv.org/pdf/2205.04733). activation_function (str): Activation function applied before log1p transformation. Choices: - `relu`: ReLU activation (standard in all Splade models). - `log1p_relu`: log(1 + ReLU(x)) variant used in Opensearch Splade models, see https://arxiv.org/pdf/2504.14839. word_embedding_dimension (int, optional): Dimensionality of the output embeddings (if needed). chunk_size (int, optional): Chunk size along the sequence length dimension (i.e., number of tokens per chunk). If None, processes entire sequence at once. Using smaller chunks the reduces memory usage but may lower the training and inference speed. Default is None. )summaxrelu log1p_relu)pooling_strategyactivation_functionword_embedding_dimensionz list[str] config_keysct|||_||jvr t d||_||j vr t d||_||_y)Nz.pooling_strategy must be either 'max' or 'sum'z9activation_function must be either 'relu' or 'log1p_relu') super__init__r SPLADE_POOLING_MODES ValueErrorr SPLADE_ACTIVATIONr chunk_size)selfr r rr __class__s /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/sentence_transformers/sparse_encoder/models/SpladePooling.pyrzSpladePooling.__init__-sc  0 4#<#< <MN N#6 d&<&< <XY Y(@%$c|d}|d}|jdj|j}|j\}}}|j}|j dk(r/t j||ftd|j|} nL|j dk(r%t j||f|j|} ntd|j |j|jd kr|n |j} td || D]} |d d | | | zd d f} |d d | | | zd d f} | | z}|j}|js|j}n|j!}|j"d k(r-|js|j}n|j!}|j dk(r1t j$|d d }t j&| |} nt j(|d }| |z } |j4| jd |_| |d<|S#t*$r7}dt-|j/vrt0j3d|d }~wwxYw)a} Forward pass of the model. Args: features: Dictionary containing input features. Expects: - 'token_embeddings': MLM logits (shape: batch_size, seq_length, vocab_size). - 'attention_mask': Attention mask (shape: batch_size, seq_length). Returns: Dictionary containing SPLADE pooled embeddings token_embeddingsattention_maskr z-inf)dtypedevicerzUnsupported pooling_strategy: Nrr )dimz out of memoryzRan out of memory during SpladePooling. Consider setting or decreasing the 'chunk_size' parameter. Smaller chunk_size reduces memory usage at the cost of slower processing, but will allow for larger batch sizes.sentence_embedding) unsqueezetorshaper r torchfullfloatzerosrrrangerelu_traininglog1p_log1pr r maximumr RuntimeErrorstrlowerloggerwarningr)rfeatures mlm_logitsrattention_mask_expanded batch_sizeseq_lenvocab_sr pooled_scoresricurrent_chunk_logitscurrent_chunk_maskmasked_current_chunk_logitscurrent_chunk_transformed chunk_pooledes rforwardzSpladePooling.forward>s01 !"23#1":":2">"A"A*BRBR"S'1'7'7$ GW""  E )!JJ G'S>S=TUV V"&!8DOOqres4" 6   8 $F-FF-r