rh$,ddlmZddlZddlmZddlZddlmZddlm Z ddl m Z eje ZGddejZy)) annotationsN)Iterable) FlopsLoss) SparseEncoderc~eZdZ d dfd Z d ddZdZed dZxZS) SpladeLossc t |||_||_||_||_| |_||n t|||_|||_ n| st|||_ |j | stjd|j r(|j tjdd|_t|ds tdy)a SpladeLoss implements the loss function for the SPLADE (Sparse Lexical and Expansion) model, which combines a main loss function with regularization terms to control efficiency. This loss function balances effectiveness (via the main loss) with efficiency by regularizing both the query and document representations to be sparse, reducing computational requirements at inference time. Args: model: SparseEncoder model loss: The principal loss function to use can be any of the SparseEncoder losses except CSR related losses and flops loss. document_regularizer_weight: Weight for the corpus regularization term. This term encourages sparsity in the document embeddings. Will be applied to positive documents and all negatives one if some are provided. In some papers, this parameter is referred to as "lambda_d" (document) or "lambda_c" (corpus). query_regularizer_weight: Weight for the query regularization term. This term encourages sparsity in the query embeddings. If None, no query regularization will be applied, it's not a problem if you are in an inference-free setup or if you are having use_document_regularizer_only=True. Else you should have a query_regularizer_weight > 0. In some papers, this parameter is referred to as "lambda_q" (query). document_regularizer: Optional regularizer to use specifically for corpus regularization instead of the default FlopsLoss. This allows for different regularization strategies for documents vs queries. query_regularizer: Optional regularizer to use specifically for query regularization instead of the default FlopsLoss. This allows for different regularization strategies for queries vs documents. document_regularizer_threshold: Optional threshold for the number of non-zero (active) elements in the corpus embeddings to be considered in the FlopsLoss. If specified, only corpus embeddings with more than this number of non-zero (active) elements will be considered. Only used when document_regularizer is None (for the default FlopsLoss). query_regularizer_threshold: Optional threshold for the number of non-zero (active) elements in the query embeddings to be considered in the FlopsLoss. If specified, only query embeddings with more than this number of non-zero (active) elements will be considered. Only used when query_regularizer is None (for the default FlopsLoss). use_document_regularizer_only: If True, all input embeddings are treated as documents and regularized together with document_regularizer_weight. Especially useful when training with symmetric texts (e.g. pairs of documents) or more. References: - For more details, see the paper "From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective" https://arxiv.org/abs/2205.04733 Requirements: 1. Input requirements depend on the chosen loss 2. Usually used with a teacher model in a knowledge distillation setup and an associated loss Example: :: from datasets import Dataset from sentence_transformers.sparse_encoder import SparseEncoder, SparseEncoderTrainer, losses student_model = SparseEncoder("distilbert/distilbert-base-uncased") teacher_model = SparseEncoder("naver/splade-cocondenser-ensembledistil") train_dataset = Dataset.from_dict( { "query": ["It's nice weather outside today.", "He drove to work."], "passage1": ["It's so sunny.", "He took the car to work."], "passage2": ["It's very sunny.", "She walked to the store."], } ) def compute_labels(batch): emb_queries = teacher_model.encode(batch["query"]) emb_passages1 = teacher_model.encode(batch["passage1"]) emb_passages2 = teacher_model.encode(batch["passage2"]) return { "label": teacher_model.similarity_pairwise(emb_queries, emb_passages1) - teacher_model.similarity_pairwise(emb_queries, emb_passages2) } train_dataset = train_dataset.map(compute_labels, batched=True) loss = losses.SpladeLoss( student_model, loss=losses.SparseMarginMSELoss(student_model), document_regularizer_weight=3e-5, query_regularizer_weight=5e-5, ) trainer = SparseEncoderTrainer(model=student_model, train_dataset=train_dataset, loss=loss) trainer.train() N) thresholdzquery_regularizer_weight is None. This means that the query regularization will not be applied. If you are in an inference free set up it's fine else you should have a query_regularizer_weight > 0.zquery_regularizer_weight should be None when use_document_regularizer_only is True. use_document_regularizer_only mean we consider all the input to be of the same type and so under the same regularization. query_regularizer_weight will be ignored.compute_loss_from_embeddingszThe provided loss does not have a 'compute_loss_from_embeddings' method, which is required for SpladeLoss. This method must have the signature `compute_loss_from_embeddings(embeddings: List[Tensor], labels: Tensor | None = None)`.)super__init__modellossdocument_regularizer_weightquery_regularizer_weightuse_document_regularizer_onlyrdocument_regularizerquery_regularizerloggingwarninghasattr ValueError) selfrrrrrrdocument_regularizer_thresholdquery_regularizer_thresholdr __class__s /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/sentence_transformers/sparse_encoder/losses/SpladeLoss.pyr zSpladeLoss.__init__sp   +F((@%-J* $/ !5,JK !  (%6D ".%.u@[%\D "  ( ( 09V OOX   - -$2O2O2[ OOJ -1D )t;<N =cP|Dcgc]}|j|d}}i}|jj||}t|tr|j |n||d<|j r/|jjtj|}n1|jjtj|dd}||jz|d<|j0|jj|d}||jz|d<|Scc}w)Nsentence_embedding base_lossdocument_regularizer_lossrquery_regularizer_loss) rrr isinstancedictupdaterrtorchcatrrr) rsentence_featureslabelssentence_feature embeddingslossesr! corpus_loss query_losss rforwardzSpladeLoss.forwardsbssM]djj!123GHs sII:::vN i & MM) $"+F;   - -33PPQVQZQZ[eQfgK33PPQVQZQZ[efgfh[iQjkK.9D<\<\.\*+  ( ( 4//LLZXY][J/9DrLs>"$ AL   8 $||r