rhu%ddlmZddlmZddlZddlmZddlmcmZ ddl m Z ddl m Z d dZGddejZGd d ejZy) ) annotations)IterableN)"SparseMultipleNegativesRankingLoss) SparseEncoderc|jd}tj||tj|dddfj|j|z }|S)z :param reconstruction: output of Autoencoder.decode (shape: [batch, n_inputs]) :param original_input: input of Autoencoder.encode (shape: [batch, n_inputs]) :return: normalized mean squared error (shape: [1]) r)dimN)meanFmse_loss broadcast_toshape)reconstructionoriginal_inputoriginal_input_meanlosss /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/sentence_transformers/sparse_encoder/losses/CSRLoss.pynormalized_mean_squared_errorrs^ )--!-4 ::nn 5 D!G$11.2F2FG9 D Kc8eZdZddfd ZddZddZdZxZS) CSRReconstructionLossc>t|||_||_y)a CSRReconstructionLoss implements the reconstruction loss component for Contrastive Sparse Representation (CSR) models. This loss ensures that the sparse encoding can accurately reconstruct the original model embeddings through three components: 1. A primary reconstruction loss (L_k) that measures the error between the original embedding and its reconstruction using the top-k sparse components. 2. A secondary reconstruction loss (L_4k) that measures the error using the top-4k sparse components. 3. An auxiliary loss (L_aux) that helps to learn residual information. Args: model: SparseEncoder model with autoencoder components beta: Weight for the auxiliary loss component (L_aux) References: - For more details, see the paper "Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation" https://arxiv.org/abs/2503.01776 Requirements: 1. The model must be configured to output the necessary reconstruction components 2. Used with SparseEncoder models that implement compositional sparse autoencoding Relations: - Used as a component within :class:`CSRLoss` combined with a contrastive loss Example: - This loss is never used standalone, but instead used within the :class:`CSRLoss` class. See that loss for more details. N)super__init__modelbeta)selfrr __class__s rrzCSRReconstructionLoss.__init__s<   rctd)Nz[CSRReconstructionLoss is not intended to be used standalone. Use it within CSRLoss instead.)NotImplementedError)rsentence_featuress rforwardzCSRReconstructionLoss.forward?s! i  rchd}d}d}|D]s}|d}|d}|d}|d} |d} tj||} tj||} t| || jz } || z }|| z }|| z }ut |}|dkDr||z}||z}||z}||dz |j |zd S) a  Compute the CSRReconstruction loss from embeddings. Args: outputs: List of dictionaries containing sentence embeddings and their sparse representations Returns: total_loss: The total reconstruction loss value gsentence_embedding_backbonedecoded_embedding_kdecoded_embedding_4kdecoded_embedding_auxdecoded_embedding_k_pre_biasrg @)reconstruction_loss_kreconstruction_loss_4kreconstruction_loss_aux)r r rdetachlenr)routputs total_L_k total_L_4k total_L_auxfeaturesxrecons_k recons_4k recons_auxreconsk_pre_biasL_kL_4kL_aux num_columnss rcompute_loss_from_embeddingsz2CSRReconstructionLoss.compute_loss_from_embeddingsDs     !H67A 56H !78I!"9:J'(FG **Q)C::a+D2*aBRBYBYB[>[\E  I $ J 5 K' !,'l ?  $I + %J ; &K&/&03&6'+yy;'>  rcd|jiS) Get the configuration dictionary. Returns: Dictionary containing the configuration parameters r)rrs rget_config_dictz%CSRReconstructionLoss.get_config_dictws ""r)?)rrrfloatreturnNone)r !Iterable[dict[str, torch.Tensor]]rBdict[str, torch.Tensor])r-zlist[dict[str, torch.Tensor]]rBrE)__name__ __module__ __qualname__rr!r;r? __classcell__rs@rrrs D 1 f#rrcPeZdZddfd Z d ddZdZed dZxZS) CSRLossct|||_||_||_t |||_|||_yt||_y)a4 CSRLoss implements a combined loss function for Contrastive Sparse Representation (CSR) models. This loss combines two components: 1. A reconstruction loss :class:`CSRReconstructionLoss` that ensures the sparse representation can faithfully reconstruct the original embedding. 2. A main loss, which in the paper is a :class:`SparseMultipleNegativesRankingLoss` that ensures semantically similar sentences have similar representations. The total loss is linear combination of the two losses. Args: model: SparseEncoder model loss: The principal loss function to use can be any of the SparseEncoder losses except flops loss and CSRReconstruction loss. If None, the default loss is used, which is the SparseMultipleNegativesRankingLoss. beta: Weight for the L_aux component in the reconstruction loss. Default is 0.1. gamma: Weight for the main loss component (MNRL a.k.a. InfoNCE by default). Default is 1.0. References: - For more details, see the paper "Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation" https://arxiv.org/abs/2503.01776 Requirements: 1. Input requirements depend on the chosen loss 2. Uses autoencoder components of the SparseEncoder model Relations: - Uses :class:`CSRReconstructionLoss` for the reconstruction component Example: :: from datasets import Dataset from sentence_transformers.sparse_encoder import SparseEncoder, SparseEncoderTrainer, losses model = SparseEncoder("sentence-transformers/all-MiniLM-L6-v2") train_dataset = Dataset.from_dict( { "anchor": ["It's nice weather outside today.", "He drove to work."], "positive": ["It's so sunny.", "He took the car to the office."], "negative": ["It's quite rainy, sadly.", "She walked to the store."], } ) loss = losses.CSRLoss(model, beta=0.1, gamma=1.0) trainer = SparseEncoderTrainer(model=model, train_dataset=train_dataset, loss=loss) trainer.train() )rrN)r) rrrrgammarreconstruction_lossrr)rrrrrNrs rrzCSRLoss.__init__sPd    $9u4#P  ,D 2T[`2a rc|Dcgc]}|j|}}|Dcgc]}|d }}|jj|}|jj||}t |t r,|j D]\} } | |jz|| <|S||jz|d<|Scc}wcc}w)Nsentence_embedding base_loss)rrOr;r isinstancedictitemsrN) rr labelssentence_featurer-outputrQlossesrRkeyvalues rr!zCSRLoss.forwardsIZZ4D4::./ZZIPQvf%9:QQ))FFwOII::;MvV i &'oo/ 1 U#djj0s  1  #,djj"8F;  [Qs B8 B=cJ|j|j|jdS)r=rrNrr]r>s rr?zCSRLoss.get_config_dicts DJJ JJrcy)Na @misc{wen2025matryoshkarevisitingsparsecoding, title={Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation}, author={Tiansheng Wen and Yifei Wang and Zequn Zeng and Zhong Peng and Yudi Su and Xinyang Liu and Bo Chen and Hongwei Liu and Stefanie Jegelka and Chenyu You}, year={2025}, eprint={2503.01776}, archivePrefix={arXiv}, primaryClass={cs.LG}, url={https://arxiv.org/abs/2503.01776}, } r>s rcitationzCSRLoss.citations r)Ng?r@)rrrznn.Module | NonerrArNrA)N)r rDrVztorch.Tensor | NonerBrE)rBstr) rFrGrHrr!r?propertyr`rIrJs@rrLrLsH9bxcg!BL_ $K  rrL)r torch.TensorrrcrBrc) __future__rcollections.abcrtorchtorch.nnnntorch.nn.functional functionalr Nsentence_transformers.sparse_encoder.losses.SparseMultipleNegativesRankingLossr2sentence_transformers.sparse_encoder.SparseEncoderrrModulerrLr_rrrnsL"$ M b#BIIb#Jcbiicr