rh+ddlmZddlZddlmZddlmZddlZddlm cm Z ddlm Z m Z ddl mZddlmZmZmZddZGd d ZGd d ZGd de j,Zy)) annotationsN)Iterable)Any)Tensornn)SentenceTransformer)CachedGISTEmbedLoss"CachedMultipleNegativesRankingLoss+CachedMultipleNegativesSymmetricRankingLossc|jd}||kDrtd|d||dd|f}tj|dd}|S)Nz Dimension zL in matryoshka_dims cannot be greater than the model's embedding dimension: .)pdim)shape ValueErrorF normalize)tensorr tensor_dims ~/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/sentence_transformers/losses/MatryoshkaLoss.pyshrinkrscb!J Zijtiu v  C#I F [[1" -F Mc(eZdZdZddZddZddZy)ForwardDecoratoraS This decorator is used to cache the output of the Sentence Transformer's forward pass, so that it can be shrank and reused for multiple loss calculations. This prevents the model from recalculating the embeddings for each desired Matryoshka dimensionality. This decorator is applied to `SentenceTransformer.forward`. cJ||_d|_g|_d|_d|_yNr)fnrcache cache_dimidx)selfrs r__init__zForwardDecorator.__init__'s% rc ||_d|_yr)rr!)r"rs rset_dimzForwardDecorator.set_dim/src|j|j|jk(r>|j|}|jj ||j|_n|j|j }d|vrt |d|j|d<t |d|j|d<|xj dz c_|S)Ntoken_embeddingssentence_embedding)r rrrappendr!r)r"featuresoutputs r__call__zForwardDecorator.__call__3s >> !T^^txx%?WWX&F JJ  f %!XXDNZZ)F  ')/7I0JDHH)UF% &'-f5I.JDHH'U#$ A  rN)returnNone)r+dict[str, Tensor]r.r0)__name__ __module__ __qualname____doc__r#r%r-rrrrs rrc2eZdZdZ d ddZddZy)CachedLossDecoratora This decorator is used with the Cached... losses to compute the underlying loss function for each Matryoshka dimensionality. This is done by shrinking the pre-computed embeddings to the desired dimensionality and then passing them to the underlying loss function once for each desired dimensionality. This decorator is applied to the `calculate_loss` method of the Cached... losses. c<||_||_||_||_y)N)rmatryoshka_dimsmatryoshka_weightsn_dims_per_step)r"rr9r:r;s rr#zCachedLossDecorator.__init__Ms#."4.rc tt|j}|jdkDr8|jt|kr t j ||j}d}|D]}|j|}|j |}|D cgc]} | D cgc]} t| |c} } } } tj} | r<| D cgc].} | D cgc] } | jj"c} 0} } } n| } |||j| g|i|zz }| st| | D]7\}}t||D]#\}}|j||jz%9|Scc} wcc} } wcc} wcc} } wNrg)rangelenr9r;randomsampler:rtorchis_grad_enableddetachrequires_grad_rzipbackwardgrad)r"repsargskwargs dim_indiceslossr!rweight minibatchr truncatedcompute_gradientsmatryoshka_reps t_minibatch d_minibatchtds rr-zCachedLossDecorator.__call__UsvC 4 456   ! #(<(>F3E8>FN)r )r9 list[int]r:zlist[float | int]r;intr.r/)rIzlist[list[Tensor]]r.r)r1r2r3r4r#r-r5rrr7r7Cs8mo/#,/BS/fi/ /rr7c^eZdZ d dfd ZddZddZed dZxZS) MatryoshkaLossct|||_||_|dgt |z}t ||}t t |dd\|_|_||_ tttf|_ t||jr1t|j |j|j|_yy)a The MatryoshkaLoss can be seen as a loss *modifier* that allows you to use other loss functions at various different embedding dimensions. This is useful for when you want to train a model where users have the option to lower the embedding dimension to improve their embedding comparison speed and costs. This loss is also compatible with the Cached... losses, which are in-batch negative losses that allow for higher batch sizes. The higher batch sizes allow for more negatives, and often result in a stronger model. Args: model: SentenceTransformer model loss: The loss function to be used, e.g. :class:`MultipleNegativesRankingLoss`, :class:`CoSENTLoss`, etc. matryoshka_dims: A list of embedding dimensions to be used for the loss function, e.g. [768, 512, 256, 128, 64]. matryoshka_weights: A list of weights to be used for the loss function, e.g. [1, 1, 1, 1, 1]. If None, then the weights will be set to 1 for all dimensions. n_dims_per_step: The number of dimensions to use per step. If -1, then all dimensions are used. If > 0, then a random sample of n_dims_per_step dimensions are used per step. The default value is -1. References: - The concept was introduced in this paper: https://arxiv.org/abs/2205.13147 - `Matryoshka Embeddings <../../../examples/sentence_transformer/training/matryoshka/README.html>`_ Inputs: +---------------------------------------+--------+ | Texts | Labels | +=======================================+========+ | any | any | +---------------------------------------+--------+ Relations: - :class:`Matryoshka2dLoss` uses this loss in combination with :class:`AdaptiveLayerLoss` which allows for layer reduction for faster inference. Example: :: from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer, losses from datasets import Dataset model = SentenceTransformer("microsoft/mpnet-base") train_dataset = Dataset.from_dict({ "anchor": ["It's nice weather outside today.", "He drove to work."], "positive": ["It's so sunny.", "He took the car to the office."], }) loss = losses.MultipleNegativesRankingLoss(model) loss = losses.MatryoshkaLoss(model, loss, [768, 512, 256, 128, 64]) trainer = SentenceTransformerTrainer( model=model, train_dataset=train_dataset, loss=loss, ) trainer.train() Nr)c |dSrr5)xs rz)MatryoshkaLoss.__init__..s `abc`drT)keyreverse)superr#modelrMr?rFsortedr9r:r;r r r cached_losses isinstancer7calculate_loss)r"rcrMr9r:r; dims_weights __class__s rr#zMatryoshkaLoss.__init__rsF    %"#s?';!; ?,>? 8;VLVdnr=s8t5d5. /  7  dD.. /"5##T%9%94;R;R#D  0rct|j|jr|j||S|jj} t |}||j_t t|j}|jdkDrH|jt|kr0tj||j}|jd}|D]I}|j|}|j|} |j||| |j||zz }K ||j_|S#||j_wxYwr=)rfrMrercforwardrr>r?r9r;r@rAsortr:r%) r"sentence_featureslabelsoriginal_forwarddecorated_forwardrLrMr!rrNs rrkzMatryoshkaLoss.forwards2 dii!3!3 499.7 7 ::-- 2 01A B !2DJJ D$8$8 9:K##a'D,@,@3{CS,S$mmK9M9MN   "D" F**3/005!))#.+rsW" $ 5""J++\KRYYKr