rh6ddlmZddlmZmZddlmZddlmZddl m Z ddl Z ddl Z ddl m Z mZddlmZmZdd lmZdd lmZdd lmZ dd ZGdd ej2Zy)) annotations)IterableIterator) nullcontext)partial)AnyN)Tensornn)SentenceTransformerutil) RandContext)StaticEmbedding)all_gather_with_grad+CachedMultipleNegativesSymmetricRankingLossc |jJ|jJtj5t ||j|jD]u\}}}t |j |dd||D]M\\}}}tj |j|j|z} | jOw dddy#1swYyxYw)zOA backward hook to backpropagate the cached gradients mini-batch by mini-batch.NTF)sentence_feature with_gradcopy_random_state random_states) cachertorch enable_gradzipembed_minibatch_iterdotflattenbackward) grad_outputsentence_featuresloss_objrgradrreps_mb_grad_mb surrogates /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/sentence_transformers/losses/CachedMultipleNegativesSymmetricRankingLoss.py_backward_hookr's >> %% %  ! ! -- -     %589JHNN\d\r\r5s % 1 dM),--%5"&+"/ . * %% !g"IIgoo&79JKkY ""$ % % % % %s BCCceZdZdejdddf d fd Z d d dZ d ddZddZdddZ dd Z dd Z xZ S)rg4@ Fct|t|dtr t d||_||_||_||_||_ ||_ tj|_ d|_d|_y)a Boosted version of :class:`MultipleNegativesSymmetricRankingLoss` (MNSRL) by GradCache (https://arxiv.org/pdf/2101.06983.pdf). Given a list of (anchor, positive) pairs, MNSRL sums the following two losses: 1. Forward loss: Given an anchor, find the sample with the highest similarity out of all positives in the batch. 2. Backward loss: Given a positive, find the sample with the highest similarity out of all anchors in the batch. For example with question-answer pairs, the forward loss finds the answer for a given question and the backward loss finds the question for a given answer. This loss is common in symmetric tasks, such as semantic textual similarity. The caching modification allows for large batch sizes (which give a better training signal) with constant memory usage, allowing you to reach optimal training signal with regular hardware. Note: If you pass triplets, the negative entry will be ignored. An anchor is just searched for the positive. Args: model: SentenceTransformer model scale: Output of similarity function is multiplied by scale value. In some literature, the scaling parameter is referred to as temperature, which is the inverse of the scale. In short: scale = 1 / temperature, so scale=20.0 is equivalent to temperature=0.05. similarity_fct: similarity function between sentence embeddings. By default, cos_sim. Can also be set to dot product (and then set scale to 1) mini_batch_size: Mini-batch size for the forward pass, this denotes how much memory is actually used during training and evaluation. The larger the mini-batch size, the more memory efficient the training is, but the slower the training will be. It's recommended to set it as high as your GPU memory allows. The default value is 32. gather_across_devices: If True, gather the embeddings across all devices before computing the loss. Recommended when training on multiple GPUs, as it allows for larger batch sizes, but it may slow down training due to communication overhead, and can potentially lead to out-of-memory errors. show_progress_bar: If True, a progress bar for the mini-batches is shown during training. The default is False. Requirements: 1. (anchor, positive) pairs 2. Should be used with large batch sizes for superior performance, but has slower training time than non-cached versions Inputs: +---------------------------------------+--------+ | Texts | Labels | +=======================================+========+ | (anchor, positive) pairs | none | +---------------------------------------+--------+ Recommendations: - Use ``BatchSamplers.NO_DUPLICATES`` (:class:`docs `) to ensure that no in-batch negatives are duplicates of the anchor or positive samples. Relations: - Like :class:`MultipleNegativesRankingLoss`, but with an additional symmetric loss term and caching mechanism. - Inspired by :class:`CachedMultipleNegativesRankingLoss`, adapted for symmetric loss calculation. Example: :: from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer, losses from datasets import Dataset model = SentenceTransformer("microsoft/mpnet-base") train_dataset = Dataset.from_dict({ "anchor": ["It's nice weather outside today.", "He drove to work."], "positive": ["It's so sunny.", "He took the car to the office."], }) loss = losses.CachedMultipleNegativesSymmetricRankingLoss(model, mini_batch_size=32) trainer = SentenceTransformerTrainer( model=model, train_dataset=train_dataset, loss=loss, ) trainer.train() References: - Efficient Natural Language Response Suggestion for Smart Reply, Section 4.4: https://arxiv.org/pdf/1705.00652.pdf - Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup: https://arxiv.org/pdf/2101.06983.pdf rzCachedMultipleNegativesSymmetricRankingLoss is not compatible with a SentenceTransformer model based on a StaticEmbedding. Consider using MultipleNegativesSymmetricRankingLoss instead.N)super__init__ isinstancer ValueErrormodelscalesimilarity_fctmini_batch_sizegather_across_devicesshow_progress_barr CrossEntropyLosscross_entropy_lossrr)selfr/r0r1r2r3r4 __class__s r&r,z4CachedMultipleNegativesSymmetricRankingLoss.__init__*sh  eAh 0P    ,.%:"!2"$"5"5"704 =Acv|rtntj}| tn|}|jD cic] \} } | | || } } } |5|5|rt | j nd}|j | d} dddddd |fScc} } w#1swYxYw#1swY |fSxYw)z Embed a mini-batch of sentences.Nsentence_embedding)rrno_graditemsr valuesr/) r7rbeginendrr random_state grad_contextrandom_state_contextkvsentence_feature_minibatchrepss r&embed_minibatchz;CachedMultipleNegativesSymmetricRankingLoss.embed_minibatchs'0{U]] 0<0D{},BRBXBXBZ%[$!Qa5o%["%[ ! T TTe{,F,M,M,OPko zz"<=>RS T T\!! &\ T T T\!!s)BB,/B B, B) %B,,B8c # K|d}|j\}}ttjd||jd|j D];\}} | |jz} |j || | |||dn||\} } | | f=yw)z5Iterate over mini-batches of sentences for embedding. input_idsrzEmbed mini-batchesdescdisableN)rr?r@rrrA)shape enumeratetqdmtranger2r4rH) r7rrrrrJ batch_sizer#ir?r@rGrAs r&rz@CachedMultipleNegativesSymmetricRankingLoss.embed_minibatch_iters-[9 ! A! KK$$) 222    %HAu$...C!%!5!5!1#"3%2%:T a@P "6" D, $ $% %sB Bc|j|d}|jj}|Dcgc]}|Dcgc]}|jc}c}}|_|Scc}wcc}}w)z1Calculate the symmetric loss and cache gradients.T) with_backward)calculate_lossdetachrequires_grad_r!r)r7rGlossrsrs r&"calculate_loss_and_cache_gradientszNCachedMultipleNegativesSymmetricRankingLoss.calculate_loss_and_cache_gradientssY""4t"<{{}++-59:rr*!qvv*:  +:s A(A# A(#A(cFtj|d}|ddDcgc]}tj|}}t|}d}|jrdt |}|Dcgc] }t |}}tj j r#tj j} | |z}tj|d}tj|||z|j} g} tjd||jd|j D]} t| |jz|} |j||| z|| z||j z}|j#|| | | }|j||| z|| z||j z}|j#|| | | }||zdz }|t|z|z }|r |j%|j'}| j)|t+| }|Scc}wcc}w) zHCalculate the symmetric loss without caching gradients (for evaluation).rN)dim)devicezCalculating lossrK)rcatlenr3r distributedis_initializedget_rankaranger`rPrQr2r4minr1r0r6rrWappendsum)r7rGrUanchorsr[ candidatesrRoffsetembedding_columnranklabelslossesr?r@forward_scores forward_lossbackward_scores backward_loss loss_mbatchrYs r&rVz:CachedMultipleNegativesSymmetricRankingLoss.calculate_losss'))DG$,0H5qeiil5 5\   % %+73GYcdEU./?@dJd  //1((113 *YYzq1 ffz&9'..Q%'[[   #...   'Eed222J?C##GFUNVc\$JJWZ^ZdZdd *.)@)@QWX]^aQb)cL"11*Ve^fWZl2[]dehlhrhrrO*.*A*A/SYZ_`cSd*eM'-71rsd"." ;W8;%%2%:% %.w "))w r9