rhkYddlmZddlmZmZddlmZddlmZddl m Z m Z ddl Z ddl Z ddl mZmZddlmZmZdd lmZdd lmZdd lmZdd lmZGd dZ ddZGddej:Zy)) annotations)IterableIterator) nullcontext)partial)AnyLiteralN)Tensornn)get_device_statesset_device_states)PreTrainedTokenizerBase)SentenceTransformer)StaticEmbedding)all_gather_with_gradc(eZdZdZddZddZddZy) RandContexta Random-state context manager class. Reference: https://github.com/luyug/GradCache. This class will back up the pytorch's random state during initialization. Then when the context is activated, the class will set up the random state with the backed-up one. c`tj|_t|\|_|_yN)torch get_rng_state fwd_cpu_stater fwd_gpu_devicesfwd_gpu_states)selftensorss /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/sentence_transformers/losses/CachedGISTEmbedLoss.py__init__zRandContext.__init__s("0024Ew4O1d1ctjj|jd|_|jj tj |jt|j|jy)NT)devicesenabled) rrandomfork_rngr_fork __enter__ set_rng_staterr rrs rr&zRandContext.__enter__s^\\**43G3GQU*V   D../$..0C0CDrcL|jj|||d|_yr)r%__exit__)rexc_typeexc_valexc_tbs rr*zRandContext.__exit__%s Hgv6 rN)returnNone)__name__ __module__ __qualname____doc__rr&r*rrrrsPE rrCachedGISTEmbedLossc |jJ|jJtj5t ||j|jD]v\}}}t |j |dd||D]N\\}}}}tj |j|j|z} | jPx dddy#1swYyxYw)zOA backward hook to backpropagate the cached gradients mini-batch by mini-batch.NTF)sentence_feature with_gradcopy_random_state random_states) cacher:r enable_gradzipembed_minibatch_iterdotflattenbackward) grad_outputsentence_featuresloss_objr7gradr:reps_mb_grad_mb surrogates r_backward_hookrJ*s >> %% %  ! ! -- -     %589JHNN\d\r\r5s % 1 dM,/--%5"&+"/ . - %(!Q"IIgoo&79JKkY ""$ % % % % %s BCCceZdZ d d fd Zd dZ d d dZ d ddZddZ d ddZddZ ddZ xZ S)r5c $t |t|dtr t d||_||_||_tjd|_ t|dr t|ds t dt|jtrt|jts t d||_d|_d|_||_|jj$|jj$k7xs|j&|j&k|_|j(r|j|_ |d vr t d ||_||_||_| |_| |_tj4|_y) a1 This loss is a combination of :class:`GISTEmbedLoss` and :class:`CachedMultipleNegativesRankingLoss`. Typically, :class:`MultipleNegativesRankingLoss` requires a larger batch size for better performance. :class:`GISTEmbedLoss` yields stronger training signals than :class:`MultipleNegativesRankingLoss` due to the use of a guide model for in-batch negative sample selection. Meanwhile, :class:`CachedMultipleNegativesRankingLoss` allows for scaling of the batch size by dividing the computation into two stages of embedding and loss calculation, which both can be scaled by mini-batches (https://arxiv.org/pdf/2101.06983.pdf). By combining the guided selection from :class:`GISTEmbedLoss` and Gradient Cache from :class:`CachedMultipleNegativesRankingLoss`, it is possible to reduce memory usage while maintaining performance levels comparable to those of :class:`GISTEmbedLoss`. You can apply different false-negative filtering strategies to discard hard negatives that are too similar to the positive. Two strategies are supported: - "absolute": Discards negatives whose similarity score is greater than or equal to ``positive_score - margin``. - "relative": Discards negatives whose similarity score is greater than or equal to ``positive_score * (1 - margin)``. Args: model: SentenceTransformer model guide: SentenceTransformer model to guide the in-batch negative sample selection. temperature: Temperature parameter to scale the cosine similarities. mini_batch_size: Mini-batch size for the forward pass, this denotes how much memory is actually used during training and evaluation. The larger the mini-batch size, the more memory efficient the training is, but the slower the training will be. It's recommended to set it as high as your GPU memory allows. The default value is 32. show_progress_bar: If True, a progress bar for the mini-batches is shown during training. The default is False. margin_strategy: Strategy used for false negative filtering. One of {"absolute", "relative"}. margin: The margin value for filtering negatives. Defaults to 0.0, together with the "absolute" strategy, this only removes negatives that are more similar to the query than the positive is to the query. contrast_anchors: If True, include anchor-anchor pairs in the loss computation, resulting in the embeddings of the anchors being pushed further apart. Defaults to True, following the original GISTEmbed paper. contrast_positives: If True, include positive-positive pairs in the loss computation, resulting in the embeddings of the positives being pushed further apart. Defaults to True, following the original GISTEmbed paper, but setting to False may yield better results in some retrieval tasks. gather_across_devices: If True, gather the embeddings across all devices before computing the loss. Recommended when training on multiple GPUs, as it allows for larger batch sizes, but it may slow down training due to communication overhead, and can potentially lead to out-of-memory errors. References: - Efficient Natural Language Response Suggestion for Smart Reply, Section 4.4: https://arxiv.org/pdf/1705.00652.pdf - Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup: https://arxiv.org/pdf/2101.06983.pdf - GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuning https://arxiv.org/abs/2402.16829 Requirements: 1. (anchor, positive) pairs or (anchor, positive, negative pairs) 2. Should be used with large batch sizes for superior performance, but has slower training time than :class:`MultipleNegativesRankingLoss` Inputs: +-------------------------------------------------+--------+ | Texts | Labels | +=================================================+========+ | (anchor, positive) pairs | none | +-------------------------------------------------+--------+ | (anchor, positive, negative) triplets | none | +-------------------------------------------------+--------+ | (anchor, positive, negative_1, ..., negative_n) | none | +-------------------------------------------------+--------+ Recommendations: - Use ``BatchSamplers.NO_DUPLICATES`` (:class:`docs `) to ensure that no in-batch negatives are duplicates of the anchor or positive samples. Relations: - Equivalent to :class:`GISTEmbedLoss`, but with caching that allows for much higher batch sizes Example: :: from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer, losses from datasets import Dataset model = SentenceTransformer("microsoft/mpnet-base") guide = SentenceTransformer("all-MiniLM-L6-v2") train_dataset = Dataset.from_dict({ "anchor": ["It's nice weather outside today.", "He drove to work."], "positive": ["It's so sunny.", "He took the car to the office."], }) loss = losses.CachedGISTEmbedLoss( model, guide, mini_batch_size=64, margin_strategy="absolute", # or "relative" (e.g., margin=0.05 for max. 95% of positive similarity) margin=0.1 ) trainer = SentenceTransformerTrainer( model=model, train_dataset=train_dataset, loss=loss, ) trainer.train() rzCachedGISTEmbedLoss is not compatible with a SentenceTransformer model based on a StaticEmbedding. Consider using GISTEmbedLoss instead.dim tokenizerzNBoth the training model and the guiding model must have a tokenizer attribute.z_Both the training model and the guiding model must use a PreTrainedTokenizer from transformers.N)absoluterelativez1margin_strategy must be 'absolute' or 'relative'.)superr isinstancer ValueErrormodelguide temperaturer CosineSimilaritysimilarity_fcthasattrrPrmini_batch_sizer;r:show_progress_barvocabmax_seq_lengthmust_retokenizemargin_strategymargincontrast_anchorscontrast_positivesgather_across_devicesCrossEntropyLosscross_entropy_loss) rrVrWrXr\r]rarbrcrdre __class__s rrzCachedGISTEmbedLoss.__init__BsmT  eAh 08   & 11b9uk*'%2Mmn n%//+BC: OO4L q  /04 =A!2 OO ! !U__%:%: : ie>R>RUZUiUi>i    "__DN ": :PQ Q. 0"4%:""$"5"5"7rcb|j|jd|jdS)Nr)rZ unsqueeze)rembed1embed2s r sim_matrixzCachedGISTEmbedLoss.sim_matrixs+""6#3#3A#68H8H8KLLrc |rtntj}| tn|}|jD cic] \} } | | || } } } |5|5|rt | j nd}|j | d} dddtj5|jr|jj| dd} |jj| } | jDcic]+\}}||j|jj-} }}|j| d}dddddd |fScc} } w#1swYxYwcc}}w#1swY.xYw#1swY2xYw)zYDo forward pass on a minibatch of the input features and return corresponding embeddings.Nsentence_embedding input_idsT)skip_special_tokens)rrno_graditemsrvaluesrVr`rP batch_decoderWtokenizetodevice)rr7beginendr8r9 random_state grad_contextrandom_state_contextkvsentence_feature_minibatchrepsdecodedkeyvalue guide_repss rembed_minibatchz#CachedGISTEmbedLoss.embed_minibatchs'0{U]] 0<0D{},BRBXBXBZ%[$!Qa5o%["%[ ! Z TTe{,F,M,M,OPko zz"<=>RS T Z''"nn992;?UY:G261D1DW1M.KeKkKkKm2=GS%UXXdjj&7&7882.2"ZZ(BCDXY  Z ZZ--!&\ T T2 Z Z Z ZsTEE=/EE="AE1=0E+ -E1E=E( $E=+E11E: 6E==Fc #$K|d}|j\}}ttjd||jd|j D]=\}} | |jz} |j || | |||dn||\} } } | | | f?yw)z`Do forward pass on all the minibatches of the input features and yield corresponding embeddings.rqrzEmbed mini-batchesdescdisableN)r7rzr{r8r9r|)shape enumeratetqdmtranger\r]r)rr7r8r9r:rqbszrGirzr{rrr|s rr>z(CachedGISTEmbedLoss.embed_minibatch_iters-[9 Q! KK$$) 222    1HAu$...C-1-A-A!1#"3%2%:T a@P .B. *D*l L0 0% 1sBBc|j||d}|jj}|Dcgc]}|Dcgc]}|jc}c}}|_|Scc}wcc}}w)zeGeneralized function to calculate the cross-entropy loss and cache the gradients wrt. the embeddings.T) with_backward)calculate_lossdetachrequires_grad_rEr;)rr reps_guidedlossrsrs r"calculate_loss_and_cache_gradientsz6CachedGISTEmbedLoss.calculate_loss_and_cache_gradientss[""4D"I{{}++-59:rr*!qvv*:  +:s A)A$ A)$A)c bt|t|k7r tdtj|d}tj|d}|ddDcgc]}tj|}}|ddDcgc]}tj|}}|j d} d} j rq|D cgc] } t | }} |D cgc] } t | }} tjjr#tjj} | | z} tj| | | z|j} g}tjd| jdj D]}|jz}j!||||d}j!||||d}|j#| |zj%d dddfd }tj&|j(tj*|jd }|j-|}|||| }|g}j.rDj!||||}j!||||}|||}|j1|j2rvj!|d| |zt5| |z| | z|d}j!|d| |zt5| |z| | z|d}|||}|j1|t|dkDrdt7dt|D]L}j!|||||}j!|||||}|||}|j1|Ntj|d }|j8z }j;|| ||t|z| z }|r |j=|j?}|j1|tA|}|Scc}wcc}wcc} wcc} w)zSGeneralized function to calculate the cross-entropy loss without caching gradients.z.reps and reps_guided must have the same lengthrrjN)ryzCalculating lossr)offsetrMcjdk(r|jz kD}n$jdk(r|djz zkD}||z}tj |<|S)NrQrRrj)rarbrinf)guided_sim_matsim_mat positive_maskmask guided_simrs rmask_false_negativesz@CachedGISTEmbedLoss.calculate_loss..mask_false_negativesJso'':5)Z$++-EFD))Z7)Z1t{{?-KLD ,=.0D!&  r)dtypery)rrNr)rz Tensor | None)!lenrUrcatsizerer distributedis_initializedget_rankarangeryrrr\r]rndiagonalvieweyerboolrollrcappendrdminrangerXrgrArsum) rrrranchors anchors_guider candidatescandidates_guide batch_sizer candidaterank range_labelslossesrzr{ap_sim guided_ap_simrrscoresaa_sim guided_aa_simpp_sim guided_pp_simrneg_simguided_neg_sim loss_mbatchrrs ` @rrz"CachedGISTEmbedLoss.calculate_losss- t9K( (MN N))DG$ +a.1 ,0H5qeiil5 52=ab/BQEIIaLBB\\!_   % %LVVi.y9VJVQabI 4Y ?b b  //1((113 *||FFZ,?W %'[[   #...  J 'E$...C__WU3%7AGF OOM%,DFVWXFYZM'//v~/FKKBPQRJ "II}':':%**UbUiUijM)..u5M*-}]FXF$$s);WE $ eC0H- X -mVD f%&&qM&5.3v|VjEX3YZ\fgh\i!%$Q'Vc\6T^K^9_`brstbu! .mVD f%:"q#j/2+A"oogeC.@*Q-PG%)__]55MO_`aOb%cN2>7KGMM'* +YYv1-Fd...F'' U30GH3v;VYcc $$&)002 MM+ &UJ 'X6{ I6BWbsPP"P'P,cZg}g}g|_|D]}g}g}g}|j|ddD]c\} } } |j| jj |j| j|j| e|j||j||jj|t j r5|j||} | jtt||| S|j||} | S)NFT)r7r8r9)rCrD) r:r>rrrris_grad_enabledr register_hookrrJr) rrClabelsrrr7reps_mbsreps_guided_mbsrandom_state_mbsrFreps_guided_mbr|rs rforwardzCachedGISTEmbedLoss.forwards3  1 8 H O! 9=9R9R!1"&:S: 65  0 ? ? AB&&~'<'<'>? '' 5 6 KK !    /    % %&6 7 8  "::4MD   w~IZeij k &&t[9D rc |j|j|j|j|j|j |j |jdS)NrWrXr\rarbrcrdrerr(s rget_config_dictz#CachedGISTEmbedLoss.get_config_dictsPZZ++#33#33kk $ 5 5"&"9"9%)%?%?  r)g{Gz? FrQgTTF)rVrrWrrXfloatr\intr]rrazLiteral['absolute', 'relative']rbrrcrrdrrerr.r/)rlr rmr r.r r)r7dict[str, Tensor]rzrr{rr8rr9rr|zRandContext | Noner.z)tuple[Tensor, Tensor, RandContext | None]) r7rr8rr9rr:zlist[RandContext] | Noner.z3Iterator[tuple[Tensor, Tensor, RandContext | None]])rlist[list[Tensor]]rrr.r )F)rrrrrrr.r )rCIterable[dict[str, Tensor]]rr r.r )r.zdict[str, Any]) r0r1r2rrnrr>rrrr __classcell__)rhs@rr5r5Asl "!"';E!%#'&+L8"L8#L8 L8  L8  L89L8L8L8!L8 $L8 L8\M,0.+.. .  .  .). 3.F37 1+11 1 0 1 = 1<`en&n5GnX\n n`@  r)rBr rCrrDr5r.r/) __future__rcollections.abcrr contextlibr functoolsrtypingrr rrr r torch.utils.checkpointr r transformersrsentence_transformersrsentence_transformers.modelsrsentence_transformers.utilrrrJModuler5r4rrrsq"." G058;.%%2%"% %.q "))q r