rh6ddlmZddlmZddlmZddlmZddlZddl Z ddlm Z m Z ddl m Z mZddlmZdd lmZGd d Z dd ZGdd eZy)) annotations)Iterator) nullcontext)partialN)Tensornn)get_device_statesset_device_states) CrossEncoder)MultipleNegativesRankingLossc(eZdZdZddZddZddZy) RandContexta Random-state context manager class. Reference: https://github.com/luyug/GradCache. This class will back up the pytorch's random state during initialization. Then when the context is activated, the class will set up the random state with the backed-up one. c`tj|_t|\|_|_yN)torch get_rng_state fwd_cpu_stater fwd_gpu_devicesfwd_gpu_states)selftensorss /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/sentence_transformers/cross_encoder/losses/CachedMultipleNegativesRankingLoss.py__init__zRandContext.__init__s("0024Ew4O1d1ctjj|jd|_|jj tj |jt|j|jy)NT)devicesenabled) rrandomfork_rngr_fork __enter__ set_rng_staterr r)rs rr!zRandContext.__enter__s^\\**43G3GQU*V   D../$..0C0CDrcL|jj|||d|_yr)r __exit__)rexc_typeexc_valexc_tbs rr$zRandContext.__exit__"s Hgv6 rN)returnNone)__name__ __module__ __qualname____doc__rr!r$rrrrsPE rr"CachedMultipleNegativesRankingLossc |jJ|jJtj5t |j |dd|j|jD]M\\}}}tj |j|j|z}|jO dddy#1swYyxYw)zOA backward hook to backpropagate the cached gradients mini-batch by mini-batch.NTF)pairs with_gradcopy_random_state random_states) cacher4r enable_gradzippredict_minibatch_iterdotflattenbackward) grad_outputr1loss_objminibatch_logits_minibatch_grad surrogates r_backward_hookrB's >> %% %  ! ! -- -     !58  + +"'&44 ,  NN6 ! 1 ! q> "2":":"T>T>VWZeeI     ! ! ! !s BB??CceZdZddejddf d fd Z d d dZ d ddZddZdd Z fd Z xZ S)r/g$@ Fct|||||||_||_t j |_d|_d|_t|jts8t|jjdt|jd|jj dk7r9t|jjd|jj dy)a Boosted version of :class:`~sentence_transformers.cross_encoder.losses.MultipleNegativesRankingLoss` that caches the gradients of the logits wrt. the loss. This allows for much higher batch sizes without extra memory usage. However, it is slightly slower. In detail: (1) It first does a quick prediction step without gradients/computation graphs to get all the logits; (2) Calculate the loss, backward up to the logits and cache the gradients wrt. to the logits; (3) A 2nd prediction step with gradients/computation graphs and connect the cached gradients into the backward chain. Notes: All steps are done with mini-batches. In the original implementation of GradCache, (2) is not done in mini-batches and requires a lot memory when the batch size is large. The gradient caching will sacrifice around 20% computation time according to the paper. Given a list of (anchor, positive) pairs or (anchor, positive, negative) triplets, this loss optimizes the following: * Given an anchor (e.g. a question), assign the highest similarity to the corresponding positive (i.e. answer) out of every single positive and negative (e.g. all answers) in the batch. If you provide the optional negatives, they will all be used as extra options from which the model must pick the correct positive. Within reason, the harder this "picking" is, the stronger the model will become. Because of this, a higher batch size results in more in-batch negatives, which then increases performance (to a point). This loss function works great to train embeddings for retrieval setups where you have positive pairs (e.g. (query, answer)) as it will sample in each batch ``n-1`` negative docs randomly. This loss is also known as InfoNCE loss with GradCache. Args: model (:class:`~sentence_transformers.cross_encoder.CrossEncoder`): A CrossEncoder model to be trained. num_negatives (int, optional): Number of in-batch negatives to sample for each anchor. Defaults to 4. scale (int, optional): Output of similarity function is multiplied by scale value. Defaults to 10.0. activation_fn (:class:`~torch.nn.Module`): Activation function applied to the logits before computing the loss. Defaults to :class:`~torch.nn.Sigmoid`. mini_batch_size (int, optional): Mini-batch size for the forward pass. This informs the memory usage. Defaults to 32. show_progress_bar (bool, optional): Whether to show a progress bar during the forward pass. Defaults to False. .. note:: The current default values are subject to change in the future. Experimentation is encouraged. References: - Efficient Natural Language Response Suggestion for Smart Reply, Section 4.4: https://arxiv.org/pdf/1705.00652.pdf - Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup: https://arxiv.org/pdf/2101.06983.pdf - `Cross Encoder > Training Examples > MS MARCO <../../../examples/cross_encoder/training/ms_marco/README.html>`_ - `Cross Encoder > Training Examples > Rerankers <../../../examples/cross_encoder/training/rerankers/README.html>`_ Requirements: 1. Your model must be initialized with `num_labels = 1` (a.k.a. the default) to predict one class. 2. Should be used with large `per_device_train_batch_size` and low `mini_batch_size` for superior performance, but slower training time than :class:`MultipleNegativesRankingLoss`. Inputs: +-------------------------------------------------+--------+-------------------------------+ | Texts | Labels | Number of Model Output Labels | +=================================================+========+===============================+ | (anchor, positive) pairs | none | 1 | +-------------------------------------------------+--------+-------------------------------+ | (anchor, positive, negative) triplets | none | 1 | +-------------------------------------------------+--------+-------------------------------+ | (anchor, positive, negative_1, ..., negative_n) | none | 1 | +-------------------------------------------------+--------+-------------------------------+ Recommendations: - Use ``BatchSamplers.NO_DUPLICATES`` (:class:`docs `) to ensure that no in-batch negatives are duplicates of the anchor or positive samples. - Use :class:`~sentence_transformers.util.mine_hard_negatives` with ``output_format="n-tuple"`` or ``output_format="triplet"`` to convert question-answer pairs to triplets with hard negatives. Relations: - Equivalent to :class:`~sentence_transformers.cross_encoder.losses.MultipleNegativesRankingLoss`, but with caching that allows for much higher batch sizes (and thus better performance) without extra memory usage. This loss also trains slower than :class:`~sentence_transformers.cross_encoder.losses.MultipleNegativesRankingLoss`. Example: :: from sentence_transformers.cross_encoder import CrossEncoder, CrossEncoderTrainer, losses from datasets import Dataset model = CrossEncoder("microsoft/mpnet-base") train_dataset = Dataset.from_dict({ "query": ["What are pandas?", "What is the capital of France?"], "answer": ["Pandas are a kind of bear.", "The capital of France is Paris."], }) loss = losses.CachedMultipleNegativesRankingLoss(model, mini_batch_size=32) trainer = CrossEncoderTrainer( model=model, train_dataset=train_dataset, loss=loss, ) trainer.train() Nz? expects a model of type CrossEncoder, but got a model of type .z; expects a model with 1 output label, but got a model with z output labels.)superrmini_batch_sizeshow_progress_barrCrossEntropyLosscross_entropy_lossr5r4 isinstancemodelr ValueError __class__r*type num_labels)rrO num_negativesscale activation_fnrJrKrQs rrz+CachedMultipleNegativesRankingLoss.__init__?sN  umD.!2"$"5"5"704 =A$**l3>>**+,++/ +;*>**+,((, (=(='>oO  &rc|rtntj}| tn|}|5|5|r t|nd}|j |}dddddd|fS#1swYxYw#1swY|fSxYw)zYDo forward pass on a minibatch of the input features and return corresponding embeddings.N)rrno_gradrcall_model_with_pairs)rr1r2r3 random_state grad_contextrandom_state_contextlogitss rpredict_minibatchz4CachedMultipleNegativesRankingLoss.predict_minibatchs'0{U]] 0<0D{}, ! ; ;5F{51D 33E: ; ;|## ; ; ;|##s"A2!A&A2&A/ +A22A>c #Kttjdt||jd|j D]>\}}||jz}|||}|j ||||dn||\} } | | f@yw)z`Do forward pass on all the minibatches of the input features and yield corresponding embeddings.rzPredict mini-batches)descdisableN)r1r2r3rZ) enumeratetqdmtrangelenrJrKr^) rr1r2r3r4ibemini_batch_pairsr]rZs rr8z9CachedMultipleNegativesRankingLoss.predict_minibatch_iters KKE $$+ 222    'DAqD(((A$Qqz #'#9#9&#"3%2%:T a@P $:$ FL ,& &% 'sBBc|j||}|j|jj}|Dcgc]}|jc}|_|Scc}w)zMCalculate the cross-entropy loss and cache the gradients wrt. the embeddings.)calculate_lossr;detachrequires_grad_gradr5)rr] batch_sizelosslogits r"calculate_loss_and_cache_gradientszECachedMultipleNegativesRankingLoss.calculate_loss_and_cache_gradientssQ""6:6 {{}++-.45Uejj5  6sA"c|ddd}|ddd}t|}|j|d|ddD]'}|j|d|j|)|ddD]'}|j|d|j|)tt ||}g}g|_|j |ddD]M\} } |j| jj|j j| Otjr5|j||} | jtt||| S|j!||} | S)NrrHFT)r1r2r3)r1r=)reget_in_batch_negativesextendlistr7r4r8appendrlrmris_grad_enabledrr register_hookrrBrk) rinputslabelsanchors candidatesro negativesr1r]r>rZrps rforwardz*CachedMultipleNegativesRankingLoss.forwards})B-AYr] \ 44VAYqr K )I NN6!9 %   i ( )   )I NN6!9 %   i ( )S*-..2.I.I"/J/  4 * l MM*113BBD E    % %l 3  4  "::6:ND   w~UTR S  &&vz:D rc@it|d|jiS)NrJ)rIget_config_dictrJ)rrQs rrz2CachedMultipleNegativesRankingLoss.get_config_dicts$U%')+U->@T@TUUr)rOr rTz int | NonerUfloatrVznn.Module | NonerJintrKboolr(r)r) r1list[list[str]]r2rr3rrZzRandContext | Noner(z!tuple[Tensor, RandContext | None]) r1rr2rr3rr4zlist[RandContext] | Noner(z+Iterator[tuple[Tensor, RandContext | None]])r]z list[Tensor]rorr(r)r{rr|rr(r) r*r+r,rSigmoidrr^r8rrrr __classcell__)rQs@rr/r/>s%&*4"**,!"'yy"y y ( y  y y y@,0 $$$ $ ) $ + $*37 ''' ' 0 ' 5 '8&PVVr)r<rr1rr=r/r(r)) __future__rcollections.abcr contextlibr functoolsrrrcrrtorch.utils.checkpointr r 0sentence_transformers.cross_encoder.CrossEncoderr Gsentence_transformers.cross_encoder.losses.MultipleNegativesRankingLossr rrBr/r.rrrsg"$" GIp.!! !1! !.[V)E[Vr