rhC$vddlmZddlmZddlZddlmZmZddlmZddl m Z GddejZ y) ) annotations) GeneratorN)Tensornn) CrossEncoder)fullnameceZdZddejf d fd Zd dZd dZ d dZddZ ddZ dd Z xZ S)MultipleNegativesRankingLossg$@ct|||_||_||_||_t j|_t|jts8t|jjdt|jd|jjdk7r9t|jjd|jjdy)ax Given a list of (anchor, positive) pairs or (anchor, positive, negative) triplets, this loss optimizes the following: * Given an anchor (e.g. a question), assign the highest similarity to the corresponding positive (i.e. answer) out of every single positive and negative (e.g. all answers) in the batch. If you provide the optional negatives, they will all be used as extra options from which the model must pick the correct positive. Within reason, the harder this "picking" is, the stronger the model will become. Because of this, a higher batch size results in more in-batch negatives, which then increases performance (to a point). This loss function works great to train embeddings for retrieval setups where you have positive pairs (e.g. (query, answer)) as it will sample in each batch ``n-1`` negative docs randomly. This loss is also known as InfoNCE loss, SimCSE loss, Cross-Entropy Loss with in-batch negatives, or simply in-batch negatives loss. Args: model (:class:`~sentence_transformers.cross_encoder.CrossEncoder`): A CrossEncoder model to be trained. num_negatives (int, optional): Number of in-batch negatives to sample for each anchor. Defaults to 4. scale (int, optional): Output of similarity function is multiplied by scale value. Defaults to 10.0. activation_fn (:class:`~torch.nn.Module`): Activation function applied to the logits before computing the loss. Defaults to :class:`~torch.nn.Sigmoid`. .. note:: The current default values are subject to change in the future. Experimentation is encouraged. References: - Efficient Natural Language Response Suggestion for Smart Reply, Section 4.4: https://arxiv.org/pdf/1705.00652.pdf Requirements: 1. Your model must be initialized with `num_labels = 1` (a.k.a. the default) to predict one class. Inputs: +-------------------------------------------------+--------+-------------------------------+ | Texts | Labels | Number of Model Output Labels | +=================================================+========+===============================+ | (anchor, positive) pairs | none | 1 | +-------------------------------------------------+--------+-------------------------------+ | (anchor, positive, negative) triplets | none | 1 | +-------------------------------------------------+--------+-------------------------------+ | (anchor, positive, negative_1, ..., negative_n) | none | 1 | +-------------------------------------------------+--------+-------------------------------+ Recommendations: - Use ``BatchSamplers.NO_DUPLICATES`` (:class:`docs `) to ensure that no in-batch negatives are duplicates of the anchor or positive samples. - Use :class:`~sentence_transformers.util.mine_hard_negatives` with ``output_format="n-tuple"`` or ``output_format="triplet"`` to convert question-answer pairs to triplets with hard negatives. Relations: - :class:`CachedMultipleNegativesRankingLoss` is equivalent to this loss, but it uses caching that allows for much higher batch sizes (and thus better performance) without extra memory usage. However, it is slightly slower. Example: :: from sentence_transformers.cross_encoder import CrossEncoder, CrossEncoderTrainer, losses from datasets import Dataset model = CrossEncoder("microsoft/mpnet-base") train_dataset = Dataset.from_dict({ "query": ["What are pandas?", "What is the capital of France?"], "answer": ["Pandas are a kind of bear.", "The capital of France is Paris."], }) loss = losses.MultipleNegativesRankingLoss(model) trainer = CrossEncoderTrainer( model=model, train_dataset=train_dataset, loss=loss, ) trainer.train() z? expects a model of type CrossEncoder, but got a model of type .z; expects a model with 1 output label, but got a model with z output labels.N)super__init__model num_negativesscale activation_fnrCrossEntropyLosscross_entropy_loss isinstancer ValueError __class____name__type num_labels)selfrrrrrs /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/sentence_transformers/cross_encoder/losses/MultipleNegativesRankingLoss.pyrz%MultipleNegativesRankingLoss.__init__ sb  * *"$"5"5"7$**l3>>**+,++/ +;*>**+,((, (=(='>oO  &cNtt||}|j|S)N)listzipcall_model_with_pairs)ranchors candidatespairss rcall_model_with_columnsz4MultipleNegativesRankingLoss.call_model_with_columnsrs$S*-.))%00rc|jj|ddd}|j|jj|jdi|d}|j dS)NTpt)padding truncationreturn_tensorsrr)r tokenizertodevicesqueeze)rr&tokenslogitss rr#z2MultipleNegativesRankingLoss.call_model_with_pairsvsf%%  &   $**##$%f%a(~~a  rc#VKt|}t|}|Dcgc] }|D]}| }}}tj|tjj d|}|j G|j t|kr/tj |j|j } n=tj|j ||zd} | |j|d} | jD]} | D cgc]} ||  c} ycc}}wcc} ww)N)dtyper) lentorcheyeboolrepeatr multinomialfloatarangereshapeT) rr$r% batch_size num_columnssublist candidatecandidates_flattenedmasknegative_indices all_indicesnegative_indices_row negative_idxs rget_in_batch_negativesz3MultipleNegativesRankingLoss.get_in_batch_negativess\ *o :D]gU\] ] ]] *EJJ7>>q+NN    )d.@.@3G[C\.\$00t?Q?QR  ,,z299*{:RTUVK*4088RH $4$6$6 ` J^_,' 5_ _ ` ^`sD)DCD) D$D)c\tj|djd|j}|jr|j |}|j r||j z}tj ||jtj}|j||}|S)Nr)dimr6)r0r5) r8catr?r@rrzerosr0longr)rr3rAlabelslosss rcalculate_lossz+MultipleNegativesRankingLoss.calculate_losss6q)11"jACC   ''/F ::djj(FZ UZZP&&vv6 rcB|d}|d}t|}|j||g}|j||ddD]#}|j|j||%|ddD]#}|j|j||%|j ||S)Nrr)r7r'rKappendrS)rinputsrQr$ positivesrAscores negativess rforwardz$MultipleNegativesRankingLoss.forwards)1I \ ..w BC44WfQRjI LI MM$66w J K L  LI MM$66w J K L""6:66rc\|j|jt|jdS)N)rrr)rrrr)rs rget_config_dictz,MultipleNegativesRankingLoss.get_config_dicts*ZZ!//%d&8&89  r) rrrz int | Nonerintrznn.Module | NonereturnNone)r$ list[str]r%rar_r)r&list[list[str]]r_r)r$rar%rbr_z Generator[list[str], None, None])r3rrAr^r_r)rWrbrQrr_r)r_zdict[str, float]) r __module__ __qualname__rSigmoidrr'r#rKrSr[r] __classcell__)rs@rr r s%&*4"**, cc"c c ( c  cJ1 !` `.=` )`0 7" rr ) __future__rcollections.abcrr8rr0sentence_transformers.cross_encoder.CrossEncoderrsentence_transformers.utilrModuler r-rrrls)"% I/s 299s r