rhW4ddlmZddlmZddlmZmZddlZddlmZm Z ddl m Z ddl m Z ddlmZdd lmZGd d e j$Zy) ) annotations)Iterable)AnyLiteralN)Tensornn)PreTrainedTokenizerBase)StaticEmbedding)SentenceTransformer)all_gather_with_gradczeZdZ d dfd ZddZd dZd dZed dZxZ S) GISTEmbedLossc $t |||_||_||_t j d|_t|dr t|ds tdt|jtrt|jts td|jj|jjk7xs|j|jk|_|jrC|jj|_ t|jdt r td|dvr td ||_||_||_||_||_t j,|_y ) ab This loss is used to train a SentenceTransformer model using the GISTEmbed algorithm. It takes a model and a guide model as input, and uses the guide model to guide the in-batch negative sample selection. The cosine similarity is used to compute the loss and the temperature parameter is used to scale the cosine similarities. You can apply different false-negative filtering strategies to discard hard negatives that are too similar to the positive. Two strategies are supported: - "absolute": Discards negatives whose similarity score is greater than or equal to ``positive_score - margin``. - "relative": Discards negatives whose similarity score is greater than or equal to ``positive_score * (1 - margin)``. Args: model: SentenceTransformer model based on a `transformers` model. guide: SentenceTransformer model to guide the in-batch negative sample selection. temperature: Temperature parameter to scale the cosine similarities. Inverse of the ``scale`` parameter in :class:`MultipleNegativesRankingLoss`. margin_strategy: Strategy used for false negative filtering. One of {"absolute", "relative"}. margin: The margin value for filtering negatives. Defaults to 0.0, together with the "absolute" strategy, this only removes negatives that are more similar to the query than the positive is to the query. contrast_anchors: If True, include anchor-anchor pairs in the loss computation, resulting in the embeddings of the anchors being pushed further apart. Defaults to True, following the original GISTEmbed paper. contrast_positives: If True, include positive-positive pairs in the loss computation, resulting in the embeddings of the positives being pushed further apart. Defaults to True, following the original GISTEmbed paper, but setting to False may yield better results in some retrieval tasks. gather_across_devices: If True, gather the embeddings across all devices before computing the loss. Recommended when training on multiple GPUs, as it allows for larger batch sizes, but it may slow down training due to communication overhead, and can potentially lead to out-of-memory errors. References: - For further details, see: https://arxiv.org/abs/2402.16829 Requirements: 1. (anchor, positive, negative) triplets 2. (anchor, positive) pairs Inputs: +---------------------------------------+--------+ | Texts | Labels | +=======================================+========+ | (anchor, positive, negative) triplets | none | +---------------------------------------+--------+ | (anchor, positive) pairs | none | +---------------------------------------+--------+ Recommendations: - Use ``BatchSamplers.NO_DUPLICATES`` (:class:`docs `) to ensure that no in-batch negatives are duplicates of the anchor or positive samples. Relations: - :class:`MultipleNegativesRankingLoss` is similar to this loss, but it does not use a guide model to guide the in-batch negative sample selection. `GISTEmbedLoss` yields a stronger training signal at the cost of some training overhead. Example: :: from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer, losses from datasets import Dataset model = SentenceTransformer("microsoft/mpnet-base") guide = SentenceTransformer("all-MiniLM-L6-v2") train_dataset = Dataset.from_dict({ "anchor": ["It's nice weather outside today.", "He drove to work."], "positive": ["It's so sunny.", "He took the car to the office."], }) loss = losses.GISTEmbedLoss(model, guide) trainer = SentenceTransformerTrainer( model=model, train_dataset=train_dataset, loss=loss, ) trainer.train() dim tokenizerzNBoth the training model and the guiding model must have a tokenizer attribute.z_Both the training model and the guiding model must use a PreTrainedTokenizer from transformers.rzIf we must retokenize because the guide model has a different tokenizer, then the Sentence Transformer model must not be based on a StaticEmbedding.)absoluterelativez1margin_strategy must be 'absolute' or 'relative'.N)super__init__modelguide temperaturerCosineSimilaritysimilarity_fcthasattr ValueError isinstancerr get_vocabmax_seq_lengthmust_retokenizer margin_strategymargincontrast_anchorscontrast_positivesgather_across_devicesCrossEntropyLosscross_entropy_loss) selfrrrr#r$r%r&r' __class__s }/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/sentence_transformers/losses/GISTEmbedLoss.pyrzGISTEmbedLoss.__init__s_l   & 11b9uk*'%2Mmn n%//+BC: OO4L q  OO % % '5??+D+D+F F u%J^J^afauauJu    !ZZ11DN$**Q-9 b ": :PQ Q. 0"4%:""$"5"5"7cb|j|jd|jdS)Nr)r unsqueeze)r*embed1embed2s r, sim_matrixzGISTEmbedLoss.sim_matrixs+""6#3#3A#68H8H8KLLr-c|Dcgc]}j|d}}tj5jr|Dcgc]"}jj |dd$}}|Dcgc]}j j|}}|Dcgc]I}|jDcic]+\}}||jj j-c}}K}}}}|Dcgc]}j |d} }dddd} d} t|dk(r |\} }  \}}n2t|dk(r |\} } }  \}}} ntdt|| jd}d}jrot| } t|}| t| } t| } tj j#r#tj j%}||z}j'| | }j'||}|j)| j+d d ddfd }tj,|j.tj0|jd }||||}|g}j2r>j'| | }j'||}|||}|j5|j6rJj'| |||z| }j'||||z|}|||}|j5|| >j'| | }j'|| }|||}|j5|tj8|d j:z }tj<|||z| j}j?||Scc}wcc}wcc}wcc}}wcc}}}wcc}w#1swYxYw)Nsentence_embedding input_idsT)skip_special_tokensz Expected 2 or 3 embeddings, got r)offsetrr/cjdk(r|jz kD}n$jdk(r|djz zkD}||z}tj |<|S)Nrrr/)r#r$torchinf)guided_sim_matsim_mat positive_maskmask guided_simr*s r,mask_false_negativesz3GISTEmbedLoss.forward..mask_false_negativesso##z1%dkk)AB%%3%q4;;)GH(}n,"YYJGDMNr-)dtypedevice)r@r)rE)N)r@z Tensor | None) rr<no_gradr"r batch_decodertokenizeitemstorElenrsizer'r distributedis_initializedget_rankr3diagonalvieweyeshapeboolr%appendr&catraranger)) r*sentence_featureslabelssentence_feature embeddingsdecoded sentenceskeyvalueguide_embeddingsnegativenegative_guideanchorpositive anchor_guidepositive_guide batch_sizer:rankap_sim guided_ap_simrCr@scoresaa_sim guided_aa_simpp_sim guided_pp_siman_sim guided_an_sim range_labelsrBs ` @r,forwardzGISTEmbedLoss.forwardsarsM]djj!123GHs s ]]_ ##->(NN//0@0Mcg/hV]$] TZZ%8%8%C$]!$]->%%(IYH^H^H`a*#uS%((4::#4#455a%!% \m GW +,-AB      z?a ) FH+; (L. _ !)3 &FHh;K 8L..?J?PQR R[[^   % %,H5H1.AN#/9!5n!E  //1((113 *2 nE #++6+:??AF   =#6#6ejjQ^QeQef &mV=Y  __VV4F OOL,GM)-@F MM& !  " "__Xfv 7J%KXVF OON6FZDW,XZhiM)-@F MM& !  __VX6F OOL.IM)-@F MM& !6q)D,<,<<||FFZ,? V &&v|<rs2"$ 08I;yBIIyr-