rh@<`ddlmZddlZddlmZmZddlmZddlm Z ddl m Z m Z ddl mZddlZddlmZmZmZdd lmZerdd lmZej0eZGd d ZGd deeZGddeZGddeZGddeeeZGddeZ GddeZ!y)) annotationsN)ABCabstractmethod) defaultdict)Iterator) accumulatecycle)Any) BatchSampler ConcatDatasetSubsetRandomSampler)is_datasets_available)Datasetc,eZdZdZdfd ZddZxZS) SetEpochMixinz Required for a BatchSampler as the Trainer will call set_epoch on the BatchSampler at the beginning of each epoch. The BatchSampler can then set the generator seed accordingly. c2t||i|d|_yNr)super__init__epoch)selfargskwargs __class__s p/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/sentence_transformers/sampler.pyrzSetEpochMixin.__init__s $)&) c||_yN)r)rrs r set_epochzSetEpochMixin.set_epochs  r)returnNone)rintr r!)__name__ __module__ __qualname____doc__rr __classcell__rs@rrrs rrcFeZdZdZ d dfd ZxZS)DefaultBatchSamplera This sampler is the default batch sampler used in the SentenceTransformer library. It is equivalent to the PyTorch BatchSampler. Args: sampler (Sampler or Iterable): The sampler used for sampling elements from the dataset, such as SubsetRandomSampler. batch_size (int): Number of samples per batch. drop_last (bool): If True, drop the last incomplete batch if the dataset size is not divisible by the batch size. valid_label_columns (List[str], optional): List of column names to check for labels. The first column name from ``valid_label_columns`` found in the dataset will be used as the label column. generator (torch.Generator, optional): Optional random number generator for shuffling the indices. seed (int): Seed for the random number generator to ensure reproducibility. Defaults to 0. cTt||||||_||_||_y)N batch_size drop_last)rrvalid_label_columns generatorseed)rdatasetr-r.r/r0r1rs rrzDefaultBatchSampler.__init__6s/ Z9M#6 " rNNrr2rr-r"r.boolr/list[str] | Noner0torch.Generator | Noner1r"r r!)r#r$r%r&rr'r(s@rr*r*#sa.15,0     .  *     rr*c`eZdZdZ d dfd ZeddZddZxZS) GroupByLabelBatchSampleraZ This sampler groups samples by their labels and aims to create batches such that each batch contains samples where the labels are as homogeneous as possible. This sampler is meant to be used alongside the ``Batch...TripletLoss`` classes, which require that each batch contains at least 2 examples per label class. Recommended for: - :class:`~sentence_transformers.losses.BatchAllTripletLoss` - :class:`~sentence_transformers.losses.BatchHardSoftMarginTripletLoss` - :class:`~sentence_transformers.losses.BatchHardTripletLoss` - :class:`~sentence_transformers.losses.BatchSemiHardTripletLoss` Args: dataset (Dataset): The dataset to sample from. batch_size (int): Number of samples per batch. Must be divisible by 2. drop_last (bool): If True, drop the last incomplete batch if the dataset size is not divisible by the batch size. valid_label_columns (List[str], optional): List of column names to check for labels. The first column name from ``valid_label_columns`` found in the dataset will be used as the label column. generator (torch.Generator, optional): Optional random number generator for shuffling the indices. seed (int): Seed for the random number generator to ensure reproducibility. Defaults to 0. ct |||||||||_|jdzdk(r t d|j ||j }tt}t|D]\} } || j| |jD cic]\} } t| dzdzx} r| | d|  c} } |_ ycc} } w)Nr-r.r/r0r1zEThe batch size for `GroupByLabelBatchSampler` must be divisible by 2.)rrr2r- ValueError_determine_labels_to_user/rlist enumerateappenditemslengroups)rr2r-r.r/r0r1labelsrE sample_idxlabelsample_indices num_samplesrs rrz!GroupByLabelBatchSampler.__init___s  ! 3   ??Q ! #de e..w8P8PQT"!*6!2 - J 5M  , - *0 %~">2a7!;; ; >,;/ /   s #C cz|xsgD]}||jvs||cStd|d|jd)Nz None of the valid_label_columns z3 are in the dataset, which only has these columns: .) column_namesr>)r2r/ column_names rr?z1GroupByLabelBatchSampler._determine_labels_to_usesa.4" ,Kg222{++ ,./B.CD--4-A-A,B! E  rc#rK|jr>|j2|jj|j|jzg}t |j j }tjt|j |jD]y}||}|j |}|j|t||jk\sA|d|j||jd}t||jk\r9{|js|r|yyyw)Nr0) r0r1 manual_seedrr@rEkeystorchrandpermrDextendr-r.)r partial_batch unique_labels label_idxrHsampless r__iter__z!GroupByLabelBatchSampler.__iter__s >>dii3 NN & &tyy4::'= > T[[--/0 DKK(8DNNS AI!),Ekk%(G   )m$7#$5doo66 -doo.? @ m$7  A~~- #0~sC$D7'8D7 D7r3r4)r2rr/r6r z list[Any]r zIterator[list[int]]) r#r$r%r&r staticmethodr?rZr'r(s@rr9r9Esv<15,0     .  *     B   rr9cReZdZ d dfd ZddZddZxZS)NoDuplicatesBatchSamplerct|||||||t|jt|jxsgzx}r|j t |}||_y)a This sampler creates batches such that each batch contains samples where the values are unique, even across columns. This is useful when losses consider other samples in a batch to be in-batch negatives, and you want to ensure that the negatives are not duplicates of the anchor/positive sample. Recommended for: - :class:`~sentence_transformers.losses.MultipleNegativesRankingLoss` - :class:`~sentence_transformers.losses.CachedMultipleNegativesRankingLoss` - :class:`~sentence_transformers.losses.MultipleNegativesSymmetricRankingLoss` - :class:`~sentence_transformers.losses.CachedMultipleNegativesSymmetricRankingLoss` - :class:`~sentence_transformers.losses.MegaBatchMarginLoss` - :class:`~sentence_transformers.losses.GISTEmbedLoss` - :class:`~sentence_transformers.losses.CachedGISTEmbedLoss` Args: dataset (Dataset): The dataset to sample from. batch_size (int): Number of samples per batch. drop_last (bool): If True, drop the last incomplete batch if the dataset size is not divisible by the batch size. valid_label_columns (List[str], optional): List of column names to check for labels. The first column name from ``valid_label_columns`` found in the dataset will be used as the label column. generator (torch.Generator, optional): Optional random number generator for shuffling the indices. seed (int): Seed for the random number generator to ensure reproducibility. Defaults to 0. r;N)rrsetrMr/remove_columnsr@r2) rr2r-r.r/r0r1 label_columnsrs rrz!NoDuplicatesBatchSampler.__init__stF  ! 3    4 45D|j2|jj|j|jztj t jt|j|jj}|rt}g}|D]}|j|jDchc]\}}|dk7s t|}}}||zrG|j|t||jk(r|n#|j!||j"s||D]}||=|ryycc}}ww)a5 Iterate over the remaining non-yielded indices. For each index, check if the sample values are already in the batch. If not, add the sample values to the batch keep going until the batch is full. If the batch is full, yield the batch indices and continue with the next batch. NrP dataset_name)r0r1rQrdictfromkeysrSrTrDr2tolistr`rCstrrBr-updater.)rremaining_indices batch_values batch_indicesindexkeyvalue sample_valuess rrZz!NoDuplicatesBatchSampler.__iter__s2 >>dii3 NN & &tyy4::'= > !MM%..T\\9JVZVdVd*e*l*l*no5LM* (>Bll5>Q>W>W>Y s U]`dr]rU s s </$$U+}%8''##M2 (~~''& -%e, -) !tsCE  E# E0A'E E c|jr"t|j|jzSt|j|jzdz |jzS)Nr=)r.rDr2r-rs r__len__z NoDuplicatesBatchSampler.__len__sG >>t||$7 7 %7!;O Orr3r4r[r r")r#r$r%rrZrsr'r(s@rr^r^sd 15,0--- - . - * -- -^"-HPrr^c`eZdZdZ d dfd ZeddZeddZxZS) MultiDatasetDefaultBatchSampleraT Abstract base batch sampler that yields batches from multiple batch samplers. This class must be subclassed to implement specific sampling strategies, and cannot be used directly. Args: dataset (ConcatDataset): A concatenation of multiple datasets. batch_samplers (List[BatchSampler]): A list of batch samplers, one for each dataset in the ConcatDataset. generator (torch.Generator, optional): A generator for reproducible sampling. Defaults to None. seed (int): Seed for the random number generator to ensure reproducibility. Defaults to 0. ct|jt|k7r tdt|||dj |dj ||_||_||_ ||_ y)NzTThe number of batch samplers must match the number of datasets in the ConcatDataset.rr,) rDdatasetsr>rrr-r.r2batch_samplersr0r1)rr2ryr0r1rs rrz(MultiDatasetDefaultBatchSampler.__init__sr w C$7 7st t ^A->-I-IUcdeUfUpUpq ," rcy)z?Yield batches from the underlying datasets in a specific order.Nrrs rrZz(MultiDatasetDefaultBatchSampler.__iter__ rcy)z,Return the number of batches in the sampler.Nr{rrs rrsz'MultiDatasetDefaultBatchSampler.__len__r|rr) r2r ryzlist[BatchSampler]r0r7r1r"r r!r[rt) r#r$r%r&rrrZrsr'r(s@rrvrvsm  -1  + *         rrvc eZdZdZddZddZy)RoundRobinBatchSamplera Batch sampler that yields batches in a round-robin fashion from multiple batch samplers, until one is exhausted. With this sampler, it's unlikely that all samples from each dataset are used, but we do ensure that each dataset is sampled from equally. Args: dataset (ConcatDataset): A concatenation of multiple datasets. batch_samplers (List[BatchSampler]): A list of batch samplers, one for each dataset in the ConcatDataset. generator (torch.Generator, optional): A generator for reproducible sampling. Defaults to None. seed (int): Seed for the random number generator to ensure reproducibility. Defaults to 0. c#<K|jr>|j2|jj|j|jz|jj Dcgc] }t |}}dgtt|z}|jDcgc] }t|}}ttt |D])}||} t||Dcgc]}||z c}+ycc}wcc}wcc}w#t$rYywxYwwr)r0r1rQrr2rxrDr@rryiterr rangenext StopIteration) rr2rJsample_offsetssamplerry dataset_idx sample_offsetidxs rrZzRoundRobinBatchSampler.__iter__+s >>dii3 NN & &tyy4::'= >37<<3H3HIs7|I ItJ{$;<<7;7J7JKG$w-KK s>':!;< K*;7M 6:>+;V6WXss]*XX  JLY   sTA#D%C=7(DD1'DD ) D5D : DD  DDDDcftd|jDt|jzS)Nc32K|]}t|ywr)rD).0rs r z1RoundRobinBatchSampler.__len__..<sCG3w<Cs)minryrDrrs rrszRoundRobinBatchSampler.__len__;s)Ct/B/BCCc$J]J]F^^^rNr[rtr#r$r%r&rZrsr{rrrrs  _rrc eZdZdZddZddZy)ProportionalBatchSamplera| Batch sampler that samples from each dataset in proportion to its size, until all are exhausted simultaneously. With this sampler, all samples from each dataset are used and larger datasets are sampled from more frequently. Args: dataset (ConcatDataset): A concatenation of multiple datasets. batch_samplers (List[BatchSampler]): A list of batch samplers, one for each dataset in the ConcatDataset. generator (torch.Generator, optional): A generator for reproducible sampling. Defaults to None. seed (int): Seed for the random number generator to ensure reproducibility. Defaults to 0. c#K|jj|j|jz|jj Dcgc] }t |}}dgtt|z}|jDcgc] }t |}}t|Dcgc]\}}t|D]}|} }}}t| |j} |jDcgc] }t|} }| D])} || } t| | Dcgc]}|| z c}+ycc}wcc}wcc}}}wcc}wcc}w#t$rYSwxYww)NrrP)r0rQr1rr2rxrDr@rryrArr rrr)rr2rJrr num_batchesrlength_dataset_indicesdataset_idx_samplerryrrs rrZz!ProportionalBatchSampler.__iter__KsC ""499tzz#9:37<<3H3HIs7|I ItJ{$;<<373F3FGs7|G G2;K2H``;3RWX^R_`Q3`3``1/T^^\7;7J7JKG$w-KK. K*;7M 6:>+;V6WXss]*XX JH`LY   slA E D2(ED7E*D<)E/E EE  E*E /EE  EEEEcdt|jDcgc] }t|c}Scc}wr)sumryrD)rrs rrsz ProportionalBatchSampler.__len__]s%0C0CDWCLDEEDs-Nr[rtrr{rrrr?s $Frr)" __future__rloggingabcrr collectionsrcollections.abcr itertoolsrr typingr rStorch.utils.datar r r sentence_transformers.utilrrxr getLoggerr#loggerrr*r9r^rvrrr{rrrs"##$' MM<   8 $  -DT 2T nXP2XPv$ m\3$ N_<_BF>Fr