rh:ddlmZddlZddlZddlZddlmZmZddlZ ddl Z ddl Z ddl m Z mZddlmZddlmZer ddl mZddlmZej,eZGd d eZy) ) annotationsN) TYPE_CHECKINGCallable)average_precision_score ndcg_score)SentenceEvaluator)cos_sim)Tensor)SentenceTransformerc eZdZdZdddedddddf dfd Z d dd Zdd Zdd Zdd Z d dd Z dZ xZ S)RerankingEvaluatoraD This class evaluates a SentenceTransformer model for the task of re-ranking. Given a query and a list of documents, it computes the score [query, doc_i] for all possible documents and sorts them in decreasing order. Then, MRR@10, NDCG@10 and MAP is compute to measure the quality of the ranking. Args: samples (list): A list of dictionaries, where each dictionary represents a sample and has the following keys: - 'query': The search query. - 'positive': A list of positive (relevant) documents. - 'negative': A list of negative (irrelevant) documents. at_k (int, optional): Only consider the top k most similar documents to each query for the evaluation. Defaults to 10. name (str, optional): Name of the evaluator. Defaults to "". write_csv (bool, optional): Write results to CSV file. Defaults to True. similarity_fct (Callable[[torch.Tensor, torch.Tensor], torch.Tensor], optional): Similarity function between sentence embeddings. By default, cosine similarity. Defaults to cos_sim. batch_size (int, optional): Batch size to compute sentence embeddings. Defaults to 64. show_progress_bar (bool, optional): Show progress bar when computing embeddings. Defaults to False. use_batched_encoding (bool, optional): Whether or not to encode queries and documents in batches for greater speed, or 1-by-1 to save memory. Defaults to True. truncate_dim (Optional[int], optional): The dimension to truncate sentence embeddings to. `None` uses the model's current truncation dimension. Defaults to None. mrr_at_k (Optional[int], optional): Deprecated parameter. Please use `at_k` instead. Defaults to None. Example: :: from sentence_transformers import SentenceTransformer from sentence_transformers.evaluation import RerankingEvaluator from datasets import load_dataset # Load a model model = SentenceTransformer("all-MiniLM-L6-v2") # Load a dataset with queries, positives, and negatives eval_dataset = load_dataset("microsoft/ms_marco", "v1.1", split="validation") samples = [ { "query": sample["query"], "positive": [text for is_selected, text in zip(sample["passages"]["is_selected"], sample["passages"]["passage_text"]) if is_selected], "negative": [text for is_selected, text in zip(sample["passages"]["is_selected"], sample["passages"]["passage_text"]) if not is_selected], } for sample in eval_dataset ] # Initialize the evaluator reranking_evaluator = RerankingEvaluator( samples=samples, name="ms-marco-dev", ) results = reranking_evaluator(model) ''' RerankingEvaluator: Evaluating the model on the ms-marco-dev dataset: Queries: 9706 Positives: Min 1.0, Mean 1.1, Max 5.0 Negatives: Min 1.0, Mean 7.1, Max 9.0 MAP: 56.07 MRR@10: 56.70 NDCG@10: 67.08 ''' print(reranking_evaluator.primary_metric) # => ms-marco-dev_ndcg@10 print(results[reranking_evaluator.primary_metric]) # => 0.6708042171399308 T@FNc t |||_||_| !tj d| d| |_n||_||_||_||_ ||_ | |_ t|jtr(t|jj|_|jD cgc](} t!| ddkDst!| ddkDs'| *c} |_d|rd|zndzd |j d z|_d d d d|j d|j g|_||_d|j |_ycc} w)Nz?The `mrr_at_k` parameter has been deprecated; please use `at_k=z ` instead.positivernegativer _rz _results_@z.csvepochstepsMAPMRR@NDCG@ndcg@)super__init__samplesnameloggerwarningat_ksimilarity_fct batch_sizeshow_progress_baruse_batched_encoding truncate_dim isinstancedictlistvalueslencsv_file csv_headers write_csvprimary_metric) selfrr!rr.r"r#r$r%r&mrr_at_ksample __class__s /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/sentence_transformers/evaluation/RerankingEvaluator.pyrzRerankingEvaluator.__init__YsZ     NN\]e\ffpq r DIDI,$!2$8!( dllD ) 3 3 56DL"& VJ5G1H11LQTU[\fUgQhklQlF  -dd KPZ[_[d[dZeeiNjj   499+ DII;   # %dii[1 s;EE$Ec(|dk7r|dk(rd|}n d|d|d}nd}|j|d|jd z }tjd |jd |d |j |}|d }|d}|d} |j D cgc]} t | d} } |j D cgc]} t | d} } tjdt |j dtj| ddtj| ddtj| ddtj| ddtj| ddtj| dtjd|dzdtjd|jd|dzdtjd|jd| dzd||jrtjj||j } tjj#| }t%| d|rdndd !5}t'j(|}|s|j+|j,|j+||||| gdddd |d"|j|d#|j| i}|j/||j}|j1|||||Scc} wcc} w#1swYgxYw)$a Evaluates the model on the dataset and returns the evaluation metrics. Args: model (SentenceTransformer): The SentenceTransformer model to evaluate. output_path (str, optional): The output path to write the results. Defaults to None. epoch (int, optional): The current epoch number. Defaults to -1. steps (int, optional): The current step number. Defaults to -1. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. z after epoch z in epoch z after z stepsrNz (truncated to )z0RerankingEvaluator: Evaluating the model on the z dataset:mapmrrndcgrrz Queries: z Positives: Min z.1fz, Mean z, Max z Negatives: Min zMAP: dz.2frz: rawzutf-8)newlinemodeencodingzmrr@r)r&rinforcompute_metricesrr+npminmeanmaxr!r.ospathjoinr,isfileopencsvwriterwriterowr-prefix_name_to_metrics store_metrics_in_model_card_data)r0model output_pathrrout_txtscoresmean_apmean_mrr mean_ndcgr2 num_positives num_negativescsv_pathoutput_file_existsfrNmetricss r4__call__zRerankingEvaluator.__call__s B;{)%1&ugWUG6BG    ( ):):(;1= =G FtyykQYZaYbbcde&&u--%=6N @D||LVVJ/0L L?C||LVVJ/0L L DLL)**=bff]>STW=XX_`b`g`ghu`vwz_{|BCECICIJWCXY\B]]pqsqwqwxEqFGJpKKRSUSZSZ[hSijmRnntuwu{u{|IuJKNtO P   eGcM#./0 d499+R3s';<= eDII;bS(=>?  "t~~ww||K?H!#!9 h8JPS^ef NjkA)OOD$4$45w) LM  N 7499+ DII;   --gtyyA --eWeUK9ML N NsK>5LA LLc^|jr|j|S|j|S)a Computes the evaluation metrics for the given model. Args: model (SentenceTransformer): The SentenceTransformer model to compute metrics for. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. )r%compute_metrices_batchedcompute_metrices_individual)r0rRs r4rCz#RerankingEvaluator.compute_metricess7((  ) )% 0 11%8 cng}g}g}|j||jDcgc]}|d c}d|j}g}|jD]*}|j|d|j|d,|j||d|j}d\} } |jD]=} || } | dz } t | d} t | d}|| | | z|z}| | |zz } | dk(s|dk(rH|j | |}t |j dkDr|d}tj| }|jj}dg| zdg|zz}d}t|d|jD]\}}||s d|dzz }n|j||jt|g|g|j |jt||@t!j"|}t!j"|}t!j"|}|||d Scc}w) aE Computes the evaluation metrics in a batched way, by batching all queries and all documents together. Args: model (SentenceTransformer): The SentenceTransformer model to compute metrics for. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. queryencode_fn_namer$rrdocument)rrrkr9r:r;) embed_inputsrr$extendr+r"shapetorchargsortcputolist enumerater!appendrrrDrF)r0rRall_mrr_scoresall_ndcg_scores all_ap_scoresr2all_query_embsall_docs all_docs_embs query_idxdocs_idxinstance query_embnum_posnum_negdocs_emb pred_scorespred_scores_argsort is_relevant mrr_scorerankindexrVrWrXs r4raz+RerankingEvaluator.compute_metrices_batchedsm ** +/<< 8VG_ 8""44 + ll 0F OOF:. / OOF:. / 0)) 8J$J`J`* # 8 TH&y1I NI(:./G(:./G$X70BW0LMH ') )H!|w!|--iBK;$$%))!n "'-- "= %//+224K#-1#-7KI()*GGO, xCCo 9s H2 cg}g}g}tj|j|j dD]y}|d}t|d}t|d}t |dk(st |dk(rB||z} dgt |zdgt |zz} |j ||gdd } |j || d d } |j | | } t | jdkDr| d} tj| }| jj} d}t|d|jD]\}}| |s d|dzz }n|j||jt| g| g|j |jt!| | |t#j$|}t#j$|}t#j$|}|||d S) aO Computes the evaluation metrics individually by embedding every (query, positive, negative) tuple individually. Args: model (SentenceTransformer): The SentenceTransformer model to compute metrics for. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. Samples)disabledescrerrrriFrfrhrjrl)tqdmrr$r)r+rmr"rorprqrrrsrtr!rurrrDrF)r0rRrvrwrxr~rerrdocsrrrrrrrrrVrWrXs r4rbz.RerankingEvaluator.compute_metrices_individual!s  $,,D*GGO, xCCrcc | |j}n#|dk(r |j}n|dk(r |j}|f|j|d|jd|S)NrerhT)r#r$convert_to_tensorr&)encode encode_queryencode_documentr#r&)r0rR sentencesrgr$kwargs encode_fns r4rmzRerankingEvaluator.embed_inputsXsn  ! I w &**I z )--I  /"**     rccXd|ji}|j|j|d<|S)Nr!r&)r!r&)r0 config_dicts r4get_config_dictz"RerankingEvaluator.get_config_dictos2tyy)    (*.*;*;K 'rc)rz list[dict[str, str | list[str]]]r!intrstrr.boolr"z4Callable[[torch.Tensor, torch.Tensor], torch.Tensor]r#rr$rr%rr& int | Noner1r)Nr6r6) rRr rS str | Nonerrrrreturnzdict[str, float])rRr )NN) rRr rzstr | list[str] | np.ndarrayrgrr$z bool | Nonerr ) __name__ __module__ __qualname____doc__r rr_rCrarbrmr __classcell__)r3s@r4r r s=DOV"'%)#'#.21.2.2 .2  .2 M .2.2 .2#.2!.2.2bik=(=7A=QT=be= =~ GDR5Dv&*)-  " 0 #  '    .rcr ) __future__rrMloggingrHtypingrrnumpyrDrprsklearn.metricsrr2sentence_transformers.evaluation.SentenceEvaluatorrsentence_transformers.utilr r )sentence_transformers.SentenceTransformerr getLoggerrrr rcr4rsR"  * ?P.M   8 $Z*Zrc