rh?@ d dl mZ d dlmZ d dlZd dlmZmZ d dlmZ d dl m
Z
G d dej Z G d d
e Z
G d d e Z G d
de Z G d de Z G d de Z G d dej Zy) ) annotations)LiteralN)Tensornn) CrossEncoder)fullnamec , e Zd ZdZd fd ZddZ xZS )BaseWeightingSchemez Training Examples > MS MARCO <../../../examples/cross_encoder/training/ms_marco/README.html>`_
Requirements:
1. Query with multiple documents (listwise approach)
2. Documents must have relevance scores/labels. Both binary and continuous labels are supported.
Inputs:
+----------------------------------------+--------------------------------+-------------------------------+
| Texts | Labels | Number of Model Output Labels |
+========================================+================================+===============================+
| (query, [doc1, doc2, ..., docN]) | [score1, score2, ..., scoreN] | 1 |
+----------------------------------------+--------------------------------+-------------------------------+
Recommendations:
- Use :class:`~sentence_transformers.util.mine_hard_negatives` with ``output_format="labeled-list"``
to convert question-answer pairs to the required input format with hard negatives.
Relations:
- :class:`~sentence_transformers.cross_encoder.losses.LambdaLoss` anecdotally performs better than
the other losses with the same input format.
Example:
::
from sentence_transformers.cross_encoder import CrossEncoder, CrossEncoderTrainer, losses
from datasets import Dataset
model = CrossEncoder("microsoft/mpnet-base")
train_dataset = Dataset.from_dict({
"query": ["What are pandas?", "What is the capital of France?"],
"docs": [
["Pandas are a kind of bear.", "Pandas are kind of like fish."],
["The capital of France is Paris.", "Paris is the capital of France.", "Paris is quite large."],
],
"labels": [[1, 0], [1, 1, 0]],
})
loss = losses.LambdaLoss(model)
trainer = CrossEncoderTrainer(
model=model,
train_dataset=train_dataset,
loss=loss,
)
trainer.train()
r7 z< supports a model with 1 output label, but got a model with z output labels.N)r
r modelr' weighting_schemeksigmaeps
reduction_logr Identity
activation_fnmini_batch_size
num_labels
ValueErrorr r )
r rS rT rU rV rW rX rZ r[ r s
r r zLambdaLoss.__init__h s D
0 G4E4G
* * ;bkkm . :: A % >>**+ ,((,
(=(='>oO
&r c l t |t r t d t | dk7 rt dt | d |\ }}|D cg c]
}t | }}t | }t | }||D cg c]
}t | c}k7 r)t d| dD cg c]
}t | c} d t || D
cg c] \ } }|D ] }
| |
f } }} }
| s,t
j d| j j d
S | j xs |} | d k r t | } g }
t d t | | D ] }| ||| z }| j j |d d d
}|j | j j } | j d i |d j d }|
j | t
j |
d }| j# | }t
j$ ||fd| j j }t
j |D cg c] }t
j& t | " c}d }t
j( t
j& | t
j | }||||f< t
j* |t- d }t
j d j- |||f< |j | j j }|j/ d d \ }}|j/ d d \ }}t
j0 |d| }|dddddf |dddddf z
}t
j2 | } t | j4 t6 s| |d kD z } | j8 xs |} t
j: ||ft j< | j j } d| d| d| f< |j? d |j? d t
j& d|dz j | j j } t
j@ d| j- z dddf } t
jB t
jD d| dz
| z ddd| f d jG | jH }!t
jD d| dz
|!dddf z }"| j5 |"| | }#|dddddf |dddddf z
jG dd }$|$jK t
jL |$ d t
jN | jP |$z jG | jH |#z jG | jH }%| jR d k( rt
jT |% }&nt
j@ |% }&|&| | z }'t
jV |' }(|(S c c}w c c}w c c}w c c}
}} w c c}w ) a/
Compute LambdaLoss for a batch of queries and their documents.
Args:
inputs: List of (queries, documents_list)
labels: Ground truth relevance scores, shape (batch_size, num_documents)
Returns:
Tensor: LambdaLoss loss over the batch
z\LambdaLoss expects a list of labels for each sample, but got a single value for each sample. zALambdaLoss expects two inputs (queries, documents_list), but got z inputs.z)Number of documents per query in inputs (z-) does not match number of labels per query (z).g T)r+
requires_gradr pt)padding
truncationreturn_tensors)dimg 7yAr* z-inf)
descendingrf r7 )rf indexN)dtyper+ )minr) g חg חA)rj maxnaturalr/ ),
isinstancer r] lenrk zipr, r- rS r+ r[ range tokenizertoviewappendcatrZ fullr9 repeat_interleave full_likerN sortgatherisfiniterT r1 rU zerosboolclamp_log2sumr< clamprW masked_fill_isnansigmoidrV rX logmean))r inputslabelsqueries docs_listdocsdocs_per_querymax_docs
batch_sizequerydocumentpairsr[ logits_listimini_batch_pairstokenslogits
logits_matrix doc_indices
batch_indices
labels_matrixlogits_matrix_sorted indices_predlabels_matrix_sorted_true_sorted_by_preds
true_diffspadded_pairs_maskrU ndcg_at_k_maskr? r maxDCGsr weights scores_diffsweighted_probaslosses
masked_losseslosss) r r zLambdaLoss.forward s ff % n
v;! `adekal`mmu vw w #09:#d)::~&\
?fc&k ? ? ;N;KKx U[ z\ KQz} E {F z\ y] ]_ `
7:'96Mcc{ud^bcRZ%"c"cc<<DJJ,=,=TR R ..<* a !%jO q#e*o6 'A $Q_)< = ZZ)) # * F YYtzz 0 01F TZZ)&)!,11"5F v & ' ;A.##F+
J#95IZIZ[
iiY OT c$i!8 OUVW // Z0H%,,WeJfg
4:
m[01
uV}E
49IIf!4L4R4R4T
m[01%(():):;
.;-?-?4UW-?-X* l"/"4"4""4"M a %||Mq U )!Q*58LQPTVWZ8XX
!NN: 6$// A 1Z!^ D
FF h h%9TXT^T^TeTef!"rr2A2v ###, ###,<<8a<033DJJ4E4EF::cHNN$4 45dAg>))eii+? @1 D PRSUWVWUWRW X^`agglpltltgu ! 12Q6'!T':JJ ''h8LM -Q4Z8;OPQSWYZPZ;[[bbgkqtbu !!%++l";SA ==l)BCIIdhhIW[bbiinrnvnviw *YY/FZZ0F 0> AB
=)) } ; @ z\ d@ !Ps X:X X%
X*8%X1c t | j | j | j | j | j
t | j | j dS )z
Get configuration parameters for this loss function.
Returns:
Dictionary containing the configuration parameters
)rT rU rV rW rX rZ r[ )r rT rU rV rW rX rZ r[ r s r get_config_dictzLambdaLoss.get_config_dictM sQ !))>)> ?ZZ88 ! / / %d&8&8 9 # 3 3
r c y)Na^
@inproceedings{wang2018lambdaloss,
title={The LambdaLoss Framework for Ranking Metric Optimization},
author={Wang, Xuanhui and Li, Cheng and Golbandi, Nadav and Bendersky, Michael and Najork, Marc},
booktitle={Proceedings of the 27th ACM international conference on information and knowledge management},
pages={1313--1322},
year={2018}
}
r/ r s r citationzLambdaLoss.citation^ s r )rS r rT zBaseWeightingScheme | NonerU
int | NonerV rN rW rN rX z Literal['natural', 'binary']rZ znn.Module | Noner[ r r r )r z list[list[str], list[list[str]]]r z list[Tensor]r r )r z#dict[str, float | int | str | None])r str) r r! r" rF r rY r r r propertyr r$ r% s @r rP rP g s 8I7J 6>*5"++-&*pp 5p p
p
p 4p (p $p
pdqf
" r rP )
__future__r typingr r, r r #sentence_transformers.cross_encoderr sentence_transformers.utilr Moduler
r' r1 r5 rC rF rP r/ r r r s| " < /"")) ",5+ 5 -) -S) S&
*
7+ 7&A Ar