rh*YldZddlmZddlmZmZddlZddlmZmZddl m Z ddl m Z dd l mZmZmZd d lmZd d lmZej*eZeedGddeZeedGddeZeedGddeZeGdde ZGddeZGddeZGddeZGddeZGd d!eZ ed"Gd#d$eZ!ed%Gd&d'eZ"ed(Gd)d*e Z#gd+Z$y),z5PyTorch DPR model for Open Domain Question Answering.) dataclass)OptionalUnionN)Tensornn)BaseModelOutputWithPooling)PreTrainedModel) ModelOutputauto_docstringlogging) BertModel) DPRConfigz6 Class for outputs of [`DPRQuestionEncoder`]. ) custom_introceZdZUdZej ed<dZee ej dfed<dZ ee ej dfed<y)DPRContextEncoderOutputa pooler_output (`torch.FloatTensor` of shape `(batch_size, embeddings_size)`): The DPR encoder outputs the *pooler_output* that corresponds to the context representation. Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer. This output is to be used to embed contexts for nearest neighbors queries with questions embeddings. pooler_outputN. hidden_states attentions __name__ __module__ __qualname____doc__torch FloatTensor__annotations__rrtuplerw/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/dpr/modeling_dpr.pyrr*S$$$=AM8E%"3"3S"89:A:>Ju00#567>r"rceZdZUdZej ed<dZee ej dfed<dZ ee ej dfed<y)DPRQuestionEncoderOutputa pooler_output (`torch.FloatTensor` of shape `(batch_size, embeddings_size)`): The DPR encoder outputs the *pooler_output* that corresponds to the question representation. Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer. This output is to be used to embed questions for nearest neighbors queries with context embeddings. rN.rrrr!r"r#r&r&=r$r"r&ceZdZUdZej ed<dZeej ed<dZ eej ed<dZ ee ej dfed<dZ ee ej dfed<y) DPRReaderOutputa start_logits (`torch.FloatTensor` of shape `(n_passages, sequence_length)`): Logits of the start index of the span for each passage. end_logits (`torch.FloatTensor` of shape `(n_passages, sequence_length)`): Logits of the end index of the span for each passage. relevance_logits (`torch.FloatTensor` of shape `(n_passages, )`): Outputs of the QA classifier of the DPRReader that corresponds to the scores of each passage to answer the question, compared to all the other passages. start_logitsN end_logitsrelevance_logits.rr) rrrrrrrr*rr+rr rr!r"r#r(r(Ps~###.2J**+248hu0018=AM8E%"3"3S"89:A:>Ju00#567>r"r(ceZdZdZdZy)DPRPreTrainedModelTct|tjrm|jjj d|j j|j%|jjjyyt|tjrz|jjj d|j j|j2|jj|jjyyt|tjrJ|jjj|jjjdyy)zInitialize the weightsg)meanstdNg?) isinstancerLinearweightdatanormal_configinitializer_rangebiaszero_ Embedding padding_idx LayerNormfill_)selfmodules r# _init_weightsz DPRPreTrainedModel._init_weightsls fbii ( MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S ).r"N)rrr_supports_sdpar@r!r"r#r-r-hs N*r"r-ceZdZdZdeffd Z ddedeedeedeeded ed ed e e e ed fffd Z e d efdZxZS) DPREncoder bert_modelr6ct||t|d|_|jjj dkr t d|j|_|jdkDrCtj|jjj |j|_ |jy)NF)add_pooling_layerrz!Encoder hidden_size can't be zero) super__init__rrDr6 hidden_size ValueErrorprojection_dimrr2 encode_proj post_initr>r6 __class__s r#rHzDPREncoder.__init__s  #FeD ?? ! ! - - 2@A A$33    "!yy)?)?)K)KVMbMbcD  r" input_idsattention_masktoken_type_ids inputs_embedsoutput_attentionsoutput_hidden_states return_dictreturn.c |j|||||||}|d} | dddddf} |jdkDr|j| } |s | | f|ddzSt| | |j|j S)NrPrQrRrSrTrUrVrr)last_hidden_staterrr)rDrKrLr rr) r>rPrQrRrSrTrUrVoutputssequence_output pooled_outputs r#forwardzDPREncoder.forwards//))'/!5#" "!*'1a0    " ,,];M#]3gabkA A)-'!//))   r"c|jdkDr|jjS|jjj S)Nr)rKrL out_featuresrDr6rI)r>s r#embeddings_sizezDPREncoder.embeddings_sizes8    "##00 0%%111r")NNNFFF)rrrbase_model_prefixrrHrrboolrr r r^propertyintra __classcell__rOs@r#rCrC}s$ y ,0+/*."'%*!! ! !(! !( !  ' !  ! #! !  )5+== >! F222r"rCcpeZdZdZdeffd Z d dededeededed ed e e e ed fffd Z xZ S)DPRSpanPredictorencoderr6c t||t||_t j |jj d|_t j |jj d|_|jy)Nrr) rGrHrCrjrr2ra qa_outputs qa_classifierrMrNs r#rHzDPRSpanPredictor.__init__s_  !&) ))DLL$@$@!DYYt||'C'CQG r"rPrQrSrTrUrVrW.cV||jn|jdd\}}|j||||||} | d} |j| } | jdd\} } | j dj } | j dj } |j | dddddf}| j||} | j||} |j|}|s | | |f| ddzSt| | || j| jS)Nr)rQrSrTrUrVrr)dim)r)r*r+rr) sizerjrlsplitsqueeze contiguousrmviewr(rr)r>rPrQrSrTrUrV n_passagessequence_lengthr[r\logitsr)r*r+s r#r^zDPRSpanPredictor.forwardsN;D:Oinn&6UbUgUgUijlklUm# O,, )'/!5#  "!*1#)<<r<#: j#++B/::< ''+668 --oaAg.FG$((_E __ZA +00< *.>?'!"+M M%!-!//))   r")NFFF)rrrrbrrHrrrcrr(r r^rfrgs@r#riris!y+/"'%*!+ + +  ' +  + # + +  fck 22 3+ r"ric$eZdZUdZeed<dZdZy)DPRPretrainedContextEncoder An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. r6N ctx_encoderrrrrrrload_tf_weightsrbr!r"r#rzrzs O%r"rzc$eZdZUdZeed<dZdZy)DPRPretrainedQuestionEncoderr{r6Nquestion_encoderr}r!r"r#rrs O*r"rc$eZdZUdZeed<dZdZy)DPRPretrainedReaderr{r6Nspan_predictorr}r!r"r#rr s O(r"rzf The bare DPRContextEncoder transformer outputting pooler outputs as context representations. ceZdZdeffd Ze d deedeedeedeedeedeed eed e e e ed fffd Z xZ S)DPRContextEncoderr6crt||||_t||_|j yN)rGrHr6rCr|rMrNs r#rHzDPRContextEncoder.__init__s.   %f- r"rPrQrRrSrTrUrVrW.c ||n|jj}||n|jj}||n|jj}| | t d||j }n!||j dd}n t d| |j n |j } |2|tj|| n||jjk7}|&tj|tj| }|j|||||||} |s| ddSt| j| j| j S) aS input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. To match pretraining, DPR input sequence should be formatted with [CLS] and [SEP] tokens as follows: (a) For sequence pairs (for a pair title+text for example): ``` tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP] token_type_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1 ``` (b) For single sequences (for a question for example): ``` tokens: [CLS] the dog is hairy . [SEP] token_type_ids: 0 0 0 0 0 0 0 ``` DPR is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Examples: ```python >>> from transformers import DPRContextEncoder, DPRContextEncoderTokenizer >>> tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base") >>> model = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base") >>> input_ids = tokenizer("Hello, is my dog cute ?", return_tensors="pt")["input_ids"] >>> embeddings = model(input_ids).pooler_output ```NDYou cannot specify both input_ids and inputs_embeds at the same timero5You have to specify either input_ids or inputs_embedsdevicedtyperrYrrrr)r6rTrUuse_return_dictrJrqrrones pad_token_idzeroslongr|rrrr r>rPrQrRrSrTrUrV input_shaperr[s r#r^zDPRContextEncoder.forward&spb2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "#..*K  &',,.s3KTU U%.%:!! @T@T  !$ ;v64;;#;#;;   !"[[EJJvVN""))'/!5## 12; &!//w?T?Tahasas  r"NNNNNNN)rrrrrHr rrrcrrr r^rfrgs@r#rrs y'++/+/*.,0/3&*X F#X !(X !( X  ' X $D> X 'tnX d^X  &fck(:: ;X X r"rzh The bare DPRQuestionEncoder transformer outputting pooler outputs as question representations. ceZdZdeffd Ze d deedeedeedeedeedeed eed e e e ed fffd Z xZ S)DPRQuestionEncoderr6crt||||_t||_|j yr)rGrHr6rCrrMrNs r#rHzDPRQuestionEncoder.__init__s.   *6 2 r"rPrQrRrSrTrUrVrW.c ||n|jj}||n|jj}||n|jj}| | t d|#|j |||j }n!||j dd}n t d| |jn |j} |2|tj|| n||jjk7}|&tj|tj| }|j|||||||} |s| ddSt| j| j | j" S) aj input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. To match pretraining, DPR input sequence should be formatted with [CLS] and [SEP] tokens as follows: (a) For sequence pairs (for a pair title+text for example): ``` tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP] token_type_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1 ``` (b) For single sequences (for a question for example): ``` tokens: [CLS] the dog is hairy . [SEP] token_type_ids: 0 0 0 0 0 0 0 ``` DPR is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Examples: ```python >>> from transformers import DPRQuestionEncoder, DPRQuestionEncoderTokenizer >>> tokenizer = DPRQuestionEncoderTokenizer.from_pretrained("facebook/dpr-question_encoder-single-nq-base") >>> model = DPRQuestionEncoder.from_pretrained("facebook/dpr-question_encoder-single-nq-base") >>> input_ids = tokenizer("Hello, is my dog cute ?", return_tensors="pt")["input_ids"] >>> embeddings = model(input_ids).pooler_output ``` NrrorrrrYrr)r6rTrUrrJ%warn_if_padding_and_no_attention_maskrqrrrrrrrr&rrrrs r#r^zDPRQuestionEncoder.forwardsb2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U%.%:!! @T@T  !$ ;v64;;#;#;;   !"[[EJJvVN''))'/!5#( 12; '!//w?T?Tahasas  r"r)rrrrrHr rrrcrr&r r^rfrgs@r#rrs y'++/+/*.,0/3&*Y F#Y !(Y !( Y  ' Y $D> Y 'tnY d^Y  'vs{);; <Y Y r"rzE The bare DPRReader transformer outputting span predictions. ceZdZdeffd Ze d deedeedeedeedeedeed e e e ed fffd Z xZ S) DPRReaderr6crt||||_t||_|j yr)rGrHr6rirrMrNs r#rHzDPRReader.__init__s.   .v6 r"rPrQrSrTrUrVrW.c||n|jj}||n|jj}||n|jj}| | t d|#|j |||j }n!||j dd}n t d| |jn |j}|tj||}|j||||||S)a input_ids (`tuple[torch.LongTensor]` of shapes `(n_passages, sequence_length)`): Indices of input sequence tokens in the vocabulary. It has to be a sequence triplet with 1) the question and 2) the passages titles and 3) the passages texts To match pretraining, DPR `input_ids` sequence should be formatted with [CLS] and [SEP] with the format: `[CLS] [SEP] [SEP] ` DPR is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left. Indices can be obtained using [`DPRReaderTokenizer`]. See this class documentation for more details. [What are input IDs?](../glossary#input-ids) inputs_embeds (`torch.FloatTensor` of shape `(n_passages, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Examples: ```python >>> from transformers import DPRReader, DPRReaderTokenizer >>> tokenizer = DPRReaderTokenizer.from_pretrained("facebook/dpr-reader-single-nq-base") >>> model = DPRReader.from_pretrained("facebook/dpr-reader-single-nq-base") >>> encoded_inputs = tokenizer( ... questions=["What is love ?"], ... titles=["Haddaway"], ... texts=["'What Is Love' is a song recorded by the artist Haddaway"], ... return_tensors="pt", ... ) >>> outputs = model(**encoded_inputs) >>> start_logits = outputs.start_logits >>> end_logits = outputs.end_logits >>> relevance_logits = outputs.relevance_logits ``` Nrrorr)rSrTrUrV) r6rTrUrrJrrqrrrr) r>rPrQrSrTrUrVrrs r#r^zDPRReader.forwards `2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U%.%:!! @T@T  !"ZZ FCN""  '/!5# #  r")NNNNNN)rrrrrHr rrrcrr(r r^rfrgs@r#rrs y'++/*.,0/3&*K F#K !(K  ' K $D> K 'tn K d^K  fck 22 3K K r"r)rrzr-rrrr)%r dataclassesrtypingrrrrrmodeling_outputsr modeling_utilsr utilsr r r bert.modeling_bertrconfiguration_dprr get_loggerrloggerrr&r(r-rCrirzrrrrr__all__r!r"r#rs<!" :- +(   H %  ?k ?  ?  ?{ ?  ?  ?k?  ?$***(52#52p6 )6 |&"4&+#5+),)  a 3a  a H b 5b  b J T #T  T n r"