rh*0dZddlZddlZddlmZddlmZmZddlZddl Zddlm Z ddl m Z m Z mZddlmZdd lmZmZdd lmZdd lmZdd lmZmZmZmZmZmZmZm Z m!Z!dd l"m#Z#ddl$m%Z%m&Z&m'Z'ddl(m)Z)m*Z*m+Z+ddl,m-Z-e+j\e/Z0Gdde jbZ2Gdde jbZ3Gdde jbZ4de3iZ5Gdde jbZ6Gdde jbZ7Gdde jbZ8Gdd eZ9Gd!d"e jbZ:Gd#d$e jbZ;Gd%d&e jbZ<Gd'd(e jbZ=Gd)d*e jbZ>Gd+d,e jbZ?Gd-d.e jbZ@e*Gd/d0e#ZAee*d12Gd3d4e)ZBe*d52Gd6d7eAZCe*d82Gd9d:eAZDe*d;2Gd<d=eAeZEe*Gd>d?eAZFe*d@2GdAdBeAZGe*dC2GdDdEeAZHe*GdFdGeAZIe*GdHdIeAZJe*GdJdKeAZKgdLZLy)MzPyTorch ERNIE model.N) dataclass)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)CacheEncoderDecoderCache)GenerationMixin)GradientCheckpointingLayer) )BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringlogging) ErnieConfigceZdZdZfdZ d deejdeejdeejdeejdeejde d ejfd Z xZ S) ErnieEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.ct|tj|j|j |j |_tj|j|j |_ tj|j|j |_ |j|_ |jr/tj|j|j |_tj|j |j |_tj"|j$|_t)|dd|_|j-dt/j0|jj3dd|j-d t/j4|j6j9t.j: dy) N) padding_idxepsposition_embedding_typeabsolute position_ids)r F) persistenttoken_type_idsdtype)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings use_task_idtask_type_vocab_sizetask_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutgetattrr(register_buffertorcharangeexpandzerosr*sizelongselfconfig __class__s {/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/ernie/modeling_ernie.pyr1zErnieEmbeddings.__init__6sf !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"!--   (* V5P5PRXRdRd(eD %f&8&8f>S>STzz&"<"<= '.v7PR\']$  ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  input_idsr- task_type_idsr* inputs_embedspast_key_values_lengthreturnc||j}n|jdd}|d}||jdd|||zf}|st|dr-|jddd|f} | j |d|} | }n:t j |t j|jj}||j|}|j|} || z} |jdk(r|j|} | | z } |jrR|:t j |t j|jj}|j|}| |z } |j| } |j!| } | S)Nr+r r-rr/devicer))rIr*hasattrr-rGrErHrJrXr6r:r(r8r;r=r>rB)rLrQr-rRr*rSrT input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr: embeddingsr8r=s rOforwardzErnieEmbeddings.forwardLs  #..*K',,.s3K ^  ,,Q0FVlIl0l-lmL  !t-.*.*=*=a*n*M'3J3Q3QR]^_R`bl3m0!A!&[ SWSdSdSkSk!l   00;M $ : :> J"%::  ' ': 5"&":":<"H  - -J   $ % KuzzRVRcRcRjRj k #'#<#<]#K . .J^^J/ \\*- rP)NNNNNr) __name__ __module__ __qualname____doc__r1rrE LongTensor FloatTensorintTensorr_ __classcell__rNs@rOr#r#3sQ 01559483759&'0E,,-0!!1!120  0 01 0 u//0 0   1 12 0!$0 0rPr#ceZdZd fd Z d dej deejdeejdeejdeedee deej d e ej fd Z xZ S) ErnieSelfAttentionct||j|jzdk7r2t |ds&t d|jd|jd|j|_t |j|jz |_|j|jz|_tj|j|j|_ tj|j|j|_ tj|j|j|_ tj|j|_|xs t#|dd|_|j$dk(s|j$d k(rF|j&|_tj(d |j&zd z |j|_|j,|_||_y) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r(r) relative_keyrelative_key_queryr )r0r1r4num_attention_headsrY ValueErrorrfattention_head_size all_head_sizerLinearquerykeyvaluer@attention_probs_dropout_probrBrCr(r7r2distance_embedding is_decoder layer_idxrLrMr(r}rNs rOr1zErnieSelfAttention.__init__s     : : :a ?PVXhHi#F$6$6#78 445Q8  $*#=#= #&v'9'9F( ' -zC $  ' '> 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD # ++"rP hidden_statesattention_mask head_maskencoder_hidden_statespast_key_valueoutput_attentionscache_positionrUc |j\}} } |j|} | j|d|j|jj dd} |du} |St |trA|jj|j} | r |j}n|j}n|}| r|n|}| rK|I rGj|jj}|j|jj}n|j!|}|j|d|j|jj dd}|j#|}|j|d|j|jj dd}|D| s|nd}j%|||jd|i\}}| rd|j|j<t'j(| |j dd}|j*dk(s|j*dk(r| jd|jd}}|Dt'j,|dz t&j.|j0 jdd}n@t'j2|t&j.|j0 jdd}t'j2|t&j.|j0 jdd}||z }|j5||j6zdz }|j9| j: }|j*dk(rt'j<d | |}||z}nE|j*dk(r6t'j<d | |}t'j<d ||}||z|z}|t?j@|jz }|||z}tBjDjG|d }|jI|}|||z}t'j(||}|jKddddjM}|jOdd|jPfz}|j|}||fS)Nr+r rqrTrorprWr.zbhld,lrd->bhlrzbhrd,lrd->bhlrdimrr ))shaperwviewrrrt transpose isinstancer is_updatedgetr}cross_attention_cacheself_attention_cachelayerskeysvaluesrxryupdaterEmatmulr(tensorrJrXrFr{r7tor/einsummathsqrtr functionalsoftmaxrBpermute contiguousrIru)rLrrrrrrr batch_sizer[_ query_layeris_cross_attentionrcurr_past_key_valuecurrent_states key_layer value_layerattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapes rOr_zErnieSelfAttention.forwardsN%2$7$7! Jjj/ !&&z2t7O7OQUQiQijtt q 3$>  %.*=>+66::4>>J %*8*N*N'*8*M*M'&4#2D.- ."<+224>>BGGI-44T^^DKKK0I!z2t7O7OQUQiQijtt1I**^4K%**B 8 8$:R:Ri1o )7It)<)C)C{DNN=M~<^*& ;&@DN--dnn=!<< Y5H5HR5PQ  ' '> 9T=Y=Y]q=q'2'8'8';Y__Q=O*L)!&j1nEJJWdWkWk!l!q!q""'l%**UbUiUi!j!o!oprtu!v"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||S"S%**+BC o--rPNNNNNNFN) r`rarbr1rErgrrer booltupler_rhris@rOrkrks#<7;15=A*.,115d.||d.!!2!23d.E--. d. ((9(9: d. ! d.$D>d.!.d. u|| d.rPrkcneZdZfdZdej dej dej fdZxZS)ErnieSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNr&) r0r1rrvr4denser>r?r@rArBrKs rOr1zErnieSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rPr input_tensorrUcr|j|}|j|}|j||z}|SNrrBr>rLrrs rOr_zErnieSelfOutput.forward 7 =1  ]3 }|'CD rPr`rarbr1rErgr_rhris@rOrr1> U\\RWR^R^rPreagerceZdZd fd ZdZ d dej deejdeejdeejdee dee d eej d e ej fd Z xZ S)ErnieAttentionct|t|j||||_t ||_t|_y)Nr(r}) r0r1ERNIE_SELF_ATTENTION_CLASSES_attn_implementationrLroutputset pruned_headsr~s rOr1zErnieAttention.__init__sF 01L1LM $;  &f- ErPc>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrr r)lenrrLrrrtrrrwrxryrrruunion)rLheadsindexs rO prune_headszErnieAttention.prune_heads#s u:? 7 49900$))2O2OQUQbQb u -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:rPrrrrrrrrUc r|j|||||||}|j|d|} | f|ddz} | S)Nrrrrrrrr )rLr) rLrrrrrrr self_outputsattention_outputoutputss rOr_zErnieAttention.forward5s\yy )"7)/)!  ;;|A F#% QR(88rPrr)r`rarbr1rrErgrrer rrr_rhris@rOrrs";*7;15=A*.,115||!!2!23E--.  ((9(9:  ! $D>!. u|| rPrcVeZdZfdZdej dej fdZxZS)ErnieIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r0r1rrvr4intermediate_sizerr hidden_actstrr intermediate_act_fnrKs rOr1zErnieIntermediate.__init__Os] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $rPrrUcJ|j|}|j|}|Sr)rrrLrs rOr_zErnieIntermediate.forwardWs& =1 00? rPrris@rOrrNs#9U\\ellrPrcneZdZfdZdej dej dej fdZxZS) ErnieOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r0r1rrvrr4rr>r?r@rArBrKs rOr1zErnieOutput.__init___s` YYv779K9KL f&8&8f>S>STzz&"<"<= rPrrrUcr|j|}|j|}|j||z}|Srrrs rOr_zErnieOutput.forwarderrPrris@rOrr^rrPrceZdZd fd Z ddej deejdeejdeejdeejdeedee d eej d e ej fd Z d Z xZ S) ErnieLayerclt||j|_d|_t |||_|j |_|j|_|jr-|j st|dt |d||_ t||_ t||_ y)Nr r}z> should be used as a decoder model if cross attention is addedr)r)r0r1chunk_size_feed_forward seq_len_dimr attentionr|add_cross_attentionrscrossattentionr intermediaterr)rLrMr}rNs rOr1zErnieLayer.__init__ns '-'E'E$')D ++#)#=#=  # #?? D6)g!hii"0Q[gp"qD -f5!&) rPrrrrencoder_attention_maskrrrrUc H|j||||||} | d} | dd} |jrB|@t|dstd|d|j | ||||||} | d} | | ddz} t |j |j|j| } | f| z} | S)N)rrrrrrr rz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) rr|rYrsrrfeed_forward_chunkrr)rLrrrrrrrrself_attention_outputsrrcross_attention_outputs layer_outputs rOr_zErnieLayer.forward|s"& )/)) "0" 2!4(, ??4@4!12 =dVDDD '+&9&9 5#&;-"3-':' # 7q9  7 ;;G0  # #T%A%A4CSCSUe  /G+rPcL|j|}|j||}|Sr)rr)rLrintermediate_outputrs rOrzErnieLayer.feed_forward_chunks,"//0@A{{#68HI rPr)NNNNNFN)r`rarbr1rErgrrer rrr_rrhris@rOrrms *"7;15=A>B*.,115-||-!!2!23-E--. - ((9(9: - !)):): ; -!-$D>-!.- u|| -^rPrcfeZdZdfd Z ddej deejdeejdeejdeejdeeeejdee d ee d ee d ee d eej d e eej e ffdZ xZ S) ErnieEncoderc t|||_tjt |j Dcgc]}t||c}|_d|_ ycc}w)NrF) r0r1rMr ModuleListrangenum_hidden_layersrlayergradient_checkpointing)rLrMr}irNs rOr1zErnieEncoder.__init__sQ  ]]USYSkSkMl#mJv$C#mn &+#$nsA%rrrrrpast_key_values use_cacheroutput_hidden_states return_dictrrUc | rdnd} |rdnd} |r|jjrdnd}|jr%|jr|rtj dd}d}|rR|jj rz'ErnieEncoder.forward..s = s)last_hidden_staterr attentionscross_attentions)rMrrtraininglogger warning_oncer|rr r from_legacy_cache enumeraterto_legacy_cacherr)rLrrrrrrrrrrrall_hidden_statesall_self_attentionsall_cross_attentionsreturn_legacy_cacher layer_modulelayer_head_mask layer_outputss rOr_zErnieEncoder.forwards#7BD$5b4%64;;;Z;Zr`d  & &4==##p" # // ?TY8Z   \  #' 1CCOTO(4 VOA|#$58H$H!.7.CilO(%'=."3- M*!,M &9]1=MBEI$(,1/4&*15R ||R !!2!23R E--. R ((9(9: R !)):): ; R "%e.?.?(@"ABR D>R $D>R 'tnR d^R !.R  uU\\"$MM NR rPrcVeZdZfdZdej dej fdZxZS) ErniePoolerct|tj|j|j|_tj |_yr)r0r1rrvr4rTanh activationrKs rOr1zErniePooler.__init__s9 YYv1163E3EF '')rPrrUc\|dddf}|j|}|j|}|S)Nr)rr!)rLrfirst_token_tensor pooled_outputs rOr_zErniePooler.forwards6+1a40 #56  6 rPrris@rOrrs#$ U\\ellrPrcVeZdZfdZdej dej fdZxZS)ErniePredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yr) r0r1rrvr4rrrrr transform_act_fnr>r?rKs rOr1z%ErniePredictionHeadTransform.__init__ s{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STrPrrUcl|j|}|j|}|j|}|Sr)rr(r>rs rOr_z$ErniePredictionHeadTransform.forward)s4 =1 --m< }5 rPrris@rOr&r&s$UU\\ellrPr&c*eZdZfdZdZdZxZS)ErnieLMPredictionHeadcHt|t||_t j |j |jd|_t jtj|j|_ |j|j_ y)NF)bias) r0r1r& transformrrvr4r3decoder ParameterrErHr-rKs rOr1zErnieLMPredictionHead.__init__2sm 5f=yy!3!3V5F5FUS LLV->->!?@ !II rPc:|j|j_yr)r-r/rLs rO _tie_weightsz"ErnieLMPredictionHead._tie_weights?s II rPcJ|j|}|j|}|Sr)r.r/rs rOr_zErnieLMPredictionHead.forwardBs$}5  ]3 rP)r`rarbr1r3r_rhris@rOr+r+1s &&rPr+cVeZdZfdZdej dej fdZxZS)ErnieOnlyMLMHeadcBt|t||_yr)r0r1r+ predictionsrKs rOr1zErnieOnlyMLMHead.__init__Js 08rPsequence_outputrUc(|j|}|Sr)r8)rLr9prediction_scoress rOr_zErnieOnlyMLMHead.forwardNs ,,_=  rPrris@rOr6r6Is#9!u||! !rPr6c$eZdZfdZdZxZS)ErnieOnlyNSPHeadclt|tj|jd|_yNrq)r0r1rrvr4seq_relationshiprKs rOr1zErnieOnlyNSPHead.__init__Us'  " &*<*fill_)rLmodules rO _init_weightsz"ErniePreTrainedModel._init_weightsqs fbii ( MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S ).rPN)r`rarbr!__annotations__base_model_prefixsupports_gradient_checkpointingrUrrPrOrIrIks &*#*rPrIz1 Output type of [`ErnieForPreTraining`]. ) custom_introceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ ee ejed<dZ ee ejed<y)ErnieForPreTrainingOutputa loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). Nlossprediction_logitsseq_relationship_logitsrr)r`rarbrcr\rrErerVr]r^rrrrrPrOr[r[s~ )-D(5$$ %,59x 1 129;?Xe&7&78?8k&)D  rPc.|jjSrr^r6r2s rOget_input_embeddingszErnieModel.get_input_embeddingss...rPc&||j_yrrg)rLrys rOset_input_embeddingszErnieModel.set_input_embeddingss*/'rPc|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrbrrr)rLheads_to_prunerrs rO _prune_headszErnieModel._prune_headssE +002 CLE5 LL  u % / / ; ;E B CrPrQrr-rRr*rrSrrrrrrrrUc | | n|jj} | | n|jj} ||n|jj}|jjr| | n|jj } nd} | | t d|#|j|||j}n!||jdd}n t d|\}}| |jn |j}d}| 5t| ts| ddjdn| j}|tj|||zf|}|pt!|j"d r4|j"j$ddd|f}|j'||}|}n&tj(|tj*| }|j-||}|jjrE|C|j\}}}||f}| tj||} |j/| }nd}|j1||jj2}|j#|||||| }|j5|||||| | | | | }|d}|j6|j7|nd}|s ||f|d dzSt9|||j:|j<|j>|j@S) task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] NFzDYou cannot specify both input_ids and inputs_embeds at the same timer+z5You have to specify either input_ids or inputs_embedsrr)rXr-rW)rQr*r-rRrSrT) rrrrrrrrrr )r  pooler_outputrrrr)!rMrruse_return_dictr|rrs%warn_if_padding_and_no_attention_maskrIrXrr rget_seq_lengthrEonesrYr^r-rGrHrJget_extended_attention_maskinvert_attention_mask get_head_maskrrbrcrrrrr) rLrQrr-rRr*rrSrrrrrrrrZrr[rXrTr\r]extended_attention_maskencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapeencoder_extended_attention_maskembedding_outputencoder_outputsr9r$s rOr_zErnieModel.forwards422C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U!, J%.%:!! @T@T!"  &"/59 "1%++B/$335 #  !"ZZ*jCY6Y)ZdjkN  !t(89*.//*H*HKZK*X'3J3Q3QR\^h3i0!A!&[ SY!Z150P0PQ_al0m ;; ! !&;&G=R=W=W=Y :  7$68O#P %-).4HQW)X&.2.H.HI_.` +.2 +&&y$++2O2OP ??%)''#9 + ,, 2"7#B+/!5#'  *!,8< 8O O4UY #]3oab6II I;-'+;;)77&11,==   rP)T)NNNNNNNNNNNNNN)r`rarbr1rhrjrnrrrErglistrerrrrr_rhris@rOr`r`s"/0C-1151504/3,0048<9==A$(,0/3&*z ELL)z !.z !. z   - z u||, z ELL)z   -z  ( 5z !) 6z "$u'8'8"9:z D>z $D>z 'tnz d^z uU\\"$PP Q!z z rPr`z Ernie Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `next sentence prediction (classification)` head. ceZdZddgZfdZdZdZe ddee jdee jdee jd ee jd ee jd ee jd ee jd ee jdee jdee dee dee de e e jeffdZxZS)ErnieForPreTrainingcls.predictions.decoder.biascls.predictions.decoder.weightct||t||_t ||_|j yr)r0r1r`rJrEclsrdrKs rOr1zErnieForPreTraining.__init__Rs4  ' (0 rPcB|jjjSrrr8r/r2s rOget_output_embeddingsz)ErnieForPreTraining.get_output_embeddings\xx##+++rPc||jj_|j|jj_yrrr8r/r-rLnew_embeddingss rOset_output_embeddingsz)ErnieForPreTraining.set_output_embeddings`,'5$$2$7$7!rPrQrr-rRr*rrSlabelsnext_sentence_labelrrrrUc  | | n|jj} |j|||||||| | |  } | dd\}}|j||\}}d}|u| st }||j d|jj |j d}||j dd| j d}||z}| s||f| ddz}||f|zS|St|||| j| jS)a: task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, ErnieForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> model = ErnieForPreTraining.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ``` N rr-rRr*rrSrrrrqr+)r\r]r^rr) rMrrrJrrrr3r[rr)rLrQrr-rRr*rrSrrrrrrr9r$r;rB total_lossloss_fctmasked_lm_lossnext_sentence_lossrs rOr_zErnieForPreTraining.forwarddsJb&1%S 'tnS d^S  uU\\"$== >S S rPrzQ Ernie Model with a `language modeling` head on top for CLM fine-tuning. c$eZdZddgZfdZdZdZe ddee jdee jdee jd ee jd ee jd ee jd ee jd ee jdee jdee jdee e jdee dee dee dee de ee jeff dZxZS)ErnieForCausalLMrrct|||jstj dt |d|_t||_|jy)NzMIf you want to use `ErnieForCausalLM` as a standalone, add `is_decoder=True.`Fre r0r1r|rwarningr`rJr6rrdrKs rOr1zErnieForCausalLM.__init__sL    NNj k%@ #F+ rPcB|jjjSrrr2s rOrz&ErnieForCausalLM.get_output_embeddingsrrPc||jj_|j|jj_yrrrs rOrz&ErnieForCausalLM.set_output_embeddingsrrPrQrr-rRr*rrSrrrrrrrrrUc ||n|jj}| d} |j||||||||| | | | ||}|d}|j|}d}| *|j|| fd|jj i|}|s|f|ddz}||f|zS|St |||j|j|j|jS)a task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` NF) rr-rRr*rrSrrrrrrrrr3rq)r\logitsrrrr) rMrrrJr loss_functionr3rrrrr)rLrQrr-rRr*rrSrrrrrrrrkwargsrr9r;lm_lossrs rOr_zErnieForCausalLM.forwards#>&1%J $D>J 'tnJ d^!J $ uU\\"$EE F%J J rPrc eZdZddgZfdZdZdZe ddee jdee jdee jd ee jd ee jd ee jd ee jd ee jdee jdee jdee dee dee de e e jeffdZddZede fdZxZS)ErnieForMaskedLMrrct|||jrtj dt |d|_t||_|jy)NzlIf you want to use `ErnieForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.FrrrKs rOr1zErnieForMaskedLM.__init__,sR     NN1   %@ #F+ rPcB|jjjSrrr2s rOrz&ErnieForMaskedLM.get_output_embeddings<rrPc||jj_|j|jj_yrrrs rOrz&ErnieForMaskedLM.set_output_embeddings@rrPrQrr-rRr*rrSrrrrrrrUc| | n|jj} |j||||||||| | | |  }|d}|j|}d}| Ft }||j d|jj | j d}| s|f|ddz}||f|zS|St|||j|jS)as task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` N) rr-rRr*rrSrrrrrrr+rqr\rrr) rMrrrJrrrr3rrr)rLrQrr-rRr*rrSrrrrrrrr9r;rrrs rOr_zErnieForMaskedLM.forwardDs:&1% &.IIrPcy)z Legacy correction: ErnieForMaskedLM can't call `generate()` from `GenerationMixin`, even though it has a `prepare_inputs_for_generation` method. Fr)rs rO can_generatezErnieForMaskedLM.can_generates rP) NNNNNNNNNNNNNr)r`rarbrr1rrrrrErgrrrrr_r classmethodrrhris@rOrr's8:Z[  ,8-1151504/3,0048<9=)-,0/3&*> ELL)> !.> !. >   - > u||, > ELL)>   ->  ( 5> !) 6> &> $D>> 'tn> d^>  uU\\"N2 3> > BJ TrPrzU Ernie Model with a `next sentence prediction (classification)` head on top. c~eZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eejd eed eed eed e e eje ffdZ xZ S)ErnieForNextSentencePredictionct||t||_t ||_|j yr)r0r1r`rJr=rrdrKs rOr1z'ErnieForNextSentencePrediction.__init__s4  ' #F+ rPrQrr-rRr*rrSrrrrrUc d| vr+tjdt| jd}| | n|jj } |j |||||||| | |  } | d}|j|}d}|2t}||jdd|jd}| s|f| ddz}||f|zS|St||| j| jS) a task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring). Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, ErnieForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> model = ErnieForNextSentencePrediction.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> logits = outputs.logits >>> assert logits[0, 0] < logits[0, 1] # next sentence was random ``` rzoThe `next_sentence_label` argument is deprecated and will be removed in a future version, use `labels` instead.Nrr r+rqr) warningswarn FutureWarningpoprMrrrJrrrrrr)rLrQrr-rRr*rrSrrrrrrr$seq_relationship_scoresrrrs rOr_z&ErnieForNextSentencePrediction.forwards ` !F * MM%  ZZ 56F%0%X 'tnX d^X  uU\\"$?? @X X rPrz Ernie Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c~eZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eejd eed eed eed e e eje ffdZ xZ S)ErnieForSequenceClassificationcnt|||j|_||_t ||_|j |j n |j}tj||_ tj|j|j|_ |jyr)r0r1 num_labelsrMr`rJclassifier_dropoutrArr@rBrvr4 classifierrdrLrMrrNs rOr1z'ErnieForSequenceClassification.__init__s   ++ ' )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rPrQrr-rRr*rrSrrrrrUc B| | n|jj} |j|||||||| | |  } | d} |j| } |j | }d}||jj |j dk(rd|j_nl|j dkDrL|jtjk(s|jtjk(rd|j_nd|j_|jj dk(rIt}|j dk(r&||j|j}n|||}n|jj dk(r=t}||jd|j |jd}n,|jj dk(rt}|||}| s|f| ddz}||f|zS|St!||| j"| j$ S) a^ task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrr regressionsingle_label_classificationmulti_label_classificationr+rqr)rMrrrJrBr problem_typerr/rErJrfr squeezerrrrrr)rLrQrr-rRr*rrSrrrrrr$rr\rrs rOr_z&ErnieForSequenceClassification.forward"s4&1%L 'tnL d^L  uU\\"$<< =L L rPrc~eZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eejd eed eed eed e e eje ffdZ xZ S)ErnieForMultipleChoicec*t||t||_|j |jn |j }t j||_t j|jd|_ |jy)Nr ) r0r1r`rJrrArr@rBrvr4rrdrs rOr1zErnieForMultipleChoice.__init__usu  ' )/)B)B)NF % %TZTnTn zz"45 ))F$6$6: rPrQrr-rRr*rrSrrrrrUc N| | n|jj} ||jdn|jd} |!|jd|j dnd}|!|jd|j dnd}|!|jd|j dnd}|!|jd|j dnd}|1|jd|j d|j dnd}|j |||||||| | |  } | d}|j |}|j|}|jd| }d}|t}|||}| s|f| ddz}||f|zS|St||| j| jS)a9 input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) task_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nr r+rrrqr) rMrrrrrIrJrBrrrrr)rLrQrr-rRr*rrSrrrr num_choicesrr$rreshaped_logitsr\rrs rOr_zErnieForMultipleChoice.forwardsd&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  ** ))'%'/!5#     ]3 / ++b+6  ')HOV4D%''!"+5F)-)9TGf$ Ev E("!//))   rPr)r`rarbr1rrrErgrrrrr_rhris@rOrrrs+ -1151504/3,004)-,0/3&*_ ELL)_ !._ !. _   - _ u||, _ ELL)_   -_ &_ $D>_ 'tn_ d^_  uU\\"$== >_ _ rPrc~eZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eejd eed eed eed e e eje ffdZ xZ S)ErnieForTokenClassificationcdt|||j|_t|d|_|j |j n |j }tj||_ tj|j|j|_ |jyNFr)r0r1rr`rJrrArr@rBrvr4rrdrs rOr1z$ErnieForTokenClassification.__init__s   ++%@ )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rPrQrr-rRr*rrSrrrrrUc | | n|jj} |j|||||||| | |  } | d} |j| } |j | }d}|9 'tn9 d^9  uU\\"$99 :9 9 rPrceZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eejd eejd eed eed eede e eje ffdZ xZ S)ErnieForQuestionAnsweringct|||j|_t|d|_t j |j|j|_|jyr) r0r1rr`rJrrvr4 qa_outputsrdrKs rOr1z"ErnieForQuestionAnswering.__init__6sU   ++%@ ))F$6$68I8IJ rPrQrr-rRr*rrSstart_positions end_positionsrrrrUc *| | n|jj} |j|||||||| | |  } | d}|j|}|j dd\}}|j dj }|j dj }d}|| t|jdkDr|j d}t| jdkDr| j d} |jd}|jd|}| jd|} t|}|||}||| }||zdz }| s||f| ddz}||f|zS|St|||| j| j S) rpNrrr r+r) ignore_indexrq)r\ start_logits end_logitsrr)rMrrrJrsplitrrrrIclamprrrr)rLrQrr-rRr*rrSrrrrrrr9rrrr ignored_indexr start_lossend_lossrs rOr_z!ErnieForQuestionAnswering.forward@s.&1%G 'tnG d^G  uU\\"$@@ AG G rPr) rrrrrrrrr`rI)Mrcrr dataclassesrtypingrrrEtorch.utils.checkpointrtorch.nnrrr activationsr cache_utilsr r generationrmodeling_layersrmodeling_outputsrrrrrrrrrmodeling_utilsr pytorch_utilsrrrutilsrrrconfiguration_ernier! get_loggerr`rModuler#rkrrrrrrrrr&r+r6r=rErIr[r`rrrrrrrr__all__rrPrOrs !" AA!5)9   .ll99,   H %IbiiIZ@.@.H bii    2RYY2l   ")) A+AJY 299Y z "))  299$BII0!ryy!&ryy& 9BII 9*?**,  : :  :& ^ %^  ^ B j .j  j Z d +_d  d Nt+ttn d %9d  d N ^ %9^  ^ Bo 1o o dJ "6J J ZT 4T T n rP