rhbdZddlZddlZddlmZmZddlZddlmZddlm Z m Z m Z ddl m Z ddlmZmZmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZmZmZdd l m!Z!ddl"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)ddl*m+Z+e&r ddl,m-Z-ddl.m/Z/e)j`e1Z2GddejfZ4GddejfZ5GddejfZ6GddejfZ7GddejfZ8GddejfZ9Gdd ejfZ:Gd!d"eZ;Gd#d$ejfZe%Gd)d*e=Z?e%d+,Gd-d.e=eZ@e%Gd/d0e=ZAe%d1,Gd2d3e=ZBe%Gd4d5e=ZCe%Gd6d7e=ZDgd8ZEy)9zPyTorch UMT5 model.N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)AttentionMaskConverter)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) DUMMY_INPUTS DUMMY_MASKauto_docstringis_torch_flex_attn_availableis_torch_fx_proxyis_torchdynamo_compilinglogging) UMT5Config) BlockMask)make_flex_block_causal_maskc&eZdZdfd ZdZxZS) UMT5LayerNormct|tjt j ||_||_y)ze Construct a layernorm module in the UMT5 style. No bias and no subtraction of mean. N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s y/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/umt5/modeling_umt5.pyr(zUMT5LayerNorm.__init__>s1 ll5::k#:; #c|jtjjdj dd}|tj ||j zz}|jjtjtjfvr%|j|jj}|j|zS)NT)keepdim) tor*float32powmeanrsqrtr-r,dtypefloat16bfloat16)r. hidden_statesvariances r2forwardzUMT5LayerNorm.forwardFs !##EMM266q9>>r4>P% Ht?T?T4T(UU  ;;   ? ?),,T[[->->?M{{]**r3)gư>)__name__ __module__ __qualname__r(rB __classcell__r1s@r2r%r%=s $ +r3r%c*eZdZdeffd ZdZxZS)UMT5DenseActDenseconfigc^t|tj|j|j d|_tj|j |jd|_tj|j|_ t|j|_ yNFbias)r'r(rLineard_modeld_ffwiwoDropout dropout_ratedropoutr dense_act_fnactr.rJr1s r2r(zUMT5DenseActDense.__init__Xsn ))FNNFKKeD))FKKeDzz&"5"56 &--.r3c|j|}|j|}|j|}t|jj t jr|j|jj jk7r`|jj jt jk7r/|j|jj j}|j |}|SN) rRrXrV isinstancerSr,r*Tensorr=int8r8r.r@s r2rBzUMT5DenseActDense.forward_s . /  ]3 tww~~u|| 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r3rCrDrEr!r(rBrFrGs@r2rIrIWs/z/ r3rIc*eZdZdeffd ZdZxZS)UMT5DenseGatedActDenserJct|tj|j|j d|_tj|j|j d|_tj|j |jd|_tj|j|_ t|j|_yrL)r'r(rrOrPrQwi_0wi_1rSrTrUrVr rWrXrYs r2r(zUMT5DenseGatedActDense.__init__os IIfnnfkkF IIfnnfkkF ))FKKeDzz&"5"56 &--.r3c,|j|j|}|j|}||z}|j|}t |j j tjr|j|j j jk7r`|j j jtjk7r/|j|j j j}|j |}|Sr[) rXrdrerVr\rSr,r*r]r=r^r8)r.r@ hidden_gelu hidden_linears r2rBzUMT5DenseGatedActDense.forwardwshhtyy78  -0 #m3  ]3 tww~~u|| 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r3r`rGs@r2rbrbns/z/r3rbc*eZdZdeffd ZdZxZS) UMT5LayerFFrJct||jrt||_nt ||_t |j|j|_ tj|j|_ y)Nr0)r'r( is_gated_actrbDenseReluDenserIr%rPlayer_norm_epsilon layer_normrrTrUrVrYs r2r(zUMT5LayerFF.__init__s_    "8"@D "3F";D 'FeZdZdZddeeffd ZdejdejfdZ dZ ddZ dd ejd eejd ee ejd eejd eejdeejf dZ xZS) UMT5Attentionz7 T5's attention using relative_attention_bias. layer_idxct||j|_||_|j|_|j |_|j |_|j|_|j|_ |j|_ |j|jz|_ ||_|9|jr-tj!d|j"j$dt'j(|j |jd|_t'j(|j |jd|_t'j(|j |jd|_t'j(|j|j d|_|jr/t'j2|j|j|_t7|_y)NzInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.FrM)r'r( is_decoderhas_relative_attention_biasrelative_attention_num_bucketsrelative_attention_max_distancerPd_kvkey_value_proj_dim num_headsn_headsrUrV inner_dimrulogger warning_oncer1rCrrOqkvo Embeddingrelative_attention_biasset pruned_heads)r.rJrxrur1s r2r(zUMT5Attention.__init__si  +++F(.4.S.S+/5/U/U,~~ "(++'' ** (?(??"     *4>>+B+B*CD,, 4<<eD4<<eD4<<eD4>>4<FFq!QPQRr3cd}|j}|j}|jsC|dz}||dkDjtj |zz }t j |}n*t j|t j| }|dz}||k}t j|j|z tj||z z }|||z z}||jtj z}t j|t j||dz }|t j|||z }|S)a Adapted from Mesh Tensorflow: https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transformer/transformer_layers.py#L593 Translate relative position to a bucket number for relative attention. The relative position is defined as memory_position - query_position, i.e. the distance in tokens from the attending position to the attended-to position. If bidirectional=False, then positive relative positions are invalid. We use smaller buckets for small absolute relative_position and larger buckets for larger absolute relative_positions. All relative positions >=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rr5r )ryrzrwr8r*longabsmin zeros_likelogfloatmath full_likewhere) r.relative_positionrelative_buckets num_buckets max_distance max_exactis_small log_ratiorelative_position_if_larges r2_relative_position_bucketz'UMT5Attention._relative_position_bucketsA*99 ;;  A K !2Q!6 : :5:: F T T  % *; < !&+N>NO`>a!b b  1$ $y0II/557)CDtxxP\_hPhGii y!89 %.ejj1I%I"%*YY &8RT_bcTc(d& " EKK2CE_``r3c| |jjj}|.tj|tj |dddf}n |dddf}tj|tj |dddf}||z }|j |}|j|} | jgdjd} | S)z%Compute binned relative position biasN)r=device)r5rr r) rr,rr*arangerrr unsqueeze) r. query_length key_lengthrcache_positioncontext_positionmemory_positionrrelative_position_bucketvaluess r2 compute_biaszUMT5Attention.compute_biass >1188??F  !$||L SYZ[\^b[bc -ag6 ,,zFSTXZ[T[\+.>>#'#A#ABS#T --.FG *44Q7 r3r@encoder_hidden_statespast_key_valueattention_masklayer_head_maskrc|jdd\}}|du} |j|} | j|d|j|jj dd} |Qt |trA|jj|j} | r |j} n|j} n|} | r|n|} | rK|I rG| j|jj}| j|jj}n|j!| }|j#| }|j|d|j|jj dd}|j|d|j|jj dd}|D| s|nd}| j%|||jd|i\}}| rd|j|j<t'j(| |j dd}|||j+zn|}|jd}|j,s;t'j.d|j||f|j0|j2}n1|j5|||j0| }|dddd| dddf}|#|ddddddd|jdf}||z}|j6rRt'j8|jd}d |t;|j6<|dd|j=f}n|}||z }t>j@jC|jEd jG|}t>j@jI||jH|jJ }|||z}t'j(||}|j ddjM}|j||d}|jO|}||fS) Nr5r6r rTr )rr=)rrrdim)ptraining)(shaperrr~r| transposer\r is_updatedgetrucross_attention_cacheself_attention_cachelayerskeysrrrupdater*matmulget_seq_lengthrxzerosrr=rrr+listboolr functionalsoftmaxrtype_asrVr contiguousr)r.r@rrrrr batch_size seq_lengthis_cross_attention query_statesrcurr_past_key_valuecurrent_states key_states value_statesscoresreal_seq_lengthr position_bias causal_maskmaskposition_bias_masked attn_weights attn_outputs r2rBzUMT5Attention.forwards"/!4!4Ra!8 J3$>vvm, #((RtG^G^_iijkmno   %*^EX*Y'2266t~~FJ!&4&J&J#&4&I&I#"0 2D.- ."<,33DNNCHHJ.55dnnELLL/J66.1L#RtG^G^_iijkmnoJ',,ZT\\4KbKbcmmnoqrsL)7It+>+E+E dnn?OQ_>`,( L&@DN--dnn=lJ,@,@A,FGKYJd*~'D'D'FFjt%%b) //!KKDLL*j9&--W]WcWcM!--FMMR`.M*!Q a*?@M  %(Aq2HJ4D4DR4H2H)HIK)K7M   ::m11!45D,-Dd''( )#0DIIK#@ #0 &&}},,V\\^,DLLVT }},,\T\\TXTaTa,b   &'/9Lll<> !++Aq1<<> !&&z:rB ff[) L((r3)FN)NNNNNNN)rCrDrE__doc__rintr(r*r]rrrtuplerBrFrGs@r2rtrts"XVY]":%,, - ^ $9=8<152615\)||\) ( 5\)!u||!45 \) !. \) "%,,/ \)!.\)r3rtc<eZdZddeeffd Z ddZxZS)UMT5LayerSelfAttentionruct|t|d||_t |j |j |_tj|j|_ y)NTrxrurl) r'r(rt SelfAttentionr%rProrprrTrUrVr.rJrur1s r2r(zUMT5LayerSelfAttention.__init__dsN *6t_hi'FeZdZddeeffd Z ddZxZS)UMT5LayerCrossAttentionruct|t|d||_t |j |j |_tj|j|_ y)NFrrl) r'r(rtEncDecAttentionr%rProrprrTrUrVrs r2r(z UMT5LayerCrossAttention.__init__sO ,VQVbkl'Ffinforrisinfanyclamprw)r.r@rrencoder_attention_maskrcross_attn_layer_head_maskr use_cacheoutput_attentionsrself_attn_weights max_dtype clamp_valuecross_attn_weightsdo_cross_attentionrs r2rBzUMT5Block.forwards,94::a= )+)) , ( (   %-- / M$7$78< >Gr3r[) NNNNNNFFNrrGs@r2rrs5/(3-/"##';r3rc`eZdZdZdeffd ZdejdejfdZxZ S)UMT5ClassificationHeadz-Head for sentence-level classification tasks.rJc(t|tj|j|j|_tj |j|_tj|j|j|_ y)N)r) r'r(rrOrPdenserTclassifier_dropoutrV num_labelsout_projrYs r2r(zUMT5ClassificationHead.__init__sZ YYv~~v~~> zzF$=$=>  &..&2C2CD r3r@rc|j|}|j|}tj|}|j|}|j |}|Sr[)rVrr*tanhrr_s r2rBzUMT5ClassificationHead.forwardsN ]3  =1  =1  ]3  m4 r3) rCrDrErr!r(r*r]rBrFrGs@r2r r s/7EzE U\\ellr3r cLeZdZUeed<dZdZdZdgZdgZ e dZ dZ dZ y ) UMT5PreTrainedModelrJ transformerTrrScvtjt}tjt}|||d}|S)N)decoder_input_ids input_idsdecoder_attention_mask)r*tensorrr)r.r input_mask dummy_inputss r2rz UMT5PreTrainedModel.dummy_inputss6LL. \\*- !*"&0 r3cx|jj}t|tr)|jj j |dzy t|ttttfr|jjj jd|dzt|drJ|jjs4|jjj jd|dzt|drz|j jj jd||jj"dzz|j j$j j'y y t|t(rpt|drc|j*jj jd|dz|j*j$j j'y y t|t,rM|j.jj jd||jj"dzzt|j.drD|j.j$.|j.j$j j'|j0jj jd||jj"dzzt|j0drF|j0j$/|j0j$j j'y y y t|t2rM|j4jj jd||jj"dzzt|j4drD|j4j$.|j4j$j j'|j6jj jd||jj8dzzt|j6drF|j6j$/|j6j$j j'y y y t|t:r|j<jj jd||jj"dzzt|j<drD|j<j$.|j<j$j j'|j>jj jd||jj"dzzt|j>drD|j>j$.|j>j$j j'|j6jj jd||jj8dzzt|j6drF|j6j$/|j6j$j j'y y y t|t@ri|jj"}|jjB}|jjD}|jFjj jd|||zdzz|jHjj jd||dzz|jJjj jd||dzz|jLjj jd|||zdzz|jNr8|jPjj jd||dzzy y y ) zInitialize the weights?)r;stdlm_head qa_outputs classifierrNN))rJinitializer_factorr\r%r,datafill_ UMT5ModelUMT5ForConditionalGenerationUMT5EncoderModelUMT5ForQuestionAnsweringsharednormal_hasattrtie_word_embeddingsr"r#rPrNzero_UMT5ForTokenClassificationr%r rrrIrRrSrQrbrdrertr{r}rrrrrxr)r.modulefactorrPr|r~s r2 _init_weightsz!UMT5PreTrainedModel._init_weightss// fm , MM   $ $Vc\ 2  , (    MM % % - -3FSL - Ivy)$++2Q2Q%%**22#2Nv|,!!((--553Ft{{ObObgkNkDl5m!!&&++113- : ;v|,!!((--553FSL5Q!!&&++113- 6 7 LL   $ $ , ,#6dkkFYFY^bEb;c , dv||V,1B1B1N !!&&,,. OO " " ' ' / /SfI\I\aeHe>f / gv/FOO4H4H4T$$))//15U/  1 2 II   ! ! ) )s4;;CVCV[_B_8` ) avyy&)fiinn.H ##))+ II   ! ! ) )s4;;CSCSX\B\8] ) ^vyy&)fiinn.H ##))+/I)  6 7 KK   # # + +&T[[EXEX]aDa:b + cv{{F+ 0@0@0L   %%++- KK   # # + +&T[[EXEX]aDa:b + cv{{F+ 0@0@0L   %%++- II   ! ! ) )s4;;CSCSX\B\8] ) ^vyy&)fiinn.H ##))+/I)  .kk))G!%!1!1 kk++G HHOO ( (cv'L^B^cgAg7h ( i HHOO ( (cv$7O ( P HHOO ( (cv$7O ( P HHOO ( (cv'L^B^cgAg7h ( i11..55::BBQW\chl[lQmBn2/r3c|jj}|jj}| tdt |rGt j |jdddz|}t j||dddfgd}n>|j|j}|dddfj|dddf<||d<| td|j|d k(||S) Nzself.model.config.decoder_start_token_id has to be defined. In UMT5 it is usually set to the pad_token_id. See UMT5 docs for more information.r6)r .rr ).rz1self.model.config.pad_token_id has to be defined.) rJdecoder_start_token_id pad_token_id ValueErrorrr*fullrcat new_zerosclone masked_fill_)r.rr8r9shifted_input_idss r2 _shift_rightz UMT5PreTrainedModel._shift_rightPs!%!C!C{{// ! )6  Y ' % 9??3B+?$+FH^ _  % +Q*RXZ [  ) 3 3IOO D )238)<)B)B)D c12g &(> f %  PQ Q&&'8D'@,O  r3N)rCrDrEr!__annotations__base_model_prefixsupports_gradient_checkpointing_can_compile_fullgraph_no_split_modules_keep_in_fp32_modulespropertyrr5rAr3r2rrsI %&*#!$ !F @oD!r3rc eZdZdfd ZdZ ddZ ddeejdfdejdejde d e f d Z e dejd e d e d ejdejde f dZxZS) UMT5Stackc t||||_|j|_t j t |jDcgc]}t||c}|_ t|j|j|_ t j|j|_d|_|j%ycc}w)NrrlF)r'r( embed_tokensrwrrrange num_layersrblockr%rProfinal_layer_normrTrUrVgradient_checkpointing post_init)r.rJrMir1s r2r(zUMT5Stack.__init__ms  ( ++]]ERXRcRcLd#eqIf$B#ef -fnn&B[B[ \zz&"5"56 ',#  $fsCc||_yr[)rMr.new_embeddingss r2set_input_embeddingszUMT5Stack.set_input_embeddingsys *r3c | | n|jj} | | n|jj} | | n|jj} | | n|jj} |$|"|j rdnd}t d|d|d|&|j}|jd|d}n8||jdd}n"|j rdnd}t d|d|d |jr%|jr| rtjd d } |(|j t d |j|}|\}}| d ur|j st d|d|j rB| rN|L|jjrtt!t!}nt!}n|j sd}||j#nd}| %t%j&|||z|j(} |1t+s'||z}t%j,|||j(}|j r2|j/||| t1|tr |j2n|| }n]|Y|ddddddf}|j5|j6}d|z t%j8|j6j:z}nd}|j rO|M|j\}}}||f}|!t%j,||j(}|j=|}nd}|j?||jj@}|j?||jj@}| rdnd}| rdnd}| r|j rdnd}|jC|}tE|jFD]P\}}||} ||}!| r||fz}|||||| |!|| | |  }"|"d}| s2||"dfz }|j sH||"dfz }R|jI|}|jC|}| r||fz}| stKd|||||fDStM|||||S)Ndecoder_zYou cannot specify both zinput_ids and zinputs_embeds at the same timer6zYou have to specify either z input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fzz$UMT5Stack.forward..s = s)last_hidden_statepast_key_valuesr@ attentionscross_attentions)'rJrroutput_hidden_statesuse_return_dictrwr:rrrRrrrrMis_encoder_decoderr r rr*rrrr+_update_causal_maskr\rr8r=rrinvert_attention_mask get_head_maskrOrV enumeraterPrQrr)#r.rrrrr\ head_maskcross_attn_head_maskrbrrre return_dictrerr_msg_prefix input_shaperrpast_key_values_lengthmask_seq_lengthrencoder_batch_sizeencoder_sequence_length_encoder_hidden_shapeencoder_extended_attention_maskall_hidden_statesall_attentionsall_cross_attentionsr@rT layer_modulerr layer_outputss# r2rBzUMT5Stack.forward|s "+!6IDKK+/??ZN*>*:.HXXvw  "#..*K!r;r?;I  &',,.s3K+/??ZN:>:J-XfWggtuv v  & &4==##p"    ( !_`` --i8M!, J  ?? #LTFRg!hii ??_4;;11&9,.,.&YO&2nO#OETE`!?!?!Afg  !"\\&(>(KTaThThN  !*B*D4zAO"ZZ OML`L`aN ??22o/BC 44$!K '(D$)9:K%..}/B/B.CK, MT]n%>>*'7 MM  ""&,,Q/ !+??AMnell;$$R(%7!; PP +')#))!, Q  KK , , 6*%%**.DD%  E*..I0CCKQZ[Kr3rrr=rc ||jdk(r|}|Stj|j}tj||f|||j }|dk7rtj |d}|tj||j |jddkDz}|ddddddfj|ddd}||j}|jd} |ddddddd| f|ddddddfj|j z} | dk(} |ddddddd| fj| ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N) fill_valuer=rr )diagonalr]r6r)rr*rrr;rtriurreshapeexpandr>rr8 masked_fill) rrrr=rrkwargsrr mask_length padding_masks r2rz?UMT5Stack._prepare_4d_causal_attention_mask_with_cache_positionjs>  %.*<*<*>!*C(K*' E*..I** -0Ye\j\q\qK!##jjqA 5<< n>S>STWeWmWmnprsWtt tK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c )6 Aq!\k\12r3r[) NNNNNNNNNNNNN)F)rCrDrEr(rXrBrr*r]r rrh staticmethodrr=rrFrGs@r2rKrKls + "#!!g `#( BellK78BllB B  B  BH4 444{{ 4  4  44r3rKc&BeZdZUdZdZeed<ddgZfdZdZ dZ d Z d Z d Z d Ze dd eej$deej&deej$deej(deej&deej&deej*deeeej&deedeej*deej*deedeedeedeedeej$deeej&eff"dZxZS) r)ao Examples: ```python >>> from transformers import UMT5Model, AutoTokenizer >>> model = UMT5Model.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> noisy_text = "UN Offizier sagt, dass weiter werden muss in Syrien." >>> label = " verhandelt" >>> inputs = tokenizer(inputs, return_tensors="pt") >>> labels = tokenizer(label=label, return_tensors="pt") >>> outputs = model(input_ids=inputs["input_ids"], decoder_input_ids=labels["input_ids"]) >>> hidden_states = outputs.last_hidden_state ```umt5rJencoder.embed_tokens.weightdecoder.embed_tokens.weightct||tj|j|j |_tj|}d|_ d|_ d|_ t||j |_ tj|}d|_ d|_ |j|_t||j |_|j#yNFT)r'r(rr vocab_sizerPr-copydeepcopyrwrtie_encoder_decoderrKencodernum_decoder_layersrOdecoderrSr.rJencoder_configdecoder_configr1s r2r(zUMT5Model.__init__s  ll6#4#4fnnE v.$)!#( -2* = v.$(!-2*$*$=$=! =  r3c|jSr[r-r.s r2get_input_embeddingszUMT5Model.get_input_embeddings {{r3c~||_|jj||jj|yr[r-rrXrrVs r2rXzUMT5Model.set_input_embeddings-$  )).9 )).9r3c|jjra|j|jj|j |j|j j|j yyr[rJr0_tie_or_clone_weightsrrMr-rrs r2 _tie_weightszUMT5Model._tie_weightsP ;; * *  & &t||'@'@$++ N  & &t||'@'@$++ N +r3c|jSr[rrs r2 get_encoderzUMT5Model.get_encoder ||r3c|jSr[rrs r2 get_decoderzUMT5Model.get_decoderrr3c|jD]7\}}|jj|jj |9y) Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrr attention prune_headsr.heads_to_prunerheadss r2 _prune_headszUMT5Model._prune_headssE +002 CLE5 LL  u % / / ; ;E B Cr3rrrrrldecoder_head_maskrmencoder_outputsrbr\decoder_inputs_embedsrrrernrrcL| | n|jj} ||n|jj}||j||| || ||}nI|rGt |t s7t |dt |dkDr|dndt |dkDr|dnd}|d}|j||| | ||||| | ||| }|s||zSt|j|j|j|j|j|j|j|jS) a+ input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. Example: ```python >>> from transformers import AutoTokenizer, UMT5Model >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5Model.from_pretrained("google/umt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1 >>> # preprocess: Prepend decoder_input_ids with start token which is pad token for UMT5Model. >>> # This is not needed for torch's UMT5ForConditionalGeneration as it does this internally using labels arg. >>> decoder_input_ids = model._shift_right(decoder_input_ids) >>> # forward pass >>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state ```Nrrr\rlrrernrr r5rar@rc rrr\rbrrrlrmrrrernr)rarbdecoder_hidden_statesdecoder_attentionsrdencoder_last_hidden_staterencoder_attentions)rJrrfrr\rlenrrrarbr@rcrd)r.rrrrrlrrmrrbr\rrrrernrr@decoder_outputss r2rBzUMT5Model.forwards[b"+!6IDKKD $D>D 'tnD d^!D "!!1!12#D $ uU&&');; <%D D r3r)z< UMT5 Model with a `language modeling` head on top. ) custom_introc(peZdZdZdZgdZfdZdZdZdZ dZ d Z e dd e ejd e ej d e ejd e ej"de ej de ej de ej$de eeej$de ede ej de ej de ejde ede ede ede ede ejdeeej eff$dZdej$fdZxZS)r*a Examples: ```python >>> from transformers import UMT5ForConditionalGeneration, AutoTokenizer >>> model = UMT5ForConditionalGeneration.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> summary = "Weiter Verhandlung in Syrien." >>> inputs = tokenizer(article, text_target=summary, return_tensors="pt") >>> outputs = model(**inputs) >>> loss = outputs.loss ```r)rrzlm_head.weightcNt|||j|_t j |j |j|_tj|}d|_ d|_ d|_ t||j|_tj|}d|_ d|_ |j|_t||j|_t j$|j|j d|_|j)y)NFTrM)r'r(rP model_dimrrrr-rrrwrrrKrrrOrrOr"rSrs r2r(z%UMT5ForConditionalGeneration.__init__s  ll6#4#4fnnE v.$)!#( -2* = v.$(!-2*$*$=$=! = yy1B1BO  r3c|jSr[rrs r2rz1UMT5ForConditionalGeneration.get_input_embeddingsrr3c~||_|jj||jj|yr[rrVs r2rXz1UMT5ForConditionalGeneration.set_input_embeddingsrr3c|jjra|j|jj|j |j|j j|j yyr[rrs r2rz)UMT5ForConditionalGeneration._tie_weightsrr3c|jSr[rrs r2rz(UMT5ForConditionalGeneration.get_encoderrr3c|jSr[rrs r2rz(UMT5ForConditionalGeneration.get_decoderrr3rrrrrlrrmrrbr\rlabelsrrrernrrc| | n|jj} ||n|jj}||j||| ||||}nI|rGt |t s7t |dt |dkDr|dndt |dkDr|dnd}|d}| || |j| }|j||| | ||||| |||| }|d}|jjr||jdzz}|j|}d}| ^td }| j|j} ||jd |j!d | jd }|s|f|ddz|z}||f|zS|St#|||j$|j&|j(|j*|j,|j&|j( S) aK input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoTokenizer, UMT5ForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5ForConditionalGeneration.from_pretrained("google/umt5-small") >>> # training >>> input_ids = tokenizer("The walks in park", return_tensors="pt").input_ids >>> labels = tokenizer(" cute dog the ", return_tensors="pt").input_ids >>> outputs = model(input_ids=input_ids, labels=labels) >>> loss = outputs.loss >>> logits = outputs.logits >>> # inference >>> input_ids = tokenizer("Studies have shown that good for you", return_tensors="pt").input_ids >>> outputs = model.generate(input_ids) >>> tokenizer.decode(outputs[0], skip_special_tokens=True) ```Nrrr r5rrr$r7 ignore_indexr6 losslogitsrbrrrdrrr)rJrrfrr\rrrArr0rr"rr8rrrrrbr@rcrdra)r.rrrrrlrrmrrbr\rrrrrernrr@rsequence_output lm_logitsrloss_fctoutputs r2rBz$UMT5ForConditionalGeneration.forwards+j"+!6IDKK_ $D>_ 'tn!_ "d^#_ $!!1!12%_ & uU&&'8 9'_ _ D)ELL)r3r*c*eZdZdZdZdgZfdZdZdZdZ dZ d Z e dd e ejd e ej d e ej d e ej de ede ede edeeej effdZxZS)r+a Examples: ```python >>> from transformers import UMT5EncoderModel, AutoTokenizer >>> model = UMT5EncoderModel.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> input_ids = tokenizer(article, return_tensors="pt").input_ids >>> outputs = model(input_ids) >>> hidden_state = outputs.last_hidden_state ```rrct||tj|j|j |_tj|}d|_ d|_ t||j |_ |jyNF)r'r(rrrrPr-rrrrgrKrrS)r.rJrr1s r2r(zUMT5EncoderModel.__init__ysh  ll6#4#4fnnE v.#( ,1) =  r3c|jSr[rrs r2rz%UMT5EncoderModel.get_input_embeddingsrr3cH||_|jj|yr[)r-rrXrVs r2rXz%UMT5EncoderModel.set_input_embeddingss$  )).9r3c|jjr1|j|jj|j yyr[)rJr0rrrMr-rs r2rzUMT5EncoderModel._tie_weightss2 ;; * *  & &t||'@'@$++ N +r3c|jSr[rrs r2rzUMT5EncoderModel.get_encoderrr3c|jD]D\}}|jj|jdjj |Fy)rrN)rrrPrrrrs r2rzUMT5EncoderModel._prune_headssP +002 PLE5 LL  u % + +A . < < H H O Pr3rrrlr\rrernrc j||n|jj}|j|||||||}|S)aQ input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). Example: ```python >>> from transformers import AutoTokenizer, UMT5EncoderModel >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5EncoderModel.from_pretrained("google/umt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model(input_ids=input_ids) >>> last_hidden_states = outputs.last_hidden_state ```r)rJrfr) r.rrrlr\rrernrs r2rBzUMT5EncoderModel.forwardsKF&1% -'tn-d^- uU&&'8 9--r3r+z UMT5 model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c$eZdZdgZddgZdeffd Ze ddee jdee jdee jd ee jd ee jd ee jd ee jd ee e jdee jdee jdee jdeedeedeedeedeeeff dZxZS)UMT5ForSequenceClassificationFdecoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weightrrrJct||t||_t ||_|j d|_yr)r'r(r)rr classification_headrSmodel_parallelrYs r2r(z&UMT5ForSequenceClassification.__init__s>  $V,#9&#A  #r3rrrrrlrrmrr\rrrrrernrc||n|jj}| d} |$| "td|jj| | | t d|j |}|j||||||||| | | | ||}|d}|j|jjj|j}ttj|jddkDr t d|j \}}}||ddfj#|d |ddd ddf}|j%|}d}| | j|j} |jj&|jj(dk(rd |j_nv|jj(dkDrL| j*tj,k(s| j*tj.k(rd |j_nd |j_|jj&d k(rSt1}|jj(dk(r&||j3| j3}n||| }n|jj&d k(rGt5}||j#d |jj(| j#d }n,|jj&d k(rt7}||| }|s|f|ddz}||f|zS|St9|||j:|j<|j>|j@|jB|jD|jF S)as input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for If no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.) rrrrlrrmrr\rrrrernrr z7All examples must have the same number of tokens.r6 regressionsingle_label_classificationmulti_label_classificationr)$rJrfNotImplementedErrorr1rCr:rAreq eos_token_idr8rrr*unique_consecutivesumrrr problem_typerr=rrrsqueezerrrrbrrrdrrr)r.rrrrrlrrmrr\rrrrrernrreos_maskrrur/sentence_representationrrrrs r2rBz%UMT5ForSequenceClassification.forwardsR|&1%>KbKbJcd   $)>)F  U !% 1 1) < "" )/#9/!5+'"7/!5##  "!*<< 8 89<<_=S=ST u'' Q8 9A =VW W%4%:%:" A{"1(A+">"C"CJPRT_"`abdfhiai"j))*AB  YYv}}-F{{''/;;))Q./;DKK,[[++a/V\\UZZ5OSYS_S_chclclSl/LDKK,/KDKK,{{''<7"9;;))Q.#FNN$4fnn6FGD#FF3D))-JJ+- B 0F0F GUWY))-II,./Y,F)-)9TGf$ Ev E.#33")"?"?&99$55&-&G&G")"?"?&99  r3)NNNNNNNNNNNNNNN)rCrDrE"_keys_to_ignore_on_load_unexpectedrr!r(rrr*rr]rrrrrrrBrFrGs@r2rrs+s)s&79VW$z$15158<=A,0487;=A59=A-1$(,0/3&*!P E,,-P !.P $E$4$45 P !))9)9 : P ELL) P $ELL1P 'u||4P "$u'8'8"9:P   1 12P  ((9(9:P ))*P D>P $D>P 'tnP d^!P " u55 6#P P r3rc0eZdZdgZdgZdeffd Ze ddee jdee jdee jdee jd ee jd ee d ee d ee d e e e jeffdZxZS)r2rz'transformer.encoder.embed_tokens.weightrJc,t|||j|_t||_t j |j|_t j|j|j|_ |jyr[) r'r(rr+rrrTrrVrOr/r%rSrYs r2r(z#UMT5ForTokenClassification.__init__sj   +++F3zz&";";< ))F$6$68I8IJ r3rrrlr\rrrernrc ||n|jj}|j|||||||} | d} |j| } |j | } d} |7 'tn7 d^7  uU\\"$99 :7 7 r3r2c&<eZdZddgZfdZdZdZdZdZdZ e dd e e jd e e jd e e jd e e jd e e jde e jde e j de eee j de e jde e jde e jde e jde ede ede ede edeee jeff"dZxZS)r,rrclt|||j|_t j |j |j|_tj|}d|_ d|_ d|_ t||j|_tj|}d|_ d|_ |j|_t||j|_|j$|_t j&|j|j$|_|j+yr)r'r(rPrrrrr-rrrwrrrKrrrOrrrOr#rSrs r2r(z!UMT5ForQuestionAnswering.__init__s  ll6#4#4fnnE v.$)!#( -2* = v.$(!-2*$*$=$=! =  ++))FNNF4E4EF r3c|jSr[rrs r2rz-UMT5ForQuestionAnswering.get_input_embeddingsrr3c~||_|jj||jj|yr[rrVs r2rXz-UMT5ForQuestionAnswering.set_input_embeddingsrr3c|jjra|j|jj|j |j|j j|j yyr[rrs r2rz%UMT5ForQuestionAnswering._tie_weightsrr3c|jSr[rrs r2rz$UMT5ForQuestionAnswering.get_encoderrr3c|jSr[rrs r2rz$UMT5ForQuestionAnswering.get_decoderrr3rrrrrlrrmrstart_positions end_positionsr\rrrrernrc||n|jj}| | n|jj} | | d} | | | td|j |}| | n|jj} ||n|jj}||j ||| ||||}nI|rGt |ts7t|dt|dkDr|dndt|dkDr|dnd}|d}|j||| d||||| ||| }|d}|j|}|jdd \}}|jd j}|jd j}d}| | t| jdkDr*| jd j|j } t| jdkDr*| jd j|j } |jd}| j#d|} | j#d|} t%| }||| }||| }||zdz }|s||f|ddz|z}||f|zS|St'||||j(|j*|j,|j.|j0|j*|j, S)aI input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. NFrrrr r5r) rrr\rbrrrlrmrrrernr6rr) r start_logits end_logitsrbrrrdrrr)rJrfrr:rArr\rrrr#splitrrrr8rrrrrbr@rcrdra)r.rrrrrlrrmrrrr\rrrrernr@rrrrr  total_loss ignored_indexr start_lossend_lossrs r2rBz UMT5ForQuestionAnswering.forwards9x&1%)F  U !% 1 1) < !*!6IDKKZ $D>Z 'tn!Z "d^#Z $ uU&&')LL M%Z Z r3r,)r+r*r,rr2r)r)Frrrtypingrrr*rtorch.nnrrr activationsr cache_utilsr r r generationrmodeling_attn_mask_utilsrmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrutilsrrrrrrrconfiguration_umt5r!!torch.nn.attention.flex_attentionr"integrations.flex_attentionr# get_loggerrCrModuler%rIrbrjrtrrrr rrKr)r*r+rr2r,__all__rIr3r2r6s " AA!CC)>9.+ !;J   H %+BII+4 .RYY<"))$B)BIIB)JRYY8bii