rh~dZddlZddlmZmZddlZddlZddlmZddlm Z ddl m Z m Z ddl mZdd lmZmZdd lmZdd lmZmZdd lmZdd lmZmZddlmZej<eZ GddejBZ"GddejFZ$GddejFZ%GddeZ&eGddeZ'eGdde'Z(edGdde'eZ)gd Z*y)!zPyTorch XGLM model.N)OptionalUnion)nn)ACT2FN)CacheEncoderDecoderCache)GenerationMixin)_prepare_4d_attention_mask!_prepare_4d_causal_attention_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions)PreTrainedModel)auto_docstringlogging) XGLMConfigc `eZdZdZd dedededeeffd Zdejffd Z xZ S) XGLMScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. num_embeddings embedding_dim padding_idx embed_scalec6t||||||_yN)super__init__r)selfrrrr __class__s y/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/xglm/modeling_xglm.pyrz XGLMScaledWordEmbedding.__init__,s  D& input_idsc<t|||jzSr)rforwardr)rr#r s r!r%zXGLMScaledWordEmbedding.forward0swy)D,<,<<s[(( T 4 #%..t||/A/A$,,J]J].^K Y Fr"c|dz}tjd|dz z }tjtj|tj j | z}tj|tj j jd|jdz}tjtj|tj|gdj|d}|dzdk(r-tj|tj|dgd}| d||ddf<|jtjS) z Build sinusoidal embeddings. This matches the implementation in tensor2tensor, but differs slightly from the description in Section 3.5 of "Attention Is All You Need". r5i'r)r;rdimN)mathlogr-exparangeint64r, unsqueezecatsincosviewzerosr@get_default_dtype)rrrhalf_dimembs r!r>z/XGLMSinusoidalPositionalEmbedding.get_embeddingFs!A%hhuoA.ii XU[[AGGISDPQll>=CCEOOPQRUXUbUbcdUeeii338a@EEnVXY 1  !))S%++na"@AqIC  ""#C Q vve--/00r" position_idspast_key_values_lengthc|j\}}||jz }d|z|z}||jjdkDr'|j||j|j |jj d|jdj|||jjdjS)Nr5rrF) sizer6r9r7rr index_selectrPshapedetach)rrUrVbszseq_lenmax_poss r!r%z)XGLMSinusoidalPositionalEmbedding.forward[s#((* W # g+ 66 T\\&&q) )   gt'9'94;K;K L||((L,=,=b,ABGGWVZVbVbVhVhikVlmttvvr"r)Nr)r'r(r)r*r+rrr7 staticmethodr>r-no_gradr.r%r/r0s@r!r2r24sNScS#SHUXMSG3GsGQYZ]Q^G1c1#1HUXM11(U]]_ wHU\\$: w[^ w wr"r2cpeZdZdZ ddededeedeedeedeef fd Z dd e jd ee jd ee d ee jd ee jdedee jde e jee jee e jffdZ xZS) XGLMAttentionz=Multi-headed attention from 'Attention Is All You Need' paper embed_dim num_headsdropout is_decoderbias layer_idxct|||_||_||_||z|_|j |z|jk7rt d|jd|d|j dz|_||_||_ tj||||_ tj||||_ tj||||_tj||||_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩rg)rrrcrdrehead_dim ValueErrorscalingrfrhrLineark_projv_projq_projout_proj)rrcrdrerfrgrhr s r!rzXGLMAttention.__init__ks "" !Y. MMI %$.. 8MdnnM]$YKr3 }}d* $"ii 94@ ii 94@ ii 94@  )YTB r" hidden_stateskey_value_statespast_key_valueattention_masklayer_head_maskoutput_attentionscache_positionreturncB |du}|j\} } } |r|jdn| } |j||jz} |St |t rA|j j|j}|r |j}n|j}n|}|r|n|}|rK|IrGj|jj}|j|jj}n|j|}|j|}|j!| | d|j"j%dd}|j!| | d|j"j%dd}|D|s|nd}j'|||jd|i\}}|rd|j |j<| |j(zd|j"f}| j!| | |j(|j"j%dd} | j*|} |j*|}|j*|}|jd} t-j.| |j%dd}|j| |j(z| | fk7r/t1d| |j(z| | fd|j||j| d| | fk7r#t1d | d| | fd|j|j!| |j(| | |z}t-j2|t-j4t-j6|j8j:|j< }|j!| |j(z| | }|j8t,j>k(rNt@jBjE|dt,jF jIt,j>}n!t@jBjE|d }||j|j(fk7r*t1d |j(fd|j|j!dddd|j!| |j(| | z}|j!| |j(z| | }|r?|j!| |j(| | }|j!| |j(z| | }nd}t@jBjK||jJ|jL}t-j.||}|j| |j(z| |j"fk7r7t1d| |j(| |j"fd|j|j!| |j(| |j"}|j%dd}|j+| | |jN}|jQ|}||fS)z#Input shape: Batch x Time x ChannelNrrFr5ryTz$Attention weights should be of size z , but is z!Attention mask should be of size )r<)rEr;rDz/Head mask for a single layer should be of size ptrainingz `attn_output` should be of size ))rXrZrqrm isinstancer is_updatedgetrhcross_attention_cacheself_attention_cachelayerskeysvaluesrorprPrk transposeupdaterdreshaper-bmmrlmaxtensorfinfor;minr<float16r functionalsoftmaxfloat32r@rer~rcrr)rrsrtrurvrwrxryis_cross_attentionr\tgt_len_src_len query_statesrcurr_past_key_valuecurrent_states key_states value_states proj_shape attn_weightsattn_weights_reshaped attn_probs attn_outputs r!r%zXGLMAttention.forwards.T9',,.Wa/A"((+w{{=1DLL@  %.*=>+66::4>>J %*8*N*N'*8*M*M'&4#-?)] ."<,33DNNCHHJ.55dnnELLL^4J;;~6L#gr4==ISSTUWXYJ',,S'2t}}MWWXY[\]L)7It+>+E+E dnn?OQ_>`,( L&@DN--dnn=DNN*B > #((gt~~t}}U__`acde +|++Z8 'Z''4 +|++Z8 //!$yyz/C/CAq/IJ    3#7'"J J6dnn8LgW^7_6`a %%'(*   %""$a'(BB 7a'8R7SS\]k]p]p]r\st(,,S$..'7SVddL 99ell5;;|7I7I+J+N+NWcWjWjkL(,,S4>>-A7GTL    .==002U]]0[^^_d_l_lmL==0020FL  &##%$..):: Et~~FWEXY',,./1+//2q!<|?P?PQTVZVdVdfmov?wwL',,S4>>-A7GTL  %1$5$5c4>>7T[$\ !055cDNN6JGU\]L$( !]]**<4<C@48*.1526"'15|2|||2#5<<0|2! |2 !. |2 "%,,/ |2 |2!.|2 u||Xell3XeELL>Q5RR S|2r"rbcDeZdZddeffd Z ddej deej deej deej deej deej d eed ee d ee d eej d ej fdZ xZ S)XGLMDecoderLayerconfigc0t||j|_t |j|j |j d||_|j|_t|j|_ |j|_ |jrWt |j|j |j d||_tj |j|_tj |j|_tj&|j|j(|_tj&|j(|j|_tj |j|_y)NT)rcrdrerfrh)rrd_modelrcrbattention_headsattention_dropout self_attnreractivation_function activation_fnactivation_dropoutadd_cross_attention encoder_attnr LayerNormencoder_attn_layer_normself_attn_layer_normrnffn_dimfc1fc2final_layer_norm)rrrhr s r!rzXGLMDecoderLayer.__init__s &nn,,,,  ~~ #F$>$>?"(";";  % % -.. 0000# !D ,.<<+GD ($&LL$@!99T^^V^^<99V^^T^^< " T^^ <r"rsrvencoder_hidden_statesencoder_attention_maskrwcross_attn_layer_head_maskrurx use_cacheryrzc 2|} |j|}|j|||||| \}} tjj ||j|j }| |z}d} |i|} |j |}|j||||||| \}} tjj ||j|j }| |z}|} |j|}|j|j|}tjj ||j|j }|j|}tjj ||j|j }| |z}|f}|r|| | fz }|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. cross_attn_layer_head_mask (`torch.FloatTensor`): mask for cross-attention heads in a given layer of size `(decoder_attention_heads,)`. past_key_value (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rsrurvrwrxryr|N)rsrtrvrwrurxry) rrrrrer~rrrrrrr)rrsrvrrrwrrurxrryresidualself_attn_weightscross_attn_weightsoutputss r!r%zXGLMDecoderLayer.forward's>!11-@ ,0>>'))+/) ,:, ( ( --mt||VZVcVc-d  =0 " ,$H 88GM040A0A+!65 :-"3-1B1 -M-MM11-4< >Gr"r) NNNNNNFTN) r'r(r)rrr-r.rrrr%r/r0s@r!rrs=z=D268<9=26=A*.,1$(15N||N!.N ( 5 N !) 6 N "%,,/ N%-U\\$:N!N$D>ND>N!.N Nr"rc,eZdZUeed<dZdZdgZdZy)XGLMPreTrainedModelrmodelTrc|jj}t|tjrY|j j jd||j%|jj jyyt|tjrf|j j jd||j2|j j |jjyyy)Nr)meanstd) rinit_stdrrrnweightdatanormal_rgzero_ Embeddingr)rmodulers r! _init_weightsz!XGLMPreTrainedModel._init_weightsskk"" fbii ( MM   & &CS & 9{{&   &&('  - MM   & &CS & 9!!- ""6#5#56<<>..r"N) r'r(r)r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modulesrr"r!rrxs" &*#+, ?r"rc"eZdZddedeej ffd Ze ddee jdee jdee jdee jdee jd ee jd ee jd ee e jd ee jd ee dee dee dee dee jdeee jeffdZxZS) XGLMModelr embed_tokensc t|||j|_|j|_|j|_|j |_|jrtj|jnd}|||_ n2t|j|j|j ||_ t|j |j|j|_t#j$t'|j(Dcgc]}t+||c}|_t#j.|j|_d|_|j5ycc}w)zZ embed_tokens (`nn.Embedding`, *optional*): output embeddings r&N)r)rhF)rrre layerdrop pad_token_idrmax_position_embeddingsmax_target_positionsscale_embeddingrGsqrtrrr vocab_sizer2embed_positionsr ModuleListrange num_layersrrr layer_normgradient_checkpointing post_init)rrrrir s r!rzXGLMModel.__init__s  ~~ ))!..$*$B$B!393I3Idii/s  # ,D  7!!6>>43C3CQ\!D  A  * * NN     mmTYZ`ZkZkTl$mq%5f%J$mn ,,v~~6&+#  %nsE0r#rvrUrr head_maskcross_attn_head_maskpast_key_values inputs_embedsrrxoutput_hidden_states return_dictryrzc | | n|jj} | | n|jj} | | n|jj} | | n|jj} | | t d|8|j |||j}|jd|d}n!| | jdd}n t d| |j|} |jr%|jr| rtjdd} d}| r|j }| rd nd}| rd nd}| r|d nd}tC||gddgD]j\}}| |jd tE|jFk7s3t d|dtE|jFd|jd dtI|jFD]{\}}| r||fz }|jr%t)jJg}||jLkr?||||||||nd|||nd|| | | }|d }| sg||dfz }|s||dfz }}|jO|}| r||fz }|r|jQ}| stSd|||||fDStU|||||S)a encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) cross_attn_head_mask (`torch.Tensor` of shape `(num_layers, attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. NzDYou cannot specify both input_ids and inputs_embeds at the same timerFz5You have to specify either input_ids or inputs_embedsz_`use_cache = True` is incompatible with gradient checkpointing`. Setting `use_cache = False`...FTzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.rr:)rr|rrrzThe `z` should be specified for z layers, but it is for .)rrwrrurxrryrr5c3$K|]}|| ywrr).0vs r! z$XGLMModel.forward..Fs=s)last_hidden_staterrs attentionscross_attentions)+rrxrruse_return_dictrl%warn_if_padding_and_no_attention_maskrXrPrrr~logger warning_oncerrr from_legacy_cacheget_seq_lengthr r-rJlongr<rLr r;rr@rrrer,ziplenr enumeraterandrrto_legacy_cacherr)rr#rvrUrrrrrrrrxrrry input_shapereturn_legacy_cacherVrsall_hidden_statesall_self_attnsall_cross_attentions attn_mask mask_nameidx decoder_layerdropout_probability layer_outputss r!r%zXGLMModel.forwardsjH2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKcd d  "  6 6y. Q#..*K!r;r?;I  &',,.s3KTU U   --i8M  & &4==##u" $ Z?"&    \  2CCOTOETE`!?!?!Afg: K8N    <<&B"88jj+4+@y''mFZFZ L (11!4L ! ,1G1S%?& (;(;[QS_& "&(<(<\Ka(b(e(e  )   --muT\\?R]a]j]j-k #7BD0d&7>#A&#dkk*::$ {*DSEUDVW%NN,Q/03 #,DKK"8 @ C#!m%55!}}&+jjn#&7)%'=3<3H3dI]Ii,@,Eos."3#- M*!,M =#3"55(4(]1-=,??(7 @: 6  -!1 1  -==?O':K^]qr  9+++%1   r"r)NNNNNNNNNNNNNN)r'r(r)rrrrrrr-r.list FloatTensorrrrrr%r/r0s@r!rrsz",,9O>-115/38<9=,07;=A04$(,0/3&*15d ELL)d !.d u||, d ( 5 d !) 6 d ELL)d 'u||4d "$u'8'8"9:d   -d D>d $D>d 'tnd d^d !.d uU\\"$MM N!d d r"rz The XGLM Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introc$eZdZdZdgZfdZe ddeejdeejdeejdeejdeejd eejd eejd ee ejd eejd eejdee dee dee dee deejde eejeff dZxZS)XGLMForCausalLMrzlm_head.weightct||t||_t j |j |jd|_|jy)NFrj) rrrrrrn hidden_sizerlm_headr)rrr s r!rzXGLMForCausalLM.__init__^sH  v& yy!3!3V5F5FUS  r"r#rvrUrrrrrrlabelsrrxrrryrzc &| | n|jj} | | n|jj} ||n|jj}|j ||||||||| | | | ||}|j |d}d}| ?|j || f|jj|jjd|}|s|f|ddz}||f|zS|St|||j|j|j|jS)a encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) cross_attn_head_mask (`torch.Tensor` of shape `(num_layers, attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. N)r#rvrUrrrrrrrrxrrryr)rrr)losslogitsrrsrr)rrxrrrr loss_functionrrrrrsrr)rr#rvrUrrrrrrrrrxrrrykwargsrrroutputs r!r%zXGLMForCausalLM.forwardfsYV2C1N-TXT_T_TqTq$8$D $++JjJj &1%Y $D>Y 'tnY d^Y !.!Y $ uU\\"$EE F%Y Y r"r)rrr)+r*rGtypingrrr-torch.utils.checkpointr activationsr cache_utilsrr generationr modeling_attn_mask_utilsr r modeling_layersr modeling_outputsrrmodeling_utilsrutilsrrconfiguration_xglmr get_loggerr'rrrModuler2rbrrrr__all__rr"r!r+s " !5)e9l-,*   H % =bll =1w 1wh\2BII\2~n1nb?/??$E #E E P f )?f  f R Br"